Synthetic control in Python: read the pre-fit before the gap

A synthetic-control estimate is credible only when the weighted donor pool tracks the treated unit before treatment under a design fixed without using the post-treatment gap. Read the pre-treatment fit, donor weights, and held-out performance before interpreting that gap.

The method has three moving parts: the treated unit that receives the policy, the donor pool of untreated units available for comparison, and the gap between the treated outcome and its weighted counterfactual after treatment. We can build each part in Python, then use a planted truth to show where an apparently clean fit fails.

Where the gap comes from

To test the method, we build data where we already know the answer. We stack a time series for the treated unit and time series for 15 donor units across 24 periods. Treatment starts in period 12, and after that we add a real effect of 6 units to the treated outcome. Because we built it, we know the truth is exactly 6.0, and we can check what each estimator reports against it.

Synthetic control follows a fixed procedure: build a weighted average of the donors that tracks the treated unit before treatment, extend that average afterward as the counterfactual, and read the gap between the treated unit and that synthetic twin as the effect. In code, we fit weights on the pre-period, apply them to the post-period, and average the difference.

import numpy as np

rng = np.random.default_rng(20260705)
T, T0, K, J, TAU = 24, 12, 3, 15, 6.0          # periods, pre-periods, factors, donors, effect
F = rng.normal(0, 1, (T, K))                   # common factors over time
donor_load = rng.normal(0, 1, (J, K))
treat_load = rng.dirichlet(np.ones(J)) @ donor_load   # treated sits INSIDE the donor range
donors  = donor_load @ F.T + rng.normal(0, 0.5, (J, T))
treated = treat_load  @ F.T + rng.normal(0, 0.5, T)
treated[T0:] += TAU                            # plant the effect after period 12

On screen the lines and the gap look clean, and that is where the problem begins.

First move: unrestricted weights and a zero-error pre-fit

Start with unrestricted least-squares weights on the pre-period. In this seeded simulation, the 12-by-15 donor design has full row rank, so the linear system has a zero-residual solution and the pre-period root-mean-square prediction error (pre-RMSPE) is 0.000. Having more donors than pre-periods makes that interpolation possible when the relevant rank condition holds; the dimension count alone does not guarantee an exact fit in every dataset.

w, *_ = np.linalg.lstsq(donors[:, :T0].T, treated[:T0], rcond=None)
pre_rmspe = np.sqrt(np.mean((treated[:T0] - donors[:, :T0].T @ w) ** 2))
gap = np.mean(treated[T0:] - donors[:, T0:].T @ w)
print(round(pre_rmspe, 3), round(gap, 3))      # -> 0.0  6.132

When we read the gap after treatment, we get about 6.132 against the true planted effect of 6.0. Across 200 simulated panels the unrestricted gap averages about 6.002, right on the truth. Overfitting did not produce a wrong average effect in this clean setup. The zero-error pre-fit still does no validating work because interpolation was mechanically available.

Why this zero-error pre-fit is not validation

In this underdetermined, full-row-rank design, unrestricted weights can interpolate any 12-period treated path. That mechanical availability makes the observed zero uninformative about donor validity. A zero is not, by itself, proof that weights were unconstrained: convex weights can also fit exactly when the treated trajectory lies in the donors' convex hull. The constraints, rank, weight concentration, and held-out performance must be inspected directly. In this simulation the unrestricted gap is also noisier from panel to panel: across 200 simulations its spread (standard deviation) is about 0.425, against 0.185 for the constrained estimator built next.

Second, unrestricted in-sample interpolation prevents pre-RMSPE from revealing tracking difficulty in this simulation. Convex constraints expose that difficulty in the invalid-pool example below, but even a constrained in-sample fit remains a model-fit measure rather than a validity certificate.

Adding a ridge penalty changes the estimator's weights and in-sample fit, not the causal estimand. The reported settings produce pre-RMSPE values around 0.018, 0.111, and 0.273 for penalties of 0.1, 1.0, and 10.0, while the gaps stay near 6. Those values are comparable only conditional on the penalty. A penalty can be selected with a pre-specified, held-out pre-treatment block; it should not be chosen by whichever full-preperiod fit or post-treatment gap looks best.

Second move: constrain the synthetic to the donor convex hull

Now we switch to the version synthetic control is actually built around. We constrain the weights: every weight is non-negative, the weights sum to one, and they minimize pre-period error under those two rules. The synthetic can then only be a weighted average of the donors, not an arbitrary extrapolation. We lift it as a small helper and read three things off it: the weights, the pre-RMSPE, and the gap.

from scipy.optimize import minimize

def synthetic_control(treated, donors, t0):
    """Convex SC: weights >= 0 that sum to 1, minimizing pre-period error.
    Returns (weights, pre_rmspe, gap). The pre_rmspe is a DIAGNOSTIC, not a target."""
    n = donors.shape[0]
    yt, yd = treated[:t0], donors[:, :t0]
    obj  = lambda w: np.sum((yt - w @ yd) ** 2)
    cons = ({"type": "eq", "fun": lambda w: np.sum(w) - 1},)
    result = minimize(obj, np.ones(n) / n, method="SLSQP",
                      bounds=[(0, 1)] * n, constraints=cons)
    if not result.success:
        raise RuntimeError(result.message)
    w = result.x
    pre_rmspe = np.sqrt(np.mean((yt - w @ yd) ** 2))
    gap = float(np.mean(treated[t0:] - w @ donors[:, t0:]))
    return w, pre_rmspe, gap

w, pre_rmspe, gap = synthetic_control(treated, donors, T0)
print(round(pre_rmspe, 3), round(gap, 3))       # -> 0.303  6.058

# Fail-loud engineering gates for this planted-truth simulation.
assert np.isclose(w.sum(), 1.0, atol=1e-8)
assert np.all(w >= -1e-8)
assert abs(gap - TAU) <= 0.25

On the same panel, the convex fit has pre-RMSPE about 0.303, a gap about 6.058 against the planted 6.0, and a tighter spread across simulations. The pre-RMSPE describes how well a convex combination tracks this treated path in the periods used to fit the weights. It can reveal a severe lack of support, but it remains in-sample and does not prove that the post-treatment counterfactual is valid.

Hold out pre-treatment periods before reading the gap

A fit-window sweep refits on 6, 9, or 12 pre-periods and then looks at the post-treatment gap. That is a useful sensitivity display, but it is not cross-validation because every comparison is interpreted using post-treatment outcomes. A cleaner design check trains the weights on an early pre-treatment block and evaluates prediction error on a later pre-treatment block that was not used to fit them.

from scipy.optimize import minimize

def fit_convex_weights(treated, donors, train_idx):
    n = donors.shape[0]
    y_train = treated[train_idx]
    x_train = donors[:, train_idx]
    objective = lambda w: np.mean((y_train - w @ x_train) ** 2)
    constraints = ({"type": "eq", "fun": lambda w: w.sum() - 1},)
    result = minimize(
        objective,
        np.full(n, 1 / n),
        method="SLSQP",
        bounds=[(0, 1)] * n,
        constraints=constraints,
    )
    if not result.success:
        raise RuntimeError(result.message)
    return result.x

train_idx = np.arange(0, 8)
validation_idx = np.arange(8, T0)
w_train = fit_convex_weights(treated, donors, train_idx)

train_rmspe = np.sqrt(np.mean(
    (treated[train_idx] - w_train @ donors[:, train_idx]) ** 2
))
validation_rmspe = np.sqrt(np.mean(
    (treated[validation_idx] - w_train @ donors[:, validation_idx]) ** 2
))
print(train_rmspe, validation_rmspe)

# Choose the donor specification and any tuning rule without using the
# post-treatment gap. After that choice is fixed, refit on all pre-periods.

The holdout must respect time order, and the validation block should be long enough to be informative. Report both errors and compare the treated unit with placebo units subjected to the same procedure. A single split is still noisy, especially with 12 pre-periods, so this check supplements rather than replaces substantive donor selection.

The pinned simulation also reports the original fit-window sensitivity display:

Third move: a donor pool with no valid counterfactual

Until now the data-generating process places the treated factor loading inside the donors' convex hull, so a convex donor combination represents its untreated factor component. Now we move that loading outside the hull. No convex combination can reproduce the resulting path, and we refit both estimators.

The unrestricted fit still reports a pre-RMSPE of 0.000. It reproduces the treated unit's pre-period path, but the gap it reports is now about 4.299 instead of 6.0: a zero-error pre-fit and a wrong effect. If a funding rule required an effect of at least 5 units, this invalid pool would reject a program whose planted effect clears the rule. The convex fit cannot reproduce that path. Its pre-RMSPE rises to about 7.613, flagging poor support in this planted invalid-pool simulation.

Where the diagnostics stop and judgment begins

Here we reach the limit of what any in-sample number can show. A small pre-RMSPE from a constrained synthetic control is necessary for a credible design: if the synthetic cannot track the treated unit before treatment, the counterfactual is weak. But a small pre-RMSPE is never sufficient. Even with good pre-period tracking, the donor pool might still be the wrong set of comparisons. No pre-RMSPE, placebo test, or robustness sweep inside the design can prove that a valid counterfactual exists. That judgment comes from knowing what the donors are.

Inference limits: what the permutation p can and can't say

Significance in a synthetic control usually comes from a permutation test. We reassign the treatment to each donor in turn, recompute a synthetic for it, and compare the real treated unit's gap against that placebo distribution. With 15 donors, the smallest p-value we can possibly get is 1 divided by 16, or about 0.062, and in our setup the permutation p sits exactly at that floor even though the planted effect is a large 6.0.

A p-value above 0.05 can mean the pool is too small to reach conventional significance even when the effect is real. When we report the permutation p, we report its floor alongside it: 1 divided by (number of donors plus one).

What we read before we look at the gap

When we are at the keyboard about to interpret a synthetic control, we walk these four checks before we trust the gap.

1. Inspect the constraints, rank, and weights before interpreting a zero. Exact fit can arise from unrestricted interpolation or from a genuine convex-hull match; the zero alone does not distinguish them or validate the counterfactual.
2. Hold out pre-treatment periods. Report training and validation RMSPE under the same pre-specified donor and tuning rules, then compare the treated unit with placebo units.
3. Argue donor-pool validity on substance, not fit. Each donor needs a real story as a plausible stand-in for the treated unit; no in-sample or held-out number can supply it.
4. Report the permutation procedure and its attainable floor. Under the rank test used here with 15 donors plus the treated unit, the floor is 1 over 16. Other permutation conventions can have different support.

The gap is the output. The finding we care about is whether the counterfactual is credible and the estimate is stable across reasonable design choices.

References

1. Abadie, A., Diamond, A., & Hainmueller, J. (2010). Synthetic control methods for comparative case studies: Estimating the effect of California's tobacco control program. Journal of the American Statistical Association, 105(490), 493-505.
2. Abadie, A. (2021). Using synthetic controls: Feasibility, data requirements, and methodological aspects. Journal of Economic Literature, 59(2), 391-425.

Reproduction

The full simulation framework is in the pinned public package. It includes the factor-model data generation, the unrestricted and convex estimators, the ridge sweep, the fit-window walk, the invalid-pool case, the permutation test, and the script that generates every figure above from the same seeded data. The fixed-seed numerical harness has passed a clean-clone release check and reproduces exactly at that commit. The figure script is included as a site-maintainer workflow, but it is not part of that clean-clone evidence gate. Shifting the treated unit inside or outside the donor range is what flips the pre-fit diagnostic from small to large.