Synthetic control in Python: read the pre-fit before the gap

Let's put ourselves in a common situation. We have panel data: one treated unit, a handful of untreated "donor" units, and an outcome measured over time. Treatment starts at some point partway through. We have Python open, and we have heard that synthetic control "reads a treatment effect off a gap." We want to see that happen with our own code.

We can do this in a few lines. We will build the synthetic a few different ways, look at what the graphs and numbers show, and find where the trap is. A synthetic control has three moving parts: the treated unit (the one that got the policy), the donor pool (untreated units we will average together), and the gap (the distance between the treated unit and its synthetic twin after treatment). Here is what a clean case looks like.

Where the gap comes from

To test the method, we build data where we already know the answer. We stack a time series for the treated unit and time series for 15 donor units across 24 periods. Treatment starts in period 12, and after that we add a real effect of 6 units to the treated outcome. Because we built it, we know the truth is exactly 6.0, and we can check what each estimator reports against it.

Synthetic control follows a fixed procedure: build a weighted average of the donors that tracks the treated unit before treatment, extend that average afterward as the counterfactual, and read the gap between the treated unit and that synthetic twin as the effect. In code, we fit weights on the pre-period, apply them to the post-period, and average the difference.

import numpy as np

rng = np.random.default_rng(20260705)
T, T0, K, J, TAU = 24, 12, 3, 15, 6.0          # periods, pre-periods, factors, donors, effect
F = rng.normal(0, 1, (T, K))                   # common factors over time
donor_load = rng.normal(0, 1, (J, K))
treat_load = rng.dirichlet(np.ones(J)) @ donor_load   # treated sits INSIDE the donor range
donors  = donor_load @ F.T + rng.normal(0, 0.5, (J, T))
treated = treat_load  @ F.T + rng.normal(0, 0.5, T)
treated[T0:] += TAU                            # plant the effect after period 12

On screen the lines and the gap look clean, and that is where the problem begins.

First move: unrestricted weights and a zero-error pre-fit

Start with the most naive move. We fit unrestricted least-squares weights on the pre-period, with no constraints, asking only for the closest match to the treated unit before treatment. Because we have more donors (15) than pre-treatment periods (12), the fit reproduces the treated unit's pre-period path with zero error: the pre-period error, measured as root-mean-square prediction error (pre-RMSPE), is 0.000.

w, *_ = np.linalg.lstsq(donors[:, :T0].T, treated[:T0], rcond=None)
pre_rmspe = np.sqrt(np.mean((treated[:T0] - donors[:, :T0].T @ w) ** 2))
gap = np.mean(treated[T0:] - donors[:, T0:].T @ w)
print(round(pre_rmspe, 3), round(gap, 3))      # -> 0.0  6.132

When we read the gap after treatment, we get about 6.132 against the true planted effect of 6.0. Here is the surprise: that number is not biased. Across 200 simulated panels the unrestricted gap averages about 6.002, right on the truth. Overfitting did not produce a wrong effect in this clean setup. The number is right and the pre-fit has zero error, and that zero-error pre-fit is still the problem.

Why the zero-error pre-fit doesn't mean what we think

A zero-error pre-fit is available for any treated unit if we give the optimizer enough slack. A pre-RMSPE of exactly zero signals that the weights were free enough to match anything we handed them, whether or not the comparison is good. Two consequences follow. First, the unrestricted gap is noisier from panel to panel: across 200 simulations its spread (standard deviation) is about 0.425, against 0.185 for the constrained estimator we build next.

Second, and more important, the zero-error fit removes the one number that would flag an unusable donor pool: a non-zero pre-RMSPE that reflects real tracking difficulty. We put that number to use next.

We might try the obvious repair: we overfit, so let's add a ridge penalty and shrink the weights. That changes the pre-RMSPE, but each penalty setting reports a different pre-fit quality for the same panel, around 0.018, 0.111, and 0.273 for penalties of 0.1, 1.0, and 10.0, while the gaps stay near 6. The dial we pick, not the data, sets the pre-fit.

Second move: constrained weights make the pre-RMSPE mean something

Now we switch to the version synthetic control is actually built around. We constrain the weights: every weight is non-negative, the weights sum to one, and they minimize pre-period error under those two rules. The synthetic can then only be a weighted average of the donors, not an arbitrary extrapolation. We lift it as a small helper and read three things off it: the weights, the pre-RMSPE, and the gap.

from scipy.optimize import minimize

def synthetic_control(treated, donors, t0):
    """Convex SC: weights >= 0 that sum to 1, minimizing pre-period error.
    Returns (weights, pre_rmspe, gap). The pre_rmspe is a DIAGNOSTIC, not a target."""
    n = donors.shape[0]
    yt, yd = treated[:t0], donors[:, :t0]
    obj  = lambda w: np.sum((yt - w @ yd) ** 2)
    cons = ({"type": "eq", "fun": lambda w: np.sum(w) - 1},)
    w = minimize(obj, np.ones(n) / n, method="SLSQP",
                 bounds=[(0, 1)] * n, constraints=cons).x
    pre_rmspe = np.sqrt(np.mean((yt - w @ yd) ** 2))
    gap = float(np.mean(treated[t0:] - w @ donors[:, t0:]))
    return w, pre_rmspe, gap

w, pre_rmspe, gap = synthetic_control(treated, donors, T0)
print(round(pre_rmspe, 3), round(gap, 3))       # -> 0.303  6.058

On the same panel, the pre-RMSPE is no longer zero (about 0.303), the gap is close to the planted 6.0 (about 6.058), and the spread across simulations tightens. That non-zero pre-RMSPE now measures something real: how well a weighted counterfactual can track the treated unit. The unrestricted fit produced no such number; the constraint is what makes it available.

Checking sensitivity to the fit window

Before we trust the constrained gap, we check whether it swings when we change how much pre-history we fit on. We slide the fit window across the last 6, 9, and 12 pre-periods and re-read the gap each time.

Third move: a donor pool with no valid counterfactual

Until now the treated unit has sat inside the range of the donors, so a valid counterfactual exists. Now we remove it. We place the treated unit outside the donors' range, so no weighted average of them can track it, and refit both estimators.

The unrestricted fit still reports a pre-RMSPE of 0.000. It reproduces the treated unit's pre-period path, but the gap it reports is now about 4.299 instead of 6.0: a zero-error pre-fit and a wrong effect. The convex fit cannot reproduce that path. Its pre-RMSPE rises to about 7.613, because no combination of these donors can track the treated unit, and that large number is the evidence of an unusable pool.

Where the diagnostics stop and judgment begins

Here we reach the limit of what any in-sample number can show. A small pre-RMSPE from a constrained synthetic control is necessary for a credible design: if the synthetic cannot track the treated unit before treatment, the counterfactual is weak. But a small pre-RMSPE is never sufficient. Even with good pre-period tracking, the donor pool might still be the wrong set of comparisons. No pre-RMSPE, placebo test, or robustness sweep inside the design can prove that a valid counterfactual exists. That judgment comes from knowing what the donors are.

Inference limits: what the permutation p can and can't say

Significance in a synthetic control usually comes from a permutation test. We reassign the treatment to each donor in turn, recompute a synthetic for it, and compare the real treated unit's gap against that placebo distribution. With 15 donors, the smallest p-value we can possibly get is 1 divided by 16, or about 0.062, and in our setup the permutation p sits exactly at that floor even though the planted effect is a large 6.0.

A p-value above 0.05 can mean the pool is too small to reach conventional significance even when the effect is real. When we report the permutation p, we report its floor alongside it: 1 divided by (number of donors plus one).

What we read before we look at the gap

When we are at the keyboard about to interpret a synthetic control, we walk these four checks before we trust the gap.

1. Read the pre-RMSPE before the gap, and distrust a pre-RMSPE of exactly zero. A zero-error pre-fit signals unconstrained weights, not a good counterfactual.
2. Use convex weights, non-negative and summing to one, so the pre-RMSPE is a diagnostic rather than a target. Unrestricted or penalized fits trade that diagnostic for a hand-tuned dial.
3. Argue donor-pool validity on substance, not fit. Each donor needs a real story as a plausible stand-in for the treated unit; no in-sample number can supply it.
4. Report the permutation p alongside its floor of 1 over (J + 1). In a small pool, "not significant" can be a design limit rather than an absent effect.

The gap is the output. The finding we care about is whether the counterfactual is credible and the estimate is stable across reasonable design choices.

Reproduction

The full simulation framework is at github.com/dphdame/tooearlytosay-analysis. It includes the factor-model data generation, the unrestricted and convex estimators, the ridge sweep, the fit-window walk, the invalid-pool case, the permutation test, and the script that generates every figure above from the same seeded data. Results reproduce exactly. Shifting the treated unit inside or outside the donor range is what flips the pre-fit diagnostic from small to large.