Difference-in-differences in Python: why the TWFE coefficient can mislead

Difference-in-differences compares the before-and-after change for a treated group against the same change for a comparison group. When every unit is treated at the same moment, that is a clean comparison and the familiar two-way fixed-effects (TWFE) regression, unit fixed effects plus period fixed effects plus a treatment dummy, recovers the effect. The trouble starts when treatment rolls out at different times, which is the common case: some states adopt a policy in one year, others later, some never. With staggered timing, TWFE starts using already-treated units as controls, and its single coefficient stops being the average effect.

To see what that does, we can plant known effects in a simulated panel and ask which estimator recovers them. We build three cohorts over ten periods: an early cohort that adopts at period 4, a late cohort that adopts at period 7, and a never-treated group. The effect turns on at adoption and grows with exposure, and the early cohort's per-period effect is larger than the late cohort's. Because we built it, we know the true average effect across all treated unit-periods is exactly 1.60.

The naive move: one TWFE coefficient

We reach for the standard specification: regress the outcome on unit fixed effects, period fixed effects, and a single dummy for being treated. We run it and get one coefficient back.

import numpy as np, statsmodels.api as sm

# panel columns: uid, period t, cohort (adoption period; -1 = never), y
post = ((cohort >= 0) & (t >= cohort)).astype(float)   # treated-and-after
X = np.column_stack([post, unit_dummies[:, 1:], period_dummies[:, 1:], np.ones(len(y))])
beta = np.linalg.lstsq(X, y, rcond=None)[0]
print(round(beta[0], 3))          # -> 0.97 on this panel; averages 1.01 over 200 (planted 1.60)

Run this across 200 simulated panels and we get an average of 1.01 against the planted 1.60, about 37% too low. The regression is coded correctly. The problem is what it averages: with staggered timing, TWFE forms some comparisons that use the already-treated early cohort as a control for the later cohort. The early cohort's own effect is still growing, so subtracting its trend removes part of the real effect, pulling the estimate down. This is the negative-weight problem (Goodman-Bacon, 2021; de Chaisemartin and D'Haultfœuille, 2020), and nothing in the TWFE output flags it.

The fix: compare only against clean controls

The repair is to never use an already-treated unit as a control. A group-time estimator (Callaway and Sant'Anna, 2021) does this directly: for each cohort and each period after it adopts, it measures the cohort's change from its last pre-treatment period, then subtracts the same change for units that are not yet treated, and averages those clean comparisons.

def att_group_time(t, cohort, y, T):
    num, den = 0.0, 0
    for g in [c for c in np.unique(cohort) if c >= 0]:
        base = int(g) - 1                                  # last pre-treatment period
        for tt in range(int(g), T):
            treated = (cohort == g)
            control = (cohort == -1) | (cohort > tt)       # never- or not-yet-treated only
            dg = y[treated & (t == tt)].mean() - y[treated & (t == base)].mean()
            dc = y[control & (t == tt)].mean() - y[control & (t == base)].mean()
            n = (treated & (t == tt)).sum()
            num += (dg - dc) * n; den += n
    return num / den

print(round(att_group_time(t, cohort, y, T), 3))   # -> 1.61 on this panel; averages 1.60 over 200

Run the same panels through the group-time estimator and we recover 1.60, the planted effect. Both regressions run without error; what differs is which comparisons each one makes.

The bias is real, and it is invisible in the output

The gap is a property of staggered heterogeneity rather than a coding slip, and rerunning the naive TWFE under four scenarios shows it directly. When every cohort has the same effect and that effect is flat over time, TWFE is right on the truth (0.49 against 0.50). Make the identical effect grow with exposure and TWFE already misses (1.06 against 1.50). Let the cohorts differ, and it misses by an amount that changes with the pattern: 1.00 against 1.60 when early adopters have the larger effect, 0.91 against 1.10 when late adopters do.

The estimand is itself a choice worth naming. Weighting cohorts by how many treated periods they contribute gives 1.60; weighting each cohort's effect equally gives 1.35, because the two cohort effects are 2.1 and 0.6. That 19% gap reflects a decision about what average to report; the group-time estimator recovers whichever one it is asked for.

What the fix does not solve

Clean controls remove the negative-weight contamination. They do not establish that the design identifies an effect at all. This is the limit of the planted-truth check: it cannot catch a parallel-trends or anticipation violation. The effect here is zero before adoption by construction, so a pre-trend test passes; a real anticipatory response, units changing behavior before treatment starts, would bias both the pre-trend test and the estimate, and no in-sample number separates it from a clean case. The fix does not escape this: the not-yet-treated units used as clean controls can themselves anticipate their coming treatment, which contaminates the very comparison the group-time estimator relies on. That is the assumption to argue from institutional knowledge, not to read off a regression.

What to prioritize, before the coefficient

Four checks matter more than the number TWFE prints, in order.

1. With staggered timing, do not read one TWFE coefficient as the effect. Ask which comparisons it averages, and whether any of them use already-treated units as controls.
2. Use a group-time or clean-control estimator (Callaway-Sant'Anna, Sun-Abraham) that only ever compares against not-yet-treated units.
3. Name the estimand. A cohort-time average can be weighted more than one way; state which average the reported number is.
4. Argue no-anticipation and parallel trends on the substance. A passing pre-trend test is necessary, not sufficient; it cannot rule out anticipation, which biases the test itself.

The coefficient is the output, read alongside its estimand and its identifying assumptions; the finding is whether the comparisons are clean and the design holds.

Reproduction

The full simulation framework, including the staggered-adoption data generation, the static TWFE and group-time estimators, the four-scenario sweep, the two estimands, and the figure scripts, is at github.com/dphdame/tooearlytosay-analysis. The setup uses simulated data with a fixed seed, so results reproduce exactly. Changing which cohort has the larger effect changes the size of the TWFE bias (from about −0.60 to −0.19 across the two cohort patterns shown) while the group-time estimator stays on the truth.