Using difference-in-differences in practice

When we think about using difference-in-differences (DiD), the first question is whether it is the right tool for the comparison in front of us, and where its conclusions stop being reliable.

Three linked analyses and one hypothetical COVID-19 income comparison anchor that question. The hypothetical comparison is a boundary case that shows when the causal design is missing. The analyst owns the estimation. The agent drafts the implementation and runs the specified diagnostics.

This is written for applied researchers who know the DiD formula but want a clearer rule for when to reach for it. A familiar failure mode: a two-group, before-and-after comparison is easy to run and easy to over-trust. The design choice that determines whether the estimate has a causal interpretation often gets made by habit rather than explicitly.

DiD supports a causal interpretation only if the comparison group represents what the treated group would have done in the absence of treatment. Each assumption below tests that claim, and each failure mode identifies the consequence.

When DiD is the right tool

DiD answers a specific question: how much did an outcome change for a treated group, relative to a comparison group, over the same period?

The method does not require treated and comparison groups to have the same outcome level. It does require a defensible counterfactual trend assumption. Whether that assumption is weaker or more plausible depends on the setting.

Consider a hypothetical COVID-19 income comparison in which the shock affects all counties. Comparing income changes in high-inequality metros with changes in low-inequality areas would describe how the two groups moved differently during the pandemic. It would not identify a causal COVID effect because neither group supplies an untreated or later-treated counterfactual. The comparison could motivate a heterogeneity hypothesis, but it would not be a causal DiD design.

DiD fits when three conditions hold:

• The treatment timing is clearly defined for a specific group.
• A valid untreated or later-treated comparison group is available.
• The groups track each other in the pre-treatment period.

When any of these fail, the sections below describe what goes wrong.

The assumptions as decisions

Each DiD assumption is a decision made before trusting the estimate.

Parallel trends. The comparison group stands in for the treated group’s untreated path. The empirical check is whether pre-treatment trends move together. The harder question is sensitivity: how much would small deviations change the estimate?

No anticipation. The treated group does not change behavior in advance of treatment. If a policy is announced well before implementation, the “pre” period may already reflect treatment effects.

Stable composition. The groups represent the same underlying units over time, rather than a changing mix. In the hypothetical COVID income comparison, defining units at the labor-market level would make the estimate reflect changes in place-level income distributions, not individual income trajectories.

Failure modes

Below are the main ways DiD stops being credible, each shown in a Too Early To Say case.

Weak power in pre-trends tests

A high p-value in a parallel-trends test does not rule out meaningful violations. In the SNAP sensitivity case, pre-treatment event-study coefficients are statistically indistinguishable from zero, yet the main effect remains significant only under small deviations from parallel trends.

Rambachan and Roth define several sensitivity restrictions, and their tuning parameters must not be conflated. Under the relative-magnitudes restriction, M-bar bounds post-treatment period-to-period changes in the untreated differential trend relative to the largest pre-treatment change. Under the smoothness restriction, M is an absolute bound on changes in that trend's slope; M = 0 imposes a linear differential trend, not exact parallel trends. The linked case article reports a relative-magnitude breakdown value written there as 0.35. Its event-study estimates, covariance matrix, target contrast, confidence-set method, and output are not public, so this page cannot verify the number or interpret it more precisely than the claimed specification. If the reported value is truly M-bar = 0.35, the claimed interpretation is that the robust confidence set first includes zero when allowed post-treatment changes reach 35 percent of the pre-period benchmark. It is an article-reported sensitivity result, not a universal threshold.

A second case shows the same issue from the failing side. In the SNAP BBCE replication, a two-way fixed effects model on a 51-state panel estimates a +1.37 percentage point effect on take-up, a 3.35 percent increase over a 0.410 baseline, with p = 0.18. The event study fails the parallel-trends check on one pre-treatment lead. Rather than discarding the design, we examine what remains once that violation is taken seriously.

Staggered adoption and small treated samples

When treatment timing varies across units, standard two-way fixed effects can combine comparisons that do not identify the desired effect under heterogeneous treatment effects. With very few clusters, conventional cluster-robust approximations can also be unreliable; the direction of the error is not guaranteed in every design.

The linked rolling-DiD page reports a nonpublic hard-case simulation with 3 treated units, 3 controls, and a planted effect of 1.0. The reported estimates are 2.061 after demeaning, 1.127 after detrending, and 1.525 after detrending plus seasonal adjustment. Those values show claimed transformation sensitivity, not that one specification is universally standard or that detrending validates the estimator. The page separately uses one treated state and three controls as a policy example. Combining that example with the simulation would incorrectly claim that a one-treated-unit run recovered the truth. The Python rebuild, data-generating process, seeds, and matched outputs are not public, so neither the exact values nor the proposed diagnosis has been independently reproduced.

Small-cluster inference needs assumptions, not a menu chosen because the sample is small. Randomization inference requires a defensible assignment or permutation mechanism and a stated null. An exact t procedure requires its own distributional structure, and HC3 is a leverage correction rather than a general cure for few clusters. Match the estimator and inference procedure to the treatment rollout, sampling process, and target before interpreting a coefficient.

Correct design, incorrect implementation

A valid design can still produce incorrect results through implementation errors. The Python implementation case documents a statsmodels event-study workflow and three common pitfalls:

• A post-treatment indicator that is collinear with year fixed effects and dropped from the model without any error.
• Standard errors clustered at the wrong level.
• Interaction terms built using the wrong reference period.

A simple safeguard is to test the code on simulated data with a known effect and confirm that the implementation recovers it before using real data.

A decision table

One rule per situation. Read across: the case in front of us, the tool to use, and the check that determines whether the result is credible.

If the situation is	Use	And check
One treated group, one comparison group, and a known treatment date	Canonical two-way fixed-effects DiD	That pre-treatment trends run in parallel
The parallel-trends test passes and the stakes are high	DiD with a pre-specified Rambachan-Roth restriction	The restriction, target contrast, confidence-set method, and breakdown parameter. The SNAP case reports M-bar = 0.35 under a claimed relative-magnitude interpretation, not a verified universal threshold.
The pre-trends test fails on a lead	A re-examination of the design rather than discarding it	Whether the violation is isolated or systematic (SNAP BBCE: +1.37 percentage points remains after one failing lead)
Units adopt treatment at different times	A heterogeneity-robust estimator matched to the comparison groups and target	Which cohorts serve as controls and which weighted effect is identified. Treat the linked rolling-lwdid rebuild as instructional until it is publicly reproduced.
The treated set includes only a few units	A design-specific estimator with inference justified from the assignment or sampling process	Whether any exact, randomization, or small-sample approximation assumptions are defensible; few units alone do not validate them
Moving from a textbook formula to implemented code	A statsmodels or linearmodels workflow	That a planted-effect simulation is recovered before applying the code to real data

The discipline

Across the three linked cases and the hypothetical boundary comparison, the same standard applies: state the assumption as a decision, validate the implementation with simulation, name the exact sensitivity restriction and target, and match the estimator and inference to the treatment rollout. A breakdown value is interpretable only with its restriction and confidence-set construction. Each linked case names its article; public code, output, and verification coverage are labeled at the study level.

• The parallel-trends sensitivity case reports a provisional Rambachan-Roth breakdown result. Its numerical inputs and output are not publicly reproduced.
• The SNAP BBCE TWFE-DiD replication shows what remains when the parallel-trends test fails on one lead.
• The rolling DiD (lwdid) case is an instructional account of an unreproduced simulation and Python rebuild. Its reported transformation comparisons are hypotheses to verify, not benchmarks.
• The difference-in-differences in Python walkthrough shows the statsmodels implementation and the diagnostics that separate signal from noise.