Validating a Double Machine Learning Estimate

Before we report a causal effect that runs a machine-learning model inside the estimator, we can plant a known effect in simulated data and check whether the code recovers it. The check is the same planted-truth routine we use for any reimplementation:¹ specify a data-generating process we control, fix the true value of the parameter, and require the implementation to return it. What is new here is the estimator, and one way of building it that runs without error and returns the wrong number.

The wrong number is attenuated. On a design with a planted treatment effect of 1.0, a naive plug-in that regresses the outcome on a machine-learning prediction of the controls returns about 0.55, a little more than half the truth.² The code imports, runs, and hands back a coefficient in a reasonable range. A funder weighing a program whose benefit per unit of treatment has to reach 1.0 to pass a cost-benefit test would read 0.55 and defund a program that, in the simulation, meets the threshold.

What double machine learning is

The setting is the partially linear model. We want the causal effect of a treatment D on an outcome Y, holding a set of controls X fixed:

Y = θ·D + g(X) + ε, and D = m(X) + ν.

Here θ is the one number we care about. The two functions g(X) and m(X) are nuisances: they describe how the controls shape the outcome and the treatment, and they can be high-dimensional and nonlinear. Double machine learning (DML), due to Chernozhukov and coauthors, lets a flexible machine-learning model estimate those nuisances while permitting root-n inference for θ under the paper's identification, moment, overlap, sampling, and nuisance-rate conditions.³ This is an asymptotic result, not a finite-sample guarantee that every DML estimate is unbiased or every standard error is valid. The method exists because the obvious shortcut fails: dropping a regularized ML fit of g(X) straight into an outcome regression lets the model's bias leak into θ.

Two ingredients control regularization and overfitting bias. The first is orthogonalization: residualize both Y and D on X and regress one residual on the other, so a small error in the nuisance estimate does not move the target moment to first order at the truth. This is the partialling-out logic of Robinson's semiparametric regression.⁴ The second is cross-fitting: estimate the nuisances on one fold of the data and form the residuals on a held-out fold, so an observation is not used to train its own nuisance prediction. Cross-fitting reduces own-observation overfit bias; it does not make fitted values and residuals independent in a finite sample. The naive plug-in skips both. It fits g(X) in-sample and never residualizes D, so the regularized fit absorbs treatment-linked variation in X and attenuates θ toward zero.

Check it against a planted effect

The tool uses the generic interface the companion piece builds:¹ verify_estimator plants a known effect, runs the estimator across many simulated draws, and reports whether the mean recovers the truth. The pinned DML package extends that helper with worker support, Monte Carlo standard error and standard deviation fields, and the underlying estimates. The abridged release sequence below imports the physical helper used by the runner. The data-generating process confounds D through the same high-dimensional X that drives Y, so the nuisances are real work, and the planted effect is 1.0.

Planting the effect is concrete: we choose the number, here 1.0, and set it as θ, the coefficient on D in the line that generates Y (Y = θ·D + g(X) + ε above). The estimator is not given that number; a correct one has to recover it from the simulated data alone. In the code below, that number is the third argument to verify_estimator.

import os

from verify_estimator import verify_estimator
from dml_plm import (
    simulate, simulate_omitted_confounder, naive_plugin, dml_plm
)

workers = min(4, os.cpu_count() or 1)

# Plant 1.0 and run 50 independent draws. The 0.10 tolerance is an
# engineering smoke-test rule, not a hypothesis test or coverage claim.
negative_control = verify_estimator(
    naive_plugin, simulate, 1.0, tol=0.10, reps=50, workers=workers
)
candidate = verify_estimator(
    dml_plm, simulate, 1.0, tol=0.10, reps=50, workers=workers
)
omitted_control = verify_estimator(
    dml_plm, simulate_omitted_confounder, 1.0,
    tol=0.10, reps=50, workers=workers
)

# Any unexpected status stops the package runner with a nonzero exit.
assert not negative_control["passed"]
assert candidate["passed"]
assert not omitted_control["passed"]

# naive ML plug-in       planted 1.00 | recovered 0.563 +/- 0.003 (MC SE) | bias -0.437 | FAIL
# double ML (cross-fit)  planted 1.00 | recovered 0.965 +/- 0.005 (MC SE) | bias -0.035 | PASS
# double ML, U omitted  planted 1.00 | recovered 1.465 | bias +0.465 | FAIL

The naive plug-in and cross-fitted DML part ways against the same planted truth. On a single reference draw the split is already visible: the naive plug-in returns 0.55 while DML returns 0.97, with a nominal 95% confidence interval of [0.90, 1.03] that covers the planted 1.0.² One interval covering the truth is an example, not a coverage study. Across the 50 draws, the naive plug-in's mean attenuation is large relative to its Monte Carlo standard error.

Estimator (planted effect = 1.0)	Recovered (50-draw mean)	What the check showed
Naive ML plug-in (in-sample, no orthogonalization)	0.56	outside the declared 0.10 engineering tolerance
Double ML (cross-fitted, orthogonal)	0.97	within the declared 0.10 engineering tolerance

A run of it looks exactly like a run of the correct estimator, which is the whole reason to plant a truth and check recovery rather than eyeball whether the coefficient looks reasonable.

Where the check stops

The check has two limits, and stating them plainly is part of using it.

The first is that “PASS” means only that the 50-draw mean falls inside a declared engineering tolerance. DML returns 0.965, not 1.0. Its mean bias of -0.035 is about seven times the reported Monte Carlo standard error of 0.005, so it should not be dismissed as Monte Carlo noise. It may reflect finite-sample bias in this design, but the current run does not identify the source or show that it shrinks with sample size. The single reference interval covers the truth, but one covered interval cannot validate nominal coverage. A stronger validation would report mean bias, the Monte Carlo standard error of that bias, empirical interval coverage over many draws, and a pre-specified sample-size sweep. The current result establishes only that this implementation is within 0.10 for this data-generating process and sample size; it does not establish finite-sample unbiasedness or consistency.

The second limit is the one that matters for a policy number. The check evaluates within-tolerance recovery given the controls we simulate. It says nothing about whether the controls we use on the real data contain every confounder. Suppose a variable U moves both the treatment and the outcome and never enters X. Run DML on that world, with U omitted from the controls, and the estimate is biased for the true effect: a mean of about 1.46 against a planted 1.0.² When we build the simulation, we choose what goes into X, and we naturally build the same control set we plan to use, so our simulated X omits U exactly as our real analysis does. The simulation never generates the confounding, the check reports clean recovery, and the omitted confounder goes undetected. DML controls first-order sensitivity to regularization error under its conditions; it does not manufacture the identifying assumption that X is complete. That assumption still has to be defended on its own, from how the treatment was assigned.

The reproduction

The generic verify_estimator helper, the DML and naive estimators under test, and the worked check that produced the numbers above are at pinned public package. Clone it, plant an effect, and the check reports whether each estimator recovers it. The package physically supplies the extended verify_estimator.py implementation imported by its runner. The runner requires the intentionally naive negative control to fail, cross-fitted DML to pass, and the omitted-confounder negative control to fail; any unexpected status exits nonzero.

The orthogonal moment, in more detail

DML estimates θ from a moment condition built on the two residuals, Y - E[Y|X] and D - E[D|X]. That moment is Neyman-orthogonal: its derivative with respect to the nuisance functions is zero at the truth, so a first-order error in the machine-learning estimates of E[Y|X] and E[D|X] does not propagate into θ. The naive plug-in uses a moment that is not orthogonal, so its nuisance error enters θ directly.

Orthogonality removes first-order sensitivity of the population moment at the truth, but finite-sample remainder terms remain. Cross-fitting estimates each observation's nuisance value on a fold that excludes it, limiting own-observation overfit without asserting literal independence between the fitted value and residual. Under the identification, regularity, and nuisance product-rate conditions, the combination supports asymptotic root-n inference and standard-error formulas. Those properties must be checked for the design at hand rather than assumed from the label DML. The formal statement, including the conditions on the nuisance estimators, is in Chernozhukov and coauthors.³ The planted-truth check is the applied shadow of that theory: it documents performance against a declared tolerance on a process we control. That is useful for implementation testing, but the omitted-confounder case shows that it does not validate identification.

Closing

Use this release checklist whenever a machine-learning model enters an estimator:

1. State the estimand, identifying assumptions, nuisance models, folds, and engineering tolerance before running the code.
2. Plant a known truth and include an intentionally broken negative control.
3. Require the negative control to fail and the candidate to pass; propagate an unexpected status to a nonzero process exit.
4. Report bias, Monte Carlo uncertainty, interval coverage, and a sample-size sweep separately, then defend the real-data control set and overlap from the study design.

A planted-truth gate tests implementation on the process we chose. It does not validate the identifying assumptions in the policy data.

Notes

1. Cholette, V. (2026, June 17). How do we know an AI's estimator does what we meant? Too Early To Say. https://tooearlytosay.com/research/methodology/validate-ai-econometric-code/ The companion article defines the verify_estimator interface and planted-truth routine. This package reuses that interface and extends the helper with worker support and additional Monte Carlo output fields.
2. Numbers are from seeded, rerunnable reproduction code (pinned public package, validate-double-ml). Against a planted true effect of 1.0 on the reference seed, the naive ML plug-in recovers 0.554 (bias -0.446) and cross-fitted double ML recovers 0.968 with a nominal 95% confidence interval of [0.904, 1.032]. Averaged over 50 independent draws the naive plug-in recovers a mean of 0.563 (Monte Carlo SE 0.003) and double ML a mean of 0.965 (Monte Carlo SE 0.005). On a data-generating process with a confounder omitted from the controls, double ML recovers a mean of 1.465 (bias +0.465) against the planted 1.0.
3. Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., & Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1), C1–C68. https://doi.org/10.1111/ectj.12097
4. Robinson, P. M. (1988). Root-N-consistent semiparametric regression. Econometrica, 56(4), 931–954. https://doi.org/10.2307/1912705