What Agents Actually Do (And What They Don’t)

A companion post proposes that specification precision and task tolerance can shape perceived AI capability. This article applies that hypothesis to agents that read project files, write code, and run analyses. Neither article reports a controlled comparison that isolates those factors from model capability.

The Multi-Language Audit Fallacy

Consider a common verification proposal: run the same estimator in R, Stata, and Python, then compare the outputs. Divergence is useful evidence that the implementations, defaults, samples, or numerical procedures are not aligned. Agreement is weaker evidence. Several implementations can faithfully execute the same misspecified design, so matching numbers are not a correctness oracle.

Software engineers call this pattern differential testing: run independent implementations on aligned inputs and investigate disagreements. It can detect syntax mistakes, default mismatches, and implementation defects. It is not a unit test against a known expected answer. Even after the data, covariates, and visible option flags are aligned, packages can differ in sample construction, nuisance estimators, fixed-effect handling, bootstrap algorithms, random-number streams, and numerical tolerances. The audit should pre-specify those choices and an agreement tolerance; it still verifies implementation consistency rather than the research design.

It stops helping with non-syntactic errors. An LLM might make up a function name in R that does not exist, and that mistake would not appear in Stata. Cross-language checking detects syntax errors like these. But when a prompt says “control for county characteristics” without listing which variables, the same ambiguous instruction goes to all three implementations. All three agents might select the same wrong variables, or different wrong variables. The ambiguity precedes implementation.

Consider a concrete case. A methods section says “include demographic controls.” Agent A in R selects population, median income, and percent nonwhite. Agent B in Stata selects population density, poverty rate, and unemployment. Agent C in Python selects a mix of both. All three pass syntax checks. All three produce different estimates. The multi-language audit reports disagreement and flags a “bug.” The real problem is that “demographic controls” was never defined. Three correct implementations of an ambiguous spec.

When one agent clusters standard errors at the state level and another at the county level because the methods section says “at the appropriate level,” disagreement identifies an unspecified choice. Differential testing can identify that ambiguity, but it cannot determine which implementation matches the intended design.

Packages can also diverge on default settings. The R did package and Stata csdid actually do differ: R defaults to multiplier bootstrap standard errors and simultaneous confidence bands, while Stata defaults to asymptotic standard errors and pointwise confidence intervals. They even use different doubly robust estimators (dripw in R, drimp in Stata). Same data, same method, different numbers by default. The fix is to read the package docs and align options explicitly. A documentation problem.

For deterministic code, the design is the verification problem. Execution is the easy part.

What Agents Actually Are

An agent is a process that reads project state (files, data, documentation), decides what to do next, takes action (writes code, runs commands, edits files), and observes results. Then it adjusts.

The distinction from a chatbot matters. A chatbot answers one question. An agent can read the data dictionary, notice a variable is coded differently than expected, write a cleaning step, run it, and check the output. The useful property is sequenced action informed by project context.

The implication is narrower: project documentation changes what task-specific information the agent can use. Pretraining can supply generic knowledge, but it cannot reliably recover undocumented project definitions such as a study’s cohort rules, variable construction, or intended clustering level.

We describe this setup in The Cold Start Problem. A CLAUDE.md file that carries project context makes more instructions available at session start and reduces the decisions left implicit. The output still requires verification. The model stays the same.

An agent’s behavior is bounded by the information, tools, model, and checks available in the run. Better project context can reduce ambiguity; it does not guarantee a correct result. That proposition identifies inputs to test rather than assigning all variation to the prompt.

When Agents Matter for Research

If agents are specification-bounded processes, which patterns matter for empirical research? Four design propositions organize the work documented on this site.

Context persistence

The framework predicts fewer unspecified choices when the agent receives variable conventions, the identification strategy, and the data structure. That is an input-design claim, not a measured model comparison.

A CLAUDE.md file that specifies “treatment is defined as the county’s first year above the 75th percentile of the distribution” removes that definition from the agent’s choice set. Without the definition, the implementation must supply one. We cover the mechanics in CLAUDE.md for Research Context; the linked article labels its savings estimates as independently unmeasured.

Phase-aware prompting

Exploration, implementation, and documentation are different tasks. They need different things from an agent.

During exploration (what does this data look like, where is missingness concentrated), we want the agent to identify unexpected patterns. During implementation (estimate this staggered DiD with these exact specifications), we want it to follow the spec precisely. During documentation (write up what was estimated and why), we want it to reference what was actually done instead of inventing a cleaner version of the analysis.

Phase-specific prompts constrain different choices in exploration, implementation, and documentation. We lay out that design in phase-specific prompting strategies. A comparative experiment would be required to estimate how much the prompt design changes output quality.

The verification tax

Agent-generated code needs verification against the specification and known truths. Successful execution establishes that code runs, not that it implements the intended design.

The linked methods-to-code article reports a 93% time reduction in one local example. No public timing record currently reproduces that comparison. The example illustrates the verification obligation: generation can be fast, while checking remains part of the work. We have written about the verification tax elsewhere.

A precise prompt names the estimator, control group, clustering level, and robustness checks, so fewer implementation choices remain implicit. That narrows the scope of review in principle. This article does not provide a comparative measurement of verification time.

Specification as the bottleneck

A well-written methodology section is the start of an implementation spec, but an estimator name alone is not nearly executable. A Callaway-Sant’Anna instruction must name the outcome column, unit and time columns, first-treatment cohort column, analysis sample, treatment reversals, anticipation window, comparison group, pre-treatment covariates and their timing, outcome-regression and propensity-score method, weights, missing-data rule, aggregation target, event-time window and reference period, clustering level, bootstrap method and repetitions, random seed, confidence-band type, and required diagnostics. A code-generation prompt should fill every field from the data dictionary and pre-analysis plan before an agent chooses a package or writes code.

A vague methods section (“we use difference-in-differences with appropriate controls”) produces vague code regardless of how capable the agent is. The agent cannot invent the research design. It cannot decide what “appropriate” means. The researcher must supply that judgment before implementation.

Fred Brooks made a version of this argument in 1986: the essential difficulty of software is deciding what to build, not the coding. The “accidental” complexity of syntax and compilation can be automated away. The “essential” complexity of specification cannot. Agents automate the accidental part faster than ever. The essential part remains.

The methods-to-code article attributes its local timing result to a precise methods section, but no public timing record tests that mechanism against a looser specification. Context window budgeting remains a practical constraint because the agent receives a bounded set of project material.

All four patterns support the same conclusion: the agent changes the allocation of work but does not eliminate research-design judgment.

The Researcher’s Role (For Now)

What remains human, and why?

Agents may shift effort from “how do I code this” to “what exactly should I measure and why.” Whether accepted output becomes faster after verification and rework is an empirical question.

Research design specifies what comparison identifies the causal effect, which assumptions are credible, and what the estimate means. An agent can propose or critique a parallel-trends argument using the information it receives, but it cannot own or defend the substantive judgment. Accountability for that decision remains with the researcher.

The prompt encodes part of the research judgment. Specifying the estimator, control group, clustering level, and robustness checks makes consequential decisions explicit. The agent can execute and challenge them; the researcher remains responsible for whether they answer the economic question.

This is why the copy-paste ceiling we wrote about earlier exists: at some point, the workflow demands more than copying model output into a script. It demands that we know what we want the script to do and why.

“AI will replace researchers” and “AI is useless for research” both assume raw model capability alone determines usefulness. Output quality also depends on the researcher-provided specification: precise requests constrain implementation choices, while vague requests leave choices unspecified.

What to Do Differently

Five changes that seem to matter, based on the patterns above.

1. Audit the specification, not the execution. When agent output is wrong, the first question should be: was the specification precise enough that two independent researchers would make the same implementation choices? If not, the fix is upstream of the agent.
2. Invest in project context. A CLAUDE.md file, clear variable naming, documented data dictionaries. These inputs affect output quality.
3. Match the prompting to the phase. Exploration prompts should invite surprises. Implementation prompts should constrain choices. Documentation prompts should reference what was actually done, not what we wish we had done.
4. Budget for verification. Plan the checks before implementation. The linked local timing example includes review but does not measure how specification precision changes review time.
5. Write the methods section first. If we cannot write a precise methods section, the agent cannot write precise code. The methods section is the spec. This is the same logic behind pre-analysis plans: locking the design before seeing results prevents loose choices from becoming researcher degrees of freedom. The same specification supports both the paper and the implementation.

Too Early to Say

Whether agents transform empirical research depends on model capability and on whether researchers specify precisely what they want. Too early to say how this applies across research settings.

For a concrete starting point, CLAUDE.md for Research Context covers project context, and From Methodology to Code documents the specification-to-implementation workflow.