The chore we keep doing by hand
Anyone who has tried to replicate a published econ paper, for a class or a referee report or a working-paper extension, has had some version of the same afternoon. We pull the package from openICPSR. We unzip it. We open the folder. Sometimes there is a README and a clean code/ directory and a master run_all.do. Sometimes there is a flat dump of a dozen scripts with names like 02_clean_v3_FINAL_use_this.do and no indication of which order they belong in. Sometimes the README says “Run in Stata 16” and our license is Stata 18 and we discover three deprecated commands by trial and error. Sometimes the scripts are full of hardcoded paths to /Users/jsmith/Desktop/project/ and we are not jsmith.
The AEA Data Editor publishes guidance for preparing replication packages. It requires a README with data provenance, software requirements, instructions, and expected outputs, and it requires the code used to produce the manuscript’s results. The guidance strongly encourages a runnable package with minimal intervention and offers an example organized as data/, code/, and results/. It does not require one master script, that exact directory layout, or a language lockfile in every package. What we have not seen is a continuously updated empirical picture of deposited-package features across the field, with measurement noise reported alongside each metric. The skill described here is a descriptive inventory, not an AEA compliance test.
Why AI?
We could try to build this with regular expressions (regex, fixed-pattern matchers) only, and the failure mode shows up fast. Stata users write seed calls in enough different ways that any strict pattern leaves cases on the table: set seed 12345, set seed`whatever’‘,set seed c(seed)', seeds set inside a wrapper program the regex never enters. R users have at least four common ways to connect a master script: a top-level run_all.R, a Makefile, or one of the workflow packages (targets, drake). Python projects mix runnable-as-a-script entry points with notebook-driven workflows and shell glue. READMEs are free text written by humans for humans, and “data are available from the authors on request” is a classification problem, not a regex problem.
The public v0.1 implementation uses filename patterns and text matching for source-tree features. Its one implemented LLM task is classification of the README’s data-availability statement; that classifier runs only when an Anthropic API key is present and otherwise records a no-op state. Ambiguous master-script detection and broader README classification remain design ideas, not implemented fallbacks. This boundary matters because pattern-only source inspection will miss legitimate variants and can produce false positives.
set seed 12345, so macro-based Stata seeds can be missed. The current master-script detector matches a defined filename list, including names such as run_all.R, Makefile, and main.py; it does not inspect targets plans, Python __main__ blocks, or multi-file execution logic. The separate data-availability classifier uses zero-shot Claude Sonnet with strict JSON output validated against its schema when credentials are available.
Getting our bearings
A short orientation. openICPSR is the Inter-university Consortium for Political and Social Research’s open archive and the deposit destination for replication packages at several AEA journals and other publications. Deposits are subject-coded; the planned crawl targets the economics subset. A well-documented package commonly includes a README, code, available data or access instructions, software requirements, run instructions, and expected outputs. A master entry point and pinned dependencies can improve execution, but they are not universal AEA requirements.
The v0.1 source-tree analyzer reads files statically with directory-name patterns, filename patterns, and regular expressions. It does not parse Python or R syntax trees. Static reading is cheap and reproducible, with the trade-off that it can misclassify unfamiliar conventions and cannot detect a bug that appears only at runtime. The output describes detected features, not verified reproducibility.
What the tool does
The proposed crawl frame covers openICPSR and AEA replication deposits from 2020 through 2026. The planned analytic sample is N=500 and the planned validation sample is N=30. Neither sample ships in the public repository.
The public v0.1 code currently accepts seed lines containing openICPSR project IDs and retrieves Wayback metadata. Direct-URL and local-zip full-package analysis are not implemented as a completed public workflow.
The per-package metrics come from inspecting files, not running them:
- Software stack detected (Stata / R / Python / Matlab / shell / mixed)
- Presence of directories matching the project’s code, data, and output name patterns
- README presence and word count
- Software-version mentions
- Runtime estimates from any documentation present
- Data-availability statement classification (whether data is public, restricted, or available on request)
- Master-script filename detection using a fixed pattern list
- Seed-setting prevalence per language (
set seed,set.seed,np.random.seed, etc.) - Hard-coded absolute path prevalence
- Dependency-manifest or lockfile presence; the current flag does not establish that every dependency is exactly pinned
- Lines of code per language
- Reproducibility-statement language in the README
New to Claude Code skills? The setup guide covers installation and first invocation.
Planned metric architecture. Path A reads Wayback-cached landing-page metadata. Authenticated Path B is a public stub, not a completed full-package implementation.
The framing is descriptive, not normative. The skill does not produce an overall compliance or reproducibility composite. Most outputs are individual fields, such as README presence, seed-call count, or software-version mention. The schema's folder_structure.layout_score is narrower: it counts how many of the three expected directories (code, data, and output) are present, so its range is 0 to 3. It is a layout-component count, not a quality judgment, and it is never combined with the other metrics. An overall “reproducibility score” would require weights that this project has not justified. The planned panel therefore reports each field separately, with measurement notes where a detector is noisy.
has_version_pins; for requirements.txt, its secondary field becomes true when it finds at least one exact == line, not only when every dependency is pinned. These are detector semantics, not certification of a reproducible environment. Lines-of-code counts call cloc. Master-script detection is filename-pattern only; there is no LLM fallback in v0.1.
Current Validation Status
The public repository lets readers inspect the Path A orchestrator, Wayback harvester, static analyzer, classifier wrapper, and output schema. It does not ship an automated test suite, default seed list, saved run output, or completed calibration file.
Earlier versions of this article reported a 10-record smoke run, 9 of 10 Wayback hits, and a 10-package calibration result. Those values are not backed by public seed or output artifacts at the pinned commit, so they are not presented as verified results.
Walking through a run
The following JSON is an illustrative target record, not an observed package output. With a user-supplied seed, the v0.1 orchestrator is designed to emit a record conforming to its public schema.
{
"package_id": "openicpsr-123456",
"primary_language": "stata",
"language_share": {"stata": 0.78, "python": 0.22},
"folder_structure": {
"has_code_dir": true, "has_data_dir": true,
"has_output_dir": false, "layout_score": 2
},
"readme": {
"present": true, "word_count": 612,
"das_class": "public_with_scripts", "das_class_confidence": 0.91,
"has_software_versions": true, "has_runtime_estimate": false
},
"code_metrics": {
"loc_total": 4820, "has_master_script": true,
"master_script_path": "run_all.do",
"seed_set_count": 2, "hardcoded_paths_count": 11
},
"dependencies": {"has_version_pins": false, "version_pin_files": []},
"measurement_notes": {
"stata_regex_uncertainty": "+/-15% on seed detection (95% CI from N=10 Stata validation)"
}
}
The following CSV is also a hypothetical format example. No public panel currently contains these rows:
doi,year,journal,software_stack,has_readme,has_master_script,sets_seed,n_loc_total
10.1257/app.20200xxxx,2024,AEJ:Applied,stata,1,1,1,4823
10.1257/aer.20210xxxx,2025,AER,r,1,0,1,1247
10.3886/E199xxx,2023,openICPSR,python,1,1,0,8910
10.3886/E201xxx,2024,openICPSR,mixed,0,0,1,2104
In the planned panel, each row is one package and each column is one metric. The design deliberately avoids an overall compliance composite. The 0-to-3 layout_score remains a transparent count of three directory indicators and should be reported with those indicators, not interpreted as a package-wide score.
The N=500 panel and N=30 hand-coded validation are future releases. Until those artifacts exist, no package prevalence, detector agreement rate, or uncertainty band should be inferred from the format examples.
The same template extends to the other noisy metrics. Master-script detection by filename match is a heuristic, not a guarantee: a package that connects its analysis through a Makefile we did not pattern-match against will read as having no master script when in fact it has a perfectly serviceable one. Hard-coded path detection is somewhat better, because the path-like string itself is a tighter regex target, but cross-platform variation (Windows backslashes, mixed forward and back, tilde-expanded home directories) still leaves room for miscounts. The measurement_notes field on every JSON record is where these caveats live, per-package. That way the noise is not a footnote in the landscape post; it travels with the data.
measurement_notes fields.
The openICPSR Cloudflare problem and Path A / Path B
Descriptive measurement only works if the packages are actually reachable, and the build pass turned up an access constraint the spec round had not flagged: openICPSR sits behind Cloudflare, which can block unauthenticated automated requests. Path A can inspect only metadata exposed in cached landing pages. It cannot establish source-tree metrics such as code contents, seed calls, hardcoded paths, or runnable master scripts. Exact metric coverage must be measured from a released seed and run; v0.1 does not ship those artifacts.
The v0.1 repository implements a Wayback-based Path A and includes only a stub for authenticated Path B. Path A also requires a user-supplied seed file. Full source-tree metrics should not be claimed until Path B is implemented and tested.
Next steps
The next release needs to add the missing smoke seed file and automated fixture tests, publish a frozen Path A run with logs and per-package JSON, then release the N=30 validation sample. Only after those gates clear should the N=500 landscape crawl and public panel be described as results.
Worth taking with us
The current result is a public code skeleton and measurement design, not a continuously updated empirical landscape. Its useful idea is descriptive, field-level reporting with uncertainty rather than an overall normative composite. The seed, test, output, validation, and panel releases remain the work required to turn that design into a public result.
Have input? Get in touch.
Cite this article
Cholette, V. (2026, May 25). A common shape for econ replication packages. Too Early To Say. https://tooearlytosay.com/research/methodology/replication-package-analytics/