Automating Peer Review Solves Manuscript Coherence and Opens the Door to Code Verification

Automating manuscript peer review solves the coherence check and opens the door to code verification. In a recent benchmark of 134 compensated referee reports across 124 social science manuscripts, a structured referee prompt on Claude Opus 4.8 beat the human report 96.3% of the time under a blinded judge.¹ Three former journal editors preferred the AI report 82.1% of the time. It also eliminated the faculty-rank gap. Human referees averaged 5.8 out of 10 when reviewing assistant professors and 6.4 when reviewing full professors, whereas the AI reviewer scored 8.6 to 8.7 across every rank.

From a computer science perspective, that result clarifies what traditional peer review actually measures. When a journal sends a manuscript PDF to a human referee, it asks the referee to act as a coherence engine. The referee checks whether the causal claims match the research design, whether the standard errors fit the clustering structure, and whether the tables align with the prose. A frontier language model performs that coherence audit more thoroughly, more consistently, and more fairly than time-constrained human readers.

Once a journal automates that initial read of a manuscript PDF, two practical questions follow. First, how should journals handle revisions when authors can run the referee prompt at home before submitting? Second, what happens to upstream data and coding errors that never appear on the PDF page?

Answering both questions requires a controlled setting with known errors. We need to watch how the referee prompt reacts as a paper is revised. Suppose we take three empirical Medicaid working papers that each contain two verified flaws in their early drafts. The first is an on-page specification flaw that creates a mathematical tension inside the PDF, and the second is an upstream code-to-table bug that never contradicts the prose. After the AI referee critiques the initial draft, an author can revise in one of three ways: repair the underlying estimator and code, soften the prose and add citations while leaving the tables untouched, or submit that same caveated draft alongside the initial AI review and a defensive author response letter.

Across the 12 resulting referee reports and 84 blinded judge audits, a clear pattern emerges. On the initial drafts, the prompt replicates its 8.69 out of 10 benchmark quality with 95.3% factual accuracy across 363 verifiable claims. On revisions, a fresh re-run on hedged prose softens its demands because the surface tension on the page is muted. Attaching the initial review and an author response letter puts the disagreement back on the page and restores the re-estimation requirement. And because neither a human referee nor a PDF-only model can see upstream code bugs that do not contradict the prose (0 of 3), automating the PDF review creates the exact opening journals need to augment peer review with a deterministic verification engine.

How the Four Revision Paths Test Journal Policy

Journals adopting AI review face a basic design choice. A journal can publish its referee prompt so authors can run it privately and revise their draft before submitting, or it can require authors to submit the initial AI review alongside their revised manuscript and an author response letter explaining any remaining disagreements.

When an author runs a public referee prompt at home and receives methodological criticisms, there are a few reasonable responses.

• Repair the underlying estimator and data script. Re-estimate the model with the appropriate clustering or population weights and update the affected tables.
• Polish the prose and expand the limitations section. Replace causal verbs with associative language and cite recent econometric literature while keeping the original point estimates and standard errors in the tables.
• Submit the initial review and an author response letter. Pair the caveated draft with a response memo arguing that the prose caveats make re-estimating the tables unnecessary.

Each of our three Medicaid papers pairs an on-page specification tension with an upstream code or table bug.

↑ Back to top

Baseline Replication: Confirming the 8.6 Benchmark and 95.3% Factual Precision

In the 134-paper benchmark, the tuned AI reviewer averaged 8.6 out of 10 on the blinded rating scale. Across our three unrevised baseline manuscripts, the Claude Opus 4.8 referee reports score 8.86, 8.60, and 8.60 (mean = 8.69).

A separate claim-verification audit extracts every concrete factual statement the referee makes about the manuscript and checks it against the text. Across all 12 reviews, the referee achieves 95.3% factual accuracy across 363 verifiable claims (346 true, 17 false), checking equation subscripts, verifying degrees of freedom, and auditing table columns line by line at a depth that few human referees have time to provide.

↑ Back to top

Finding 1: Why Stateful Rebuttal Audits Outperform Private Stateless Re-Runs

In Case 1, the unrevised draft pairs a strong causal claim with a 5.3-fold understated standard error (0.030 vs. 0.1583) and a failed within-cohort permutation test (p = 0.3913). Because the bold claim clashes directly with the small sample of six treated states in the regression table, the AI referee issues a “rethink” verdict (8.86 out of 10) and demands that the authors re-estimate the model.

Comparing the two prose-hedging conditions on the exact same revised manuscript text and tables shows why the journal workflow matters.

• Stateless Re-Run on Hedged Prose. Suppose an author leaves the 0.030 standard error and p = 0.3913 permutation result untouched in the regression table, softens the causal verbs, and adds a limitations section citing staggered difference-in-differences and parallel-trends sensitivity estimators.⁴ When a fresh, memoryless instance of the AI referee reads the revised draft with no record of the initial review, the surface tension between the prose and the table is muted. Just as a human referee reading a caveated draft for the first time is often more forgiving when limitations are acknowledged upfront, the stateless AI run softens its verdict from “rethink” to “major revisions”. It also downgrades the permutation critique from requiring re-estimation to merely reframing the text.
• Stateful Review with the Initial Audit and Author Response Letter. Now compare what happens when the initial AI review and the author’s response memo travel with the paper. The response memo puts the analytical disagreement directly in front of the coherence engine. That allows the model to compare what the first review asked for against what the authors actually changed in the tables. Across all three papers, the referee responds by cross-examining the memo against the tables. In Case 1, it restores the re-estimation requirement on both flaws and returns its verdict to “major revisions (bordering on rethink)”. In Case 3, it uses the memo’s admissions to escalate the cohort-trend critique from a minor observation to a mandatory re-estimation requirement.

That contrast gives journal editors a practical rule. Publishing a prompt for private use alone helps honest authors improve their drafts, but allows strategic authors to iterate until prose caveats soften a fresh review. Requiring the initial AI review and an author response inside the journal’s submission portal turns the coherence engine into a cross-examiner that holds out for true re-estimation.

Pairwise judging between the true data fix and prose hedging confirms the value of re-estimation. The blinded judge prefers the review of the repaired estimator over the review of the hedged draft 100% of the time in Case 1 and Case 3. Once the main specification is fixed, the AI referee stops policing causal adjectives and helps the author catch finer arithmetic discrepancies on the page, such as noticing in Case 3 that the text refers to “34 ACA expansion states” while (51 − 18) = 33 states appear in the subgroup column (8.03 − 6.38 = 1.65 pp, not 1.66 pp).

↑ Back to top

Finding 2: From Automating Coherence to Augmenting Verification

Whenever an empirical flaw creates a mathematical contradiction inside the PDF, the AI referee catches it reliably. In Case 2, across all three unfixed versions of the manuscript, the referee catches the 10:1 base-rate distortion 100% of the time because both the 9.09% downsampled test prevalence and the $10.9 billion national savings projection (0.303% population prevalence) are printed in the text.

When a coding or table error does not contradict the prose, however, a PDF-only review catches it 0% of the time.

• Variable-definition mismatches in upstream scripts (Case 3 Flaw 1). In the initial draft of Case 3, the manuscript states that the numerator and denominator measure adult Medicaid enrollment, while the data script sums adult and child Medicaid enrollment in the numerator. Because the prose never mentions child enrollment, the baseline review cannot see the bug from the PDF.
• Sample-window coding mismatches (Case 3 Flaw 2). The baseline text in Case 3 states that eight states switched Marketplace branding during the 2013–2024 sample window, when the policy coding file included seven states that adopted by 2014 (leaving only Kentucky in 2017). Without access to the state coding file, a PDF reader cannot see that seven of the eight switches sit at the boundary of the panel.
• Model-to-table transcription swaps (Case 2 Flaw 2). In Case 2, the out-of-time validation table swaps the Random Forest (0.7932) and XGBoost (0.8226) AUCs relative to the raw model output file. Because both numbers are plausible AUC values, a PDF reader has no way to detect the swap (0 of 3).

Is that a weakness of the AI prompt compared to a human referee? Not at all. A human referee reading a manuscript PDF in ScholarOne or Editorial Manager has the exact same blind spot, because neither human referees nor PDF-only prompts execute the underlying code. Large-scale empirical audits show how often those errors slip past human peer review. Across 30,717 published articles and more than 250,000 test statistics in eight journals, 49.6% of human-reviewed papers contained at least one reporting inconsistency where the printed test statistic and degrees of freedom did not match the printed p-value, and 12.5% contained a gross error large enough to flip statistical significance.⁸ Similarly, when economists attempted to reproduce 67 published macroeconomics papers across 13 journals using the authors’ own code and data, only 49% reproduced the published tables even with author assistance (33% without assistance).⁹ That 51% non-reproduction rate is why the American Economic Association established a dedicated Data Editor office to verify code-to-table reproducibility after conditional acceptance.¹⁰

This is where automating the traditional referee report nudges empirical science from automation to augmentation. Historically, journals could only afford to run computational reproducibility checks at the very end of peer review (after human editors and referees had already spent months reading the PDF), because human review consumed nearly all editorial bandwidth. Once a language-model coherence engine automates baseline manuscript review at near-zero marginal cost, journals no longer have to wait until conditional acceptance to verify whether the numbers in the tables match the code.

↑ Back to top

How Journals and Authors Can Pair Both Engines

The equity and quality gains from adopting automated manuscript review are immediate. Beyond removing the 0.6-point rank penalty that human referees impose on assistant professors, standardized review addresses well-documented gender disparities in peer review. Longitudinal evidence from economics journals shows that female economists face stricter writing standards during peer review and spend three to six additional months revising prose.⁶ Observational evidence across biomedical articles similarly shows that male authors more frequently frame findings with promotional terms (“novel,” “unprecedented”) that sway human readers.⁷ An equation-focused referee prompt that evaluates every submission on the same rubric and penalizes promotional overclaiming levels that playing field from day one.

We can pair the coherence engine with a lightweight verification engine at submission.

1. Engine 1: Automated Code-to-Table Verification. Alongside the manuscript PDF, authors or submission portals run a deterministic verification script. That script cross-checks every number in the manuscript’s LaTeX or Markdown tables against the raw JSON and CSV outputs of the estimation pipeline. Engine 1 enforces population base-rate rescaling on downsampled classifiers, verifies sample-flow arithmetic (51 − 18 = 33), and catches variable-definition or row swaps (0.7932 vs. 0.8226) before review begins.
2. Engine 2: Stateful Coherence Review with Author Response. Rather than relying on stateless private runs alone, journals run the referee prompt inside the submission portal. That way the editor and the Round 1 referee see the initial unvarnished audit alongside the author’s response letter. Adding one explicit rule to the prompt (stating that citing a robustness or clustering method in the limitations section does not lower the remedy requirement unless those estimates appear in the tables) ensures that prose caveats never substitute for empirical fixes.

↑ Back to top

References and Endnotes

1. King, G., & Worthington, S. (2026). AI article reviews dominate human peer reviews at their best: Evidence and policy choices. Working paper, Harvard University (October 25, 2026). https://garyking.org/peerless ↩
2. Callaway, B., & Sant'Anna, P. H. C. (2021). Difference-in-differences with multiple time periods. Journal of Econometrics, 225(2), 200–230. https://doi.org/10.1016/j.jeconom.2020.12.001 ↩
3. Rambachan, A., & Roth, J. (2023). A more credible approach to parallel trends. Review of Economic Studies, 90(5), 2555–2591. https://doi.org/10.1093/restud/rdad018 ↩
4. Roth, J. (2022). Pretest with caution: Event-study estimates after testing for parallel trends. American Economic Review: Insights, 4(3), 305–322. https://doi.org/10.1257/aeri.20210236 ↩
5. Bekker, J., & Davis, J. (2020). Learning from positive and unlabeled data: A survey. Machine Learning, 109(4), 719–760. https://doi.org/10.1007/s10994-020-05877-5 ↩
6. Hengel, E. (2022). Publishing while female: Are women held to higher standards? Evidence from peer review. The Economic Journal, 132(648), 2951–2991 (earlier version circulated as Econometrica working paper). https://doi.org/10.1093/ej/ueac032 ↩
7. Lerchenmueller, M. J., Sorenson, O., & Jena, A. B. (2019). Gender differences in how scientists present the importance of their research: Observational study. BMJ, 367, l6573. https://doi.org/10.1136/bmj.l6573 ↩
8. Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2016). The prevalence of statistical reporting errors in psychology (1985–2013). Behavior Research Methods, 48(4), 1205–1226. https://doi.org/10.3758/s13428-015-0664-2 ↩
9. Chang, A. C., & Li, P. (2017). A preanalysis plan to replicate sixty economics research papers that worked half of the time. American Economic Review, 107(5), 60–64. https://doi.org/10.1257/aer.p20171034 ↩
10. Vilhuber, L., & Cavanagh, J. (2025). Report of the AEA Data Editor. AEA Papers and Proceedings, 115, 944–957. https://doi.org/10.1257/pandp.115.944 ↩