When we ask a language model to audit an empirical paper, we face a practical budget question. An audit checks whether the scripts, tables, and prose all describe the same sample, weights, standard errors, and variables. Should we pay five times the token cost to run four passes in parallel? Should we split the files across specialized workers, or does one full-context pass already find the bugs?
We can test that question across 504 isolated audit calls on 18 econometric packets containing 45 planted flaws. One full-packet pass caught 99.0% of the planted flaws: 100.0% for claude-opus-5-5 and 97.9% for gpt-6.1-sol (98.6% weighting all 45 flaws equally). That single pass used 13,808 net tokens per packet ($0.098 in list-price equivalent). Running four full passes with a verifier used 74,513 net tokens, 5.40 times the single-pass mean ($0.560), and moved recall by +0.21 percentage points to 99.2%. Splitting each packet across parallel workers with a verifier used 50,651 net tokens ($0.545) and delivered 96.7% recall.
If one full pass already catches almost every planted bug, two practical questions follow. First, why do some multi-agent evaluations report large recall jumps from pooling four passes? Second, what changes when we split a research packet so each worker sees only two files at a time?
Consider what an econometric consistency audit looks like in our El Nino sea-surface temperature packet. Four quiet errors sit between the code and the text. The estimation script swaps the row labels for seasonal wave and seasonal phase in the regression table. The prose reports a coefficient of 0.392568, while the table prints 0.329568. Model 3 sets the Newey-West lag parameter to 12 in code, whereas the methods section claims 6. Finally, the estimation script overwrites the main sea-surface temperature series with an alternative column before fitting the regression.
Why does one full-packet pass catch these errors so reliably? Every mismatch lives between two pieces of the packet: the script does one thing, and the table or prose says another. If we give a frontier model all four pieces at once, it compares each table row directly against the code. When we hold the whole packet in view, Claude catches all four El Nino errors on every pass. The OpenAI model catches three of the four on every pass, missing only the swapped seasonal labels.
Benchmark Design and Sandboxed Execution
The benchmark pairs 18 self-contained econometric packets from authentic datasets in statsmodels, linearmodels, and scikit-learn.1 Each packet contains four pieces (prose findings, a summary table, a regression table, and sample and estimation scripts) across three specifications, spanning 1864 to 2087 words. Across 15 flawed packets, 45 deterministic mutations span 9 error categories (22 authored by Claude and 23 by the OpenAI model, with a 39-flaw subset removing six items whose specifications received cross-family review) alongside 3 unplanted control packets. Running 504 isolated audit calls (288 discovery calls and 216 verifier calls, split evenly at 252 Claude and 252 OpenAI calls) inside operating-system sandboxes with zero repository or shell access keeps each call independent.
How Keyword Checklists Inflate Measured Scaling Gains: The Vocabulary Lottery
Why do multi-pass evaluations so often report big recall gains when one full pass already catches 99.0%? Our benchmark shows how the grading rule itself can create that apparent scaling gain.
Suppose a model reads the El Nino packet and explains the input bug in plain English. It notes that the script replaces the main sea-surface temperature series with the alternative series before fitting the regression. A researcher reading that diagnosis can verify the bug directly. Now suppose an automated benchmark grades the report with a rigid keyword checklist. That checklist demands every listed phrase verbatim, including shorthand terms like "RHS" or "source column" that never appear in the packet. Under that exact-keyword checklist, single-pass recall drops from 99.0% to 50.8%. Pooling four independent passes jumps to 61.9%, an apparent gain of +11.11 percentage points (paired t = 3.94, Holm p = 0.004).
The underlying audit reports do not change between the two grading rules. Across eight full-packet passes per flaw, 13 of the 45 flaws pass the exact-keyword checklist on all eight draws. Another 14 pass on zero draws because no pass uses the exact checklist string. The remaining 18 flaws pass on some draws and fail on others, with pass rates from 12.5% to 87.5%. When one draw has probability p of hitting the checklist words, four draws raise the hit rate to 1 − (1 − p)⁴ > p. We call this mechanism the vocabulary lottery. If we grade an audit with a rigid word checklist, we pay five times the tokens for phrasing variety.
How do we measure error discovery rather than phrasing variation? We can score each report with pre-frozen grouped-term rules that accept natural synonyms for each part of a diagnosis, paired with blind dual-family semantic coding. Under grouped-term scoring, all 18 of those partially matched flaws score 8 out of 8 across the full-packet passes. Blind semantic coding raises the single-pass mean to 99.2%. In a post-result audit, both blinded coders tied 479 of 488 credited findings directly to the planted flaw. Of the 9 remaining items, 7 were valid catches in non-allocated slices and 2 accompanied a direct sign-reversal catch. If we grade by grouped terms and blind semantic review, we see that one pass already did the work.
Once we score by grouped terms, four full passes shift single-pass recall by +0.21 percentage points, and file-pair splitting moves four-pass recall by -2.50 points. Routing mixed-family slices through an opposing-family verifier alters the sliced mean by -1.67 points (0.00 against the stronger individual provider on the remaining 14 packets). All three Holm-adjusted p-values equal 0.493, and exact sign-flip p-values range from 0.500 to 1.000. Across setups, 87% to 100% of packets reach 100% recall, and 0% score zero. On Claude’s slices, raw discovery reached 100.0%, and its 96.7% primary score reflects a 120-word cap penalty on one call. On the OpenAI model’s slices, 96.7% reflects two missed column-label swaps. Crediting valid catches from any slice leaves delivered recall at 96.7% for the sliced mean and 96.7% for same-family verification versus 95.0% for cross-family verification (96.7% primary vs. 97.2% any-slice discovery across all six balanced slice assignments).
The Python walkthrough below computes the paired recall contrasts from the 15 packet vectors (scaled by denominators of 32 and 96). It contrasts grouped-term scoring against the exact-keyword checklist.
import numpy as np
from scipy import stats
# Packet-level recall across the 15 flawed packets (common denominators 32 and 96).
single_pass = np.array([32,32,32,32,32,32,32,32,31,32,28,32,32,32,32]) / 32
four_pass = np.array([32,32,32,32,32,32,32,32,32,32,28,32,32,32,32]) / 32
other_grouped = np.array([[0,0,0,0,0,0,0,0,-4,0,-8,0,0,0,0],[0,0,0,0,0,0,0,0,-4,0,-4,0,0,0,0]]) / 32
literal_single = np.array([76,64,45,48,48,90,84,48,18,32,48,12,44,42,33]) / 96
literal_four = np.array([96,80,48,48,60,96,96,72,36,32,48,48,48,48,36]) / 96
other_literal = np.array([[-32,0,-12,0,0,0,-32,-24,0,-16,-12,0,0,-48,-12],[0,-16,-12,0,12,0,0,0,-12,-16,-12,0,-16,0,0]]) / 96
for label, one, four, others, expected in [
("grouped_terms", single_pass, four_pass, other_grouped, [99.0, 99.2, 0.21, 1.00, 0.493]),
("exact_keywords", literal_single, literal_four, other_literal, [50.8, 61.9, 11.11, 3.94, 0.004])]:
differences = np.vstack((four - one, others))
se = differences.std(axis=1, ddof=1) / np.sqrt(15)
t_stat = differences.mean(axis=1) / se
p_raw = 2 * stats.t.sf(np.abs(t_stat), df=14)
order = np.argsort(p_raw)
holm_p = np.maximum.accumulate(p_raw[order] * [3, 2, 1])[np.argsort(order)]
summary = [round(100*one.mean(), 1), round(100*four.mean(), 1), round(100*differences[0].mean(), 2), round(t_stat[0], 2), round(holm_p[0], 3)]
assert np.allclose(summary, expected)
print(label, summary)
| Architecture | Recall: discovery → delivered | Packets at 100% | False alarms: discovery → delivered | Nitpicks: discovery → delivered | Net tokens |
|---|---|---|---|---|---|
| Claude · Single full-packet pass | 100.0% → 100.0%95% CI 100.0% to 100.0% | 15/15 | 0.43 → 0.43 | 1.26 → 1.26 | 16,615$0.166 list USD |
| Codex · Single full-packet pass | 97.9% → 97.9%95% CI 94.3% to 101.5% | 13/15 | 0.00 → 0.00 | 0.00 → 0.00 | 11,000$0.030 list USD |
| Two-family mean · Single full-packet pass | 99.0% → 99.0%95% CI 97.1% to 100.8% | 13/15 | 0.22 → 0.22 | 0.63 → 0.63 | 13,808$0.098 list USD |
| Claude · Four full passes + verification | 100.0% → 100.0%95% CI 100.0% to 100.0% | 15/15 | 1.72 → 0.61 | 5.06 → 0.28 | 92,681$0.953 list USD |
| Codex · Four full passes + verification | 98.3% → 98.3%95% CI 94.8% to 101.9% | 14/15 | 0.00 → 0.00 | 0.00 → 0.00 | 56,344$0.167 list USD |
| Two-family mean · Four full passes + verification | 99.2% → 99.2%95% CI 97.4% to 101.0% | 14/15 | 0.86 → 0.31 | 2.53 → 0.14 | 74,513$0.560 list USD |
| Claude · Four slices + verification | 96.7% → 96.7%95% CI 89.5% to 103.8% | 14/15 | 3.56 → 0.44 | 10.83 → 0.28 | 71,479$0.972 list USD |
| Codex · Four slices + verification | 96.7% → 96.7%95% CI 91.8% to 101.5% | 13/15 | 0.33 → 0.00 | 0.11 → 0.06 | 29,824$0.118 list USD |
| Two-family mean · Four slices + verification | 96.7% → 96.7%95% CI 91.1% to 102.2% | 13/15 | 1.94 → 0.22 | 5.47 → 0.17 | 50,651$0.545 list USD |
| 2+2 slices · Same-family verification | 96.7% → 96.7%95% CI 89.5% to 103.8% | 14/15 | 1.61 → 0.06 | 5.33 → 0.17 | 48,715$0.540 list USD |
| 2+2 slices · Cross-family verification | 96.7% → 95.0%95% CI 87.2% to 102.8% | 13/15 | 1.61 → 0.11 | 5.33 → 0.00 | 48,612$0.491 list USD |
Table 1 averages recall across the 15 flawed packets and averages false alarms, nitpicks, and token costs across all 18 packets. Token counts include verifiers when present and subtract empty-call overhead. Single-pass rows represent one discovery call, whereas multi-call rows pool four discovery calls before two verifier calls.
What Happens When We Split Context Across Workers
What happens when a workflow splits a packet into pieces so each worker sees only two files at a time? Teams split context when a project exceeds one window or when they want workers to inspect file pairs separately. In our sliced setups, four workers each read two of the packet’s four pieces in overlapping pairs. Together, those four pairs contain about twice the packet’s words.
Splitting files slightly lowers recall on bugs visible in only one pair. Across all 45 flaws, bugs visible in multiple pairs stayed at 100.0% recall. Single-pair bugs slipped from 98.2% in one full pass to 94.1% in the sliced mean and 91.2% with cross-family verification. Column-label swaps fell from 90.0% to 70.0% and 60.0%. Standard-error choices went 100.0%, 90.0%, 80.0%, and filter boundaries went 97.5%, 100.0%, 100.0%. The other six error categories stayed at 100.0%: sample-size discrepancies, weights and scale, regressor substitutions, text-table numbers, sign reversals, and input overwrites.
High single-pass recall also explains why pooling full passes barely changes discovery. Claude had zero misses across 270 within-family pair comparisons. Within the OpenAI model, the pair correlation φ reached 0.850 (95% CI [0.742, 1.000], from 6 joint misses, 3 discordant pairs, and 261 joint detections; family-specific flaw variance 0.0112, 95% CI [-0.00003, 0.0328]). On the 39-flaw strictly independent subset, Claude caught 100.0% of flaws from both author families. The OpenAI model caught 98.8% of Claude-authored flaws and 94.7% of its own.
The main cost of splitting context is not missed flaws, but false-alarm and nitpick noise. When a model audits a table and prose without seeing the data script, it still tries to write a thorough review. Whenever the prose leaves a filter or variable definition implicit, a worker without the script flags that gap as a warning. Without the data script in view, a thorough model turns every unanswered question in the prose into a false alarm or nitpick.
Comparing four full passes against four slices isolates the effect of splitting context at an equal four-call budget. Across the 72 full-packet calls per family, the OpenAI model reported 185 findings with zero false alarms or nitpicks, while Claude reported 335 findings including 31 false alarms (0.43 per single call) and 91 nitpicks (1.26 per single call). Under sliced discovery, those raw counts climbed to 64 and 195 for Anthropic versus 6 and 2 for the second provider (only 2 of 288 discovery runs hit the 12-item limit). Across all 18 packets, splitting context raised Claude’s false alarms 2.06-fold (from 1.72 to 3.56 per packet) and its nitpicks 2.14-fold (from 5.06 to 10.83). Surface nitpicks grew from 26.4% to 54.0% of Claude’s visible report tokens, while the OpenAI model moved from 0.00 to 0.33 false alarms and 0.00 to 0.11 nitpicks.
Separating the 15 flawed packets from the 3 unplanted controls clarifies that jump. On the 15 flawed packets, splitting context raised Claude’s false alarms 3.43-fold (from 0.93 to 3.20) and its nitpicks 3.73-fold (from 2.73 to 10.20). On the 3 unplanted controls, where no planted bugs anchor the review, Claude already produced high volume in four full passes (5.67 false alarms and 16.67 nitpicks vs. 5.33 and 14.00 on slices). Across all 950 raw reported findings (488 grouped-term hits, 6 semantic matches, 288 nitpicks, 101 false alarms, and 67 genuine baseline discrepancies: 33 on controls and 34 on flawed packets), blinded dual coding reached κ = 0.792 across three categories and κ = 0.785 on material versus minor findings (against κ = 0.359 and 0.395 for the wording tie-breaker). Even on the 22 findings that both coders independently judged false (out of 101, with 79 splits comprising 55 single-coder and 24 tie-breaker cases), Claude’s four-call total rose from 2 on full passes to 17 on slices.
How a Full-Context Verifier Earns Its Tokens
How do we stop split-context workers from filling audit reports with false alarms and formatting complaints? We place a full-context verifier behind the sliced workers. When the verifier sees the candidate findings alongside all four pieces of the packet, it checks whether the script already answers a table question. Across all 18 packets, full-context verification filtered out 87.5% of Claude’s sliced false alarms (from 3.56 down to 0.44 per packet). It also filtered out 97.4% of surface nitpicks (from 10.83 down to 0.28, cutting nitpick token share from 54.0% to 5.0%) while keeping 100% of discovered true flaws. On four full passes, verification filtered 64.5% of false alarms (from 1.72 to 0.61). If we give the verifier the full packet, we recover single-pass precision without losing a single true catch.
Does the verifier need to come from a second model family to filter out those false alarms? We can test that question by holding the mixed-family candidate pool fixed (two Claude slices plus two OpenAI slices per packet, starting with 1.61 false alarms and 5.33 nitpicks at 96.7% recall). Same-family verification kept every discovered flaw (96.7% recall) and filtered out 96.6% of false alarms (down to 0.06 per packet). Swapping to the opposing provider rejected 93.1% of unverified warnings (0.11 per packet), dropped one valid catch (95.0% accuracy), and cleared the final 0.17 minor formatting notes. All three paired differences sit near zero, with 95% confidence intervals spanning zero (-1.67 percentage points on recall, [-5.24, +1.91]; +0.056 on false alarms, [-0.062, +0.173]; -0.167 on nitpicks, [-0.357, +0.024]). What restores precision after splitting is giving the verifier the whole packet, not swapping model families.
Standard recall metrics miss the verifier’s real contribution because a verifier cannot add new discoveries. For a human researcher, however, every false alarm wastes time chasing a non-existent bug. We measure that trade-off with Net Audit Value, V = TP − w·FA, which subtracts w true-positive units per false alarm. When we pool two unverified full passes, additive false alarms cancel in I = V(both) − V(A) − V(B), leaving minus the count of shared catches (-3.000 for same-family Claude, -2.900 for same-family OpenAI, -2.917 for cross-family per flawed packet). If we pool unverified full passes near the recall ceiling, we pay twice to hear the same bugs.
import numpy as np
# Actual packet keys include two-, three-, and four-flaw packets.
planted = np.array([3,3,4,2,4,2,3,2,4,3,4,2,3,2,4])
assert planted.sum() == 45
a_detect = planted.astype(float) # Claude finds every full-packet flaw.
b_detect = np.array([3,3,4,2,4,2,3,2,3.75,3,3,2,3,2,4])
a_false = np.array([0,0,.75,.25,0,1.25,0,0,.75,0,0,0,.5,0,0])
b_false = np.zeros(15)
both_detect = b_detect # Average overlap over all cross-family worker pairs.
weights = np.array([0, .5, 1, 2])
for w in weights:
value_a, value_b = a_detect-w*a_false, b_detect-w*b_false
value_both = planted-w*(a_false+b_false) # Additive false alarms cancel.
interaction = value_both-value_a-value_b
assert np.allclose(interaction, -both_detect) # We recover the per-packet identity.
assert np.isclose(interaction.mean(), -2.9166666666666665)
# Rows: Codex slices, 2+2 same, 2+2 cross, two-family slice mean, Claude slices.
true_positive, false_alarm = np.array([43,43,42,43,43])/15, np.array([0,1,2,4,8])/15
value = true_positive[:, None]-false_alarm[:, None]*weights
assert np.allclose(value, [[43/15]*4, [43/15,17/6,2.8,41/15], [2.8,41/15,8/3,38/15], [43/15,41/15,2.6,7/3], [43/15,2.6,7/3,1.8]])
assert np.allclose(value[0], 2.8666666666666667)
print("Cross-family union interaction:", interaction.mean(), "\nVerified values:", value)Comparing net audit value and token cost clarifies when each setup makes sense. Among verified sliced setups in Table 2, the OpenAI four-slice audit achieves the highest net value at every positive penalty (2.867 per flawed packet) with zero delivered false alarms, 29,824 net tokens, and $0.118 list USD (versus $0.491 for cross-family slices at a 0.914 median token ratio to the sliced mean, and $0.972 for Claude slices; on four full passes, the OpenAI model catches 44 of 45 flaws, or 97.8% flaw-weighted vs. 98.3% packet mean, for V = 2.933 at $0.167). Within each model family, a single full-packet run beats every multi-call alternative: OpenAI reaches 97.9% recall and V = 2.917 at 11,000 net tokens ($0.030 list USD). Anthropic’s solo run hits 100.0% accuracy (scoring V = 3.000 at w = 0 and 2.533 at w = 2) while consuming 16,615 tokens ($0.166 unbilled cost).
| Architecture | w = 0.00 | w = 0.25 | w = 0.50 | w = 1.00 | w = 2.00 |
|---|---|---|---|---|---|
| Codex · Four slices + verification29,824 tokens · $0.118 list USD | 2.867 | 2.867 | 2.867 | 2.867 | 2.867 |
| 2+2 slices · Same-family verification48,715 tokens · $0.540 list USD | 2.867 | 2.850 | 2.833 | 2.800 | 2.733 |
| 2+2 slices · Cross-family verification48,612 tokens · $0.491 list USD | 2.800 | 2.767 | 2.733 | 2.667 | 2.533 |
| Two-family mean · Four slices + verification50,651 tokens · $0.545 list USD | 2.867 | 2.800 | 2.733 | 2.600 | 2.333 |
| Claude · Four slices + verification71,479 tokens · $0.972 list USD | 2.867 | 2.733 | 2.600 | 2.333 | 1.800 |
Grading rules also matter when surviving unverified warnings are rare. Across the 11 false alarms delivered by the sliced setups on flawed packets, 10 came from tie-breaker judgments on methodological commentary, and neither blinded coder pair marked them wrong. Under the strict dual-coder definition, the OpenAI, Claude, and same-family sliced setups tie at 2.867 across all penalty weights, and cross-family verification sits at 2.800.
Three Rules for Designing AI Code Audits
- Check whether one full-context pass already hits the ceiling before multiplying calls. Running four full passes with verification consumed 5.16 times (OpenAI) and 5.62 times (Claude) the median net tokens of a single pass in the same family while moving recall by just +0.21 percentage points. If we test one full pass on a small sample of planted bugs first, we can save multi-pass budgets for manuscripts that actually exceed a single window.
- Replace rigid keyword checklists with grouped terms and semantic coding. Demanding exact checklist phrases made four passes look +11.11 percentage points better when single-pass recall was already 99.0%. If we score benchmarks by pre-frozen synonym groups and blind semantic review, we avoid mistaking phrasing variety for new error discovery.
- Pair split-context workers with a full-context verifier. Splitting packets across workers multiplied Claude’s false alarms and surface nitpicks more than threefold on flawed packets, but a full-context verifier filtered out 87.5% of the false alarms and 97.4% of the nitpicks while keeping every discovered bug. If we split a large codebase across parallel workers without a full-context verifier, we shift the filtering burden onto human researchers.
The limits of this benchmark follow from its design. Our packets are self-contained 2,000-word exercises with scripted mutations, and every planted bug sits inside at least one file pair. Natural errors in full-length working papers can span wider pipelines that cannot fit in one prompt. Our test also covers one pair of frontier models without a binding output-token cap, so it cannot establish results for smaller models or compressed repositories.
References and Replication
- Packaged empirical datasets from
statsmodels,linearmodels, andscikit-learn, including Grunfeld (1950) Investment Data; U.S. Strike Duration Data; Engel (1857) food expenditure data; El Nino - Sea Surface Temperatures; Mauna Loa Weekly Atmospheric CO2 Data (with Mauna Loa attributed to CDIAC / Scripps Institution of Oceanography). Our study repository retains the dataset provenance record, blinded scoring outputs, and standalone Python scripts that reproduce the tables, figures, and paired tests above.