Cycling Through Bad Ideas Faster: An Article-Reported Medicaid Example

Do state Medicaid programs that go by names like TennCare, HUSKY, MassHealth, and Cardinal Care pull in more enrollees than the ones that just call themselves “[State] Medicaid”? The closest direct empirical test is a 2024 preregistered survey experiment by McIntyre, McCrain, and Pavliv, which found that respondents under the branded names were less sure what the program was[1]. This article describes a cross-state test of whether that confusion appears in behavior. The source datasets are public; the transformed analysis files and results are not. What changes when the cost of exploring the question falls?

Cheap iteration looks dangerous at first. Faster cycles feel like more opportunities to miss something, more pressure to ship the first robust-looking result, more risk of blind data mining. The experience supports the opposite concern: cheap iteration can lower the sunk-cost pressure to publish a weak result. It also makes it easier to test a promising idea more extensively, because the bandwidth that used to go into data wrangling now goes into testing alternative codings, learning unfamiliar techniques, and verifying citations against primary sources.

The judgment about what is worth pursuing determines whether something becomes a paper, on either side of the submission[2]. A companion piece, on the editor-side of the same question, treats the published record as data to ask what "important" has actually meant.

This post is an article-reported example of that judgment from the researcher’s side. It describes a two-week exploration with three coding rules, four model specifications, confounder diagnostics, a literature pretest, and a stratified mechanism check. Those workflow details and timings are not independently logged. The reusable idea is the name-proximity coding rule shown below. No public replication archive currently supports the project-specific results.

Idea 1: just code “branded” by judgment

The article starts by judgment-coding program names, then describes a staggered Callaway-Sant’Anna DiD on adult applications per eligible[3]. It reports +22% at event time zero with a confidence interval excluding zero. The underlying treatment file and model output are not public.

A +22% result would support a striking story about branded states pulling more applications. It also runs against McIntyre’s survey-experimental finding (n = 5,807 respondents[1]), making verification especially important.

The article says the estimate fails for two reasons: judgment calls make the coding unstable, and added controls move the reported effect toward zero. Those are article-reported diagnostics, not independently reproduced findings.

Idea 2: use a string-similarity benchmark instead

We need a less subjective coding. The first algorithmic move is to benchmark state program names against the 2014 ACA Marketplace plan corpus using trigram-Jaccard similarity. If a state’s program name is similar to commercial-insurance plan names, it counts as branded. We code it up and look at the top-scoring states.

The metric has a structural flaw, and looking at the top-scoring states reveals it immediately. The QHP corpus is saturated with “Care” and “Health” suffixes, which inflates similarity scores for state programs whose only distinctive element is a kid-program label. Georgia’s combined “Georgia Medicaid / PeachCare for Kids” scores above the threshold because of “PeachCare for Kids” matching dental and children’s QHP plans, not because of any adult-program branding. New York’s “New York State Medicaid / Child Health Plus” has the same problem. Three states score above the threshold for entirely the wrong reason.

The article describes this discovery as a single afternoon: write the scoring code, inspect the top-scoring states, trace unexpected rankings to their driving trigrams, and discard the benchmark. Its comparison with a week-long coauthor cycle is an author estimate, not a timed benchmark.

Idea 3: measure distance from “Medicaid” itself

The third candidate benchmark is distance from generic program-name anchors rather than similarity to commercial exchange plans. For this rule, those anchors are “Medicaid,” “[State Name] Medicaid,” “[State Abbreviation] Medicaid,” and “Medical Assistance.” Higher distance means less trigram overlap with that specified set, which the analysis proposes as a measure of consumer-facing distinctiveness.

This produces an inspectable coding rule that can be tested in other naming settings. The core algorithm is short:

import re

def trigrams(s: str) -> set[str]:
    s = re.sub(r"[^a-z0-9 ]", " ", s.lower())
    s = re.sub(r"\s+", " ", s).strip()
    return {s[i:i+3] for i in range(len(s)-2)} if len(s) >= 3 else set()

def jaccard(a: set, b: set) -> float:
    return len(a & b) / len(a | b) if (a and b) else 0.0

def distance_from_anchors(name: str, anchors: list[str]) -> float:
    """1 minus max trigram-Jaccard similarity against any anchor."""
    t = trigrams(name)
    anchor_sets = [trigrams(a) for a in anchors]
    if not t or any(not a for a in anchor_sets):
        raise ValueError("name and every anchor must contain at least 3 normalized characters")
    return 1.0 - max(jaccard(t, a) for a in anchor_sets)

For state Medicaid program names, the anchor set is ["Medicaid", f"{state_name} Medicaid", f"{state_abbr} Medicaid", "Medical Assistance"]. We score each state’s adult-program name segment against the four anchors, take the closest match, and call the state branded if the distance is at least 0.85.

Trigram-Jaccard similarity itself is not new. Broder’s 1997 paper defines resemblance for sets of document shingles and develops a sampling approach for estimating it; Charikar’s 2002 paper develops locality-sensitive hashing constructions for several similarity measures, including cosine similarity[4][5]. Neither paper is a source for the exact deterministic function above. Applied microeconomics also uses text-similarity measures, including Gentzkow and Shapiro’s newspaper-language analysis[6].

What this article proposes is a specified distance-based coding rule for state Medicaid branding. The page does not document a systematic literature search sufficient to establish that the application is novel or that earlier studies uniformly relied on unreported judgment. The excerpt makes the rule inspectable, but the project result is not yet reproducible without the inputs and outputs. Marzilli Ericson and Starc examined how standardization shifted brand market shares in the Massachusetts exchange[7]; that is a related branding application, not validation of this program-name rule.

The technique could generalize to SNAP rebrands, TANF state programs, CHIP program names, ACA Marketplace plans, or other settings with ambiguous categorical treatments. The code excerpt specifies the scoring rule; a reusable public module has not yet been released.

The article reports 22 branded states under the 0.85 threshold and 6 in-window staggered cohorts. Those assignments should be treated as article-reported until the treatment file and scoring output are public.

What does the article report when controls are added?

The Article-Reported Controls Ladder

The article reports that rerunning the staggered DiD on the new coding produces a +1.2 percentage-point coefficient at event time zero, described as barely significant. That estimate is the stated starting point for the controls ladder, not a publicly reproduced result.

The reported +1.2-point estimate is fragile. What checks should we run? The standard ones for a state-panel DiD: regional shock heterogeneity, then expansion-cohort heterogeneity, then political controls, then renewal-infrastructure controls. We add each layer and record the point estimate.

The article reports that HHS region by year fixed effects widen the interval without moving the point estimate much. It then reports a sign change after adding expansion-cohort by year fixed effects, with the branding coefficient moving to −2.2 percentage points and the interval crossing zero. No public model output currently reproduces that sequence.

Article-reported controls ladder: the coefficient is described as moving from approximately +0.012 in the initial specification to approximately −0.022 after expansion-cohort by year controls, with a reported 95% interval of roughly [−6.6, +2.1] percentage points. The chart formerly shown here was removed because no public model output currently supports it.

The article's proposed mechanism is that brand timing overlaps with ACA expansion timing, so an uncontrolled brand variable can absorb expansion-related enrollment trajectories. That explanation is plausible, but the cohort file and decomposition needed to check it are not public.

The article also reports a significant event-minus-three pre-period coefficient in its fully controlled specification. If reproduced, that would indicate residual cohort heterogeneity and limit a causal reading of the event-time-zero estimate.

Reading the Article-Reported Interval

Conditional on the unpublished specification, the reported interval spans about 6.6 percentage points of enrollment loss to 2.1 percentage points of enrollment gain. It should not be read as a verified bound until the model and inputs are reproduced.

The McIntyre mechanism implies a subgroup test. If state-specific names create recognition confusion, states with low ex parte renewal rates provide a potential subgroup test. The article reports a 19-month stratification from March 2023 through September 2024 with only two in-window branded cohorts in the low-ex-parte stratum. It describes the result as underpowered and opposite the predicted direction. The subgroup output is not public.

That small, nonpublic stratification does not provide persuasive evidence. The substantive check is whether branded states differ systematically from unbranded ones on the renewal-infrastructure variables that would offset confusion.

The article also reports comparisons across six potential confounders and says only governor party differs materially between branded and unbranded states. The chart formerly shown here was removed because its underlying comparison table is not public.

If branded states had administrative infrastructure that mechanically muted the McIntyre recognition channel, a controlled null could be consistent with a real behavioral effect that the infrastructure offsets. The article reports no detectable difference on its selected renewal-infrastructure measures. That diagnostic cannot be independently audited until the comparison table is public.

Even if reproduced, the design could not establish that branding never matters, overturn McIntyre's survey result, or predict a pure name-change experiment. A within-state design around a documented rebrand would be a stronger next test.

Ending or extending an analysis

An analytical thread can end when the evidence does not support the idea or continue with additional tests. The project-specific evidence here still needs a public reproduction package before readers can evaluate that decision.

The article says the workflow discarded two coding ideas and developed a third into a controls ladder, a confounder diagnostic, and a mechanism check. Its afternoon-versus-week time comparison is an author estimate. Readers can inspect the sequence of checks; the elapsed-time and empirical-result claims still need a public record.

The failed coding attempt also produced the distance-from-anchor scoring rule. The first algorithmic coding failed in a specific direction. That failure indicated that a different reference target was needed. The low cost of testing it made the new module practical. In a slower workflow the QHP-corpus failure would have prompted a return to judgment coding. Cheap iteration changes what techniques researchers build, in addition to changing which specifications they run.

Two verification failures from this iteration loop

Separate verification passes detected both failures.

A subagent verifying citations for this article found that an earlier draft had attributed Aizer (2007), “Public Health Insurance, Program Take-up, and Child Health,” to the Journal of Public Economics. The paper is published in Review of Economics and Statistics, 89(3):400–415. The wrong-journal attribution would have appeared in the reference list and embarrassed the author at any reader who pulled the citation. The verification subagent flagged it because the verification protocol fetches the actual source rather than generating the citation from memory.

An earlier verification round in a different session attributed authorship of a Constantin et al. paper to “Constantin/Kenney/Simon/Chua” when the actual authors are Constantin/Chua/McCullough. The verification memo’s tight formatting created an unwarranted appearance of authority. The AI produced a confident, incorrect claim about names it had never actually fetched. The procedure that detects these errors treats verification memos as inputs to verify against primary sources, never as authoritative outputs. Citation verification is a consequential source of confident AI errors and therefore requires a separate verification pass.

We can see that AI is unreliable in known directions: verbatim quotes from inaccessible sources, citation metadata from memory, confidently-formatted memos that report names the system never looked up. The iteration cost of building separate verification passes against those known failure modes is low enough that solo researchers should build the passes and treat the AI outputs as input artifacts to check.

What a public replication archive must contain

No public repository currently exists at the archive address previously named on this page. A complete release for this example should contain:

• naming_proximity.py: the standalone reusable scoring module
• state_treatment_v1.csv, v2.csv, v3.csv: three treatment files representing the judgment, QHP-Jaccard, and distance-from-Medicaid codings
• state_unwinding_metrics.csv: cumulative ex parte and procedural disenrollment rates by state from CMS data.medicaid.gov, computed over March 2023 through September 2024
• run_controlled_did.R: the R script that produces the M1 through M7 controls ladder
• A README documenting variable definitions, data sources, and the exact steps to reproduce the figures shown above

Until those files and a run record are public, readers can inspect the code excerpt and research logic but cannot reproduce the project-specific classifications, figures, or estimates.

---

References

[1] McIntyre, A., McCrain, J., & Pavliv, D. (2024). Medicaid by any other name? Investigating malleability of partisan attitudes toward the public program. Journal of Health Politics, Policy and Law, 49(3), 451–471. https://doi.org/10.1215/03616878-11066320

[2] Cunningham, S., & Simon, K. (2026, May 20). What a panel of economists said about AI in the production of research. Scott’s Mixtape Substack. Writeup of moderated panel at NBER Applications of AI in Healthcare meeting, Cambridge, MA, May 8, 2026; panelists Kosali Simon, Scott Cunningham, David Bradford, and Coady Wing. https://causalinf.substack.com/p/what-a-panel-of-economists-said-about

[3] Callaway, B., & Sant’Anna, P. H. C. (2021). Difference-in-differences with multiple time periods. Journal of Econometrics, 225(2), 200–230. https://doi.org/10.1016/j.jeconom.2020.12.001

[4] Broder, A. Z. (1997). On the resemblance and containment of documents. In Proceedings of the Compression and Complexity of Sequences 1997 (pp. 21–29). IEEE Computer Society. https://doi.org/10.1109/SEQUEN.1997.666900

[5] Charikar, M. S. (2002). Similarity estimation techniques from rounding algorithms. In Proceedings of the Thirty-Fourth Annual ACM Symposium on Theory of Computing (pp. 380–388). ACM. https://doi.org/10.1145/509907.509965

[6] Gentzkow, M., & Shapiro, J. M. (2010). What drives media slant? Evidence from U.S. daily newspapers. Econometrica, 78(1), 35–71. https://doi.org/10.3982/ECTA7195

[7] Marzilli Ericson, K. M., & Starc, A. (2016). How product standardization affects choice: Evidence from the Massachusetts Health Insurance Exchange. Journal of Health Economics, 50, 71–85. https://doi.org/10.1016/j.jhealeco.2016.09.005