Can a Classifier Find What Investigators Miss?

The HHS file covers 84 months. This article previously reported about 1.8 million unique providers, but no saved public output derives that count. The final model sample is also under reconciliation, so the sample, label rules, and temporal split must be frozen before interpreting any metric.

What Published Studies Achieve

The fraud detection literature reports a wide range of model performance, typically measured by AUC (area under the receiver operating characteristic curve), which captures how well a model ranks positive examples above negatives. Here are studies spanning the main approaches (supervised, unsupervised, and ensemble) and data sources (Medicare Part B, Part D, inpatient):

Study	Data	Method	Key Metric
Shekhar, Leder-Luis & Akoglu (2026)	Medicare inpatient	Unsupervised ensemble	Nearly 5x lift over random
Johnson & Khoshgoftaar (2023)	Medicare Part B/D	XGBoost, Random Forest	AUC ~0.83 (imbalanced)
Tajrobehkar et al. (2024)	Medicare ophthalmology	Stacking ensemble	AUC 0.907
Herland et al. (2018)	Medicare Part B + D + DMEPOS	Logistic Regression	AUC 0.816

These studies use Medicare data, which has richer fields than the public Medicaid file (submitted charges, place of service, provider specialty directly in the data). Several supervised Medicare studies use LEIE-derived labels; Shekhar, Leder-Luis, and Akoglu validate an unsupervised ranking against U.S. Department of Justice anti-fraud lawsuits.

What might we expect from a classifier trained on public Medicaid data with seven columns and no diagnosis codes? An AUC between 0.70 and 0.85 is a planning range, not an empirical result. Any operational assessment must also report precision, recall, calibration, and subgroup error at a prespecified threshold.

What SHAP Tells Us About the Labels

Often the most informative output from a supervised model is the feature importance structure rather than the predictions themselves. SHAP values (SHapley Additive exPlanations) come from the same cooperative game theory that economists use to allocate contributions among players in a coalition [5]. The logic works like a decomposition: just as we might decompose a wage gap into portions attributable to education, experience, and industry, SHAP decomposes each fraud prediction into the contribution of each billing feature. Every feature gets a signed credit: positive if it pushed the prediction toward “fraud,” negative if it pushed away. The feature contributions plus the expected or base model output sum to the prediction in the explainer’s output space.

The article reports that SHAP ranks billing intensity as the dominant signal after restricting labels to specified fraud-related exclusion codes. It also reports roughly 140 claims per month for fraud-excluded providers and 63 for non-excluded providers. Those feature rankings and group comparisons remain article-reported until the sample is reconciled and the model rerun.

The article also reports a reversal in SHAP directions when all LEIE exclusion types define the positive class. Observation-window differences and label contamination are plausible mechanisms, but the reported reversal has not yet been reproduced from one frozen label version.

Other studies confirm this pattern. SHAP combined with unsupervised anomaly detection on Belgian GP data uncovered a previously unknown billing trend [6]. In one ophthalmology fraud model, a constructed ratio (total payments divided by total patients) emerged as the most predictive SHAP feature, more informative than any raw variable [3]. The takeaway: SHAP shows which features drive the model output, which may diverge from what we intended.

Once we know which features drive the model output, the next question is whether the model itself needs to be complex.

Does ML Actually Beat Logistic Regression?

Most fraud detection papers compare random forests, gradient-boosted trees, and neural networks, reporting whichever achieves the highest AUC. Few compare against logistic regression, the simplest baseline. (ML here refers to machine learning methods more complex than logistic regression: random forests, gradient-boosted trees, and neural networks.)

A systematic review examined exactly this question across 71 clinical prediction studies [7]. At low risk of bias (proper validation, no data leakage, adequate sample size), the performance difference between ML and logistic regression was exactly zero: 0.00, with a 95% confidence interval from -0.18 to +0.18. ML only appeared superior in studies with high risk of bias, where methodological shortcuts inflated performance.

That review covered clinical prediction tasks (mortality, readmission, diagnosis) rather than fraud detection specifically. Fraud may involve more complex feature interactions that tree-based methods capture. Still, the burden of proof should run the other direction: demonstrate that the complex model outperforms the simple one on properly validated data, rather than assume it does.

Why does this matter for fraud detection on Medicaid data? If logistic regression matches XGBoost or random forest in discrimination, the simpler model wins. A state Medicaid Fraud Control Unit can explain to a judge exactly why logistic regression flagged a provider. It runs without GPU clusters or hyperparameter tuning pipelines. Analysts who understand regression coefficients can update it directly.

A broader argument in Nature Machine Intelligence makes the same point: for high-stakes decisions, interpretable models should be the default unless a black box demonstrably outperforms them [8]. In fraud detection, that “demonstrably outperforms” threshold is rarely met.

Temporal Validation: Why Random Cross-Validation Can Mislead

A subtlety trips up much of the fraud detection literature. For time-ordered data, random k-fold cross-validation can produce optimistic estimates by training on later observations and evaluating earlier ones. If we train on 2022 data and test on 2020 data, the fitted relationship can use patterns that only became apparent after the fact. In real deployment, prediction moves forward in time.

How big is the gap? That depends on the data-generating process, the validation design, and how much billing and enforcement patterns change. A systematic review of 2,030 external validations of cardiovascular prediction models found a median 11% decrease in discrimination, with performance drops ranging from near-zero to over 30% depending on how different the validation population was from the development data [12]. That review is not fraud-specific, so it cannot establish an expected AUC drop for Medicaid fraud models. It does show why performance should be measured again on later data rather than inferred from a random split.

A chronological validation can split the data by time: train on 2018-2021, validate on 2022, and test on 2023-2024. That design approximates prospective use when the deployment target is later-period screening. The exact split must follow the intended deployment date and label-availability process. Grouping by provider National Provider Identifier (NPI) addresses a different leakage risk when repeated provider records cross folds; it should be incorporated without allowing future records to enter an earlier training set.

Temporal data leakage is a recurring methodological problem in fraud-detection research, and several reviews have flagged it. When a study skips temporal validation, its reported AUC may be optimistic under leakage or drift. It should not be treated as a guaranteed upper bound, because performance can move in either direction when the population, measurement process, or enforcement environment changes.

How to Interpret a Small Incremental Gain

The article describes a prospective classifier trained on during-panel exclusions and tested on providers excluded after the panel. The following values are article-reported and under reconciliation.

The article reports a prospective AUC of 0.725 (95% CI: 0.676-0.777) and a non-fraud placebo AUC of 0.579 (95% CI: 0.482-0.670). Non-overlapping reported intervals would be consistent with a difference, but do not by themselves verify that the signal is fraud-specific. The samples and calculations must be reproduced.

At a reported top-5% screening threshold, the article says the classifier captures 23% of future fraud exclusions. That is a screening claim, not a verdict, and it should not guide audit allocation until the threshold result and its false-positive burden are reproduced.

The article further reports AUC values of 0.723 after dropping months_active and 0.670 for a rate-only specification. These sensitivity results and the related 2.3-times billing-intensity comparison inherit the same unresolved sample.

The cited MFCU report provides external context for program recoveries [11]. The article's 0.098% base rate, below-2% precision statement, and claim that 23% of future exclusions fall in the top 5% are model results under reconciliation. False positives also carry potential investigative and access costs that an operational evaluation would need to measure.

The article reports a within-panel AUC of 0.830 and a prospective AUC of 0.725, a difference of 0.105. That comparison is a useful validation target, not a verified benchmark.

The appropriate conclusion is narrower: a seven-column public billing file can support a prospective screening experiment. Whether billing intensity provides useful discrimination remains to be established by a frozen rerun, calibration checks, subgroup error analysis, and comparison with simple baselines.

Supervised and Unsupervised: Complements

Up to this point we have been asking one question: can a supervised classifier, trained on historical labels, predict future fraud? But there is another approach entirely. What if we skip the labels?

Dimension	Supervised	Unsupervised
Detects	Known patterns from historical labels	Novel anomalous behavior
Labels needed	Yes (LEIE/S&I)	No
Bias risk	Inherits enforcement bias from labels	Less susceptible
Strength	High precision for established fraud types	Can discover new schemes
Weakness	Misses novel fraud; reflects historical priorities	Higher false positive rate

The unsupervised ensemble discussed in Post 3 achieved nearly fivefold lift over random targeting of hospitals in Medicare inpatient data [1]. Instead of estimating “what do excluded providers look like,” unsupervised methods ask “which providers look most unlike their peers?” That reframing bypasses the enforcement bias baked into LEIE labels, though it introduces its own difficulty: peer groups must be well-defined, and anomalous billing is not the same as fraud.

In practice, supervised and unsupervised methods serve different roles in the same pipeline. A supervised classifier trained on fraud-specific labels can screen for known patterns. An unsupervised anomaly detector can flag providers whose billing is unusual for reasons the labels never captured. Both still depend on the investigator who examines the medical records.

What Does All This Tell Us?

These four posts establish a research design, not a verified fraud signal. The public file is large and monthly, the available labels are selective, and observation windows differ. The billing comparison and prospective classifier remain under reconciliation.

If a frozen rerun reproduces useful discrimination, the model could be evaluated as an audit-prioritization tool. It must never be treated as a verdict, and its threshold-specific false positives require explicit review.

A credible fraud-detection program would require better labels, richer data, prespecified validation, simple baselines, equity audits, false-positive tracking, and appeal mechanisms. The public Medicaid spending file is one possible input to that system.

---

References

1. Shekhar, S., Leder-Luis, J., & Akoglu, L. (2026). Can machine learning target health care fraud? Evidence from Medicare hospitalizations. Journal of Policy Analysis and Management, 45, e70078.
2. Johnson, J.M. & Khoshgoftaar, T.M. (2023). Data-centric AI for healthcare fraud detection. SN Computer Science, 4(4), 389.
3. Tajrobehkar, M. et al. (2024). Utilization analysis and fraud detection in Medicare via machine learning. medRxiv, 2024.12.30.24319784. [Preprint.]
4. Herland, M. et al. (2018). Big data fraud detection using multiple Medicare data sources. Journal of Big Data, 5, 29.
5. Lundberg, S.M. & Lee, S.I. (2017). A unified approach to interpreting model predictions. NeurIPS, 30.
6. De Meulemeester, H., De Smet, F., van Dorst, J., Derroitte, E., & De Moor, B. (2025). Explainable unsupervised anomaly detection for healthcare insurance data. BMC Medical Informatics and Decision Making, 25, 14.
7. Christodoulou, E. et al. (2019). A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. Journal of Clinical Epidemiology, 110, 12-22.
8. Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1, 206-215.
9. Athey, S. (2017). Beyond prediction: Using big data for policy problems. Science, 355(6324), 483-485.
10. Mullainathan, S. & Obermeyer, Z. (2022). Diagnosing physician error: A machine learning approach to low-value health care. Quarterly Journal of Economics, 137(2), 679-727.
11. OIG OEI-09-25-00200. (2025). Medicaid Fraud Control Units fiscal year 2024 annual report.
12. Wessler, B.S. et al. (2021). External validations of cardiovascular clinical prediction models: A large-scale review of the literature. Circulation: Cardiovascular Quality and Outcomes, 14(8), e007858.

---

Updated July 2026 to label article-reported model results that remain under reconciliation.

This series references the working paper: Cholette, V. (2026). What Do Medicaid Fraud Classifiers Actually Detect? SSRN Working Paper.