Why is 91% accuracy misleading in this example?

With an illustrative 9% positive and 91% negative split, predicting every biller is legitimate yields about 91% accuracy and zero recall for the positive class. Precision, recall, average precision (AP), and precision at k show different parts of performance.

How do we handle class imbalance in Python?

Options include class weighting, resampling within training folds, and metrics aligned with the rare class. This page provides instructional excerpts; its named complete implementation and reported model results are not publicly reproduced.

Imbalanced Classification: An Article-Reported Example

Classifying Medicaid billers as potential fraud cases can look like a textbook machine-learning problem. The article's example assumes roughly 9% positive cases and 91% negative cases. Under that illustrative distribution, a model that predicts every biller is legitimate scores about 91% accuracy while identifying zero positive cases. The mathematical lesson is general: accuracy can conceal failure on a rare class. The project-specific distribution and results on this page are not publicly reproduced.

Several approaches handle class imbalance: SMOTE generates synthetic minority examples, LightGBM and CatBoost offer built-in class-weight handling with fast training, and deep tabular models can work at scale. Here we'll combine Random Forest with controlled undersampling for transparency, since that combination makes it easier to see exactly how each piece of the pipeline affects performance. The Alternatives section covers other options in more detail.

This walkthrough presents the components of a scikit-learn workflow, from stratified splitting through forward-chaining temporal cross-validation. It is an instructional sequence rather than a complete public implementation. If logistic regression is familiar territory but random forests and gradient boosting are not, the examples show how the pieces fit together.

To build a classifier that performs well on those better metrics, we need tools that handle imbalance explicitly and give us multiple evaluation lenses. Here is the technical stack.

Tool Stack: scikit-learn, XGBoost, and Temporal CV

Instructional pipeline components
Component	Tool	Purpose
Models	RF, XGBoost, LogisticRegression	Three classifiers: one familiar, two tree-based
Imbalance	Undersampling + class weights	Changes the training distribution and error costs; must be validated
Validation	Forward-chaining temporal CV	Train on past, validate on future (no data leakage)
Metrics	AUC-ROC, average precision (AP), precision@k	Multiple evaluation lenses

The examples target Python 3.10+, scikit-learn 1.3, XGBoost 2.0, pandas, and numpy. No public environment lockfile or end-to-end run record is available for this article.

Step 1: Load Data and Sanity-Check It

The article describes approximately 38,000 Medicaid biller records with 25 continuous features and 1 categorical feature. It defines the target as a binary exclusion flag. The underlying analytical dataset is not public here, so these counts are article-reported inputs for the teaching example.

import pandas as pd
import numpy as np

df = pd.read_csv("medicaid_billers.csv")
print(f"Shape: {df.shape}")
print(f"Class distribution:\n{df['excluded'].value_counts(normalize=True)}")
print(f"Missing values:\n{df.isnull().sum()[df.isnull().sum() > 0]}")

A few things to check immediately:

Class balance. If roughly 9-10% of records are positive (excluded), we're looking at a ~10:1 imbalance. Not extreme by fraud-detection standards, but enough to make accuracy useless as a metric.
Missing values. Any feature with >30% missingness probably needs to be dropped or imputed with care. Median imputation works for continuous features here; for the categorical feature, a dedicated "Unknown" category avoids information loss.
Outliers. Billing-volume features often have extreme right tails. Let's leave them unclipped for now. Tree-based models handle skew well, and outliers in fraud data are often the signal, not the noise.

Step 2: Split with Stratification and Per-Fold Undersampling

Cross-validation splits the data into "folds," training the model on some folds and testing on the held-out fold, then rotating. It gives us a more honest estimate of performance than a single train/test split. But this is where the first subtle mistake typically happens. It's tempting to undersample the majority class once, globally, then split into folds. But that approach means every fold trains on the same subset of negatives. The model memorizes those specific controls rather than learning generalizable patterns.

Instead, undersampling needs to happen within each training fold, with a deterministic but fold-specific seed. Notice in the code below that np.random.RandomState(42 + fold_idx) gives each fold its own reproducible random-number stream. Calling np.random.seed(42) once before the loop would not repeat the same draw in every fold; the global generator advances. The reason to prefer fold-specific generators is that each fold's sample stays reproducible even if unrelated random operations are added elsewhere.

from sklearn.model_selection import StratifiedKFold

skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
RATIO = 3  # 3 negatives per positive

for fold_idx, (train_idx, val_idx) in enumerate(skf.split(X, y)):
    X_train_full, y_train_full = X.iloc[train_idx], y.iloc[train_idx]
    X_val, y_val = X.iloc[val_idx], y.iloc[val_idx]

    # Per-fold deterministic undersampling
    train_fraud = X_train_full[y_train_full == 1]
    train_legit = X_train_full[y_train_full == 0]

    rng = np.random.RandomState(42 + fold_idx)
    n_neg_train = min(len(train_fraud) * RATIO, len(train_legit))
    train_neg_idx = rng.choice(len(train_legit), size=n_neg_train, replace=False)

    X_train = pd.concat([train_fraud, train_legit.iloc[train_neg_idx]])
    y_train = pd.concat([
        y_train_full[y_train_full == 1],
        y_train_full[y_train_full == 0].iloc[train_neg_idx]
    ])

Step 3: Handle Class Imbalance (Belt and Suspenders)

Why test both undersampling and class weights? Undersampling changes the class distribution in the training data, while class weights change the loss assigned to errors. Combining them can improve ranking metrics in some datasets, but it can also overcorrect. Neither technique guarantees calibrated probabilities because both alter the effective training prior or loss. Calibration must be checked on untouched validation data with the deployment prevalence, and recalibration may be needed.

Adding class_weight='balanced' applies a second adjustment at the loss-function level. Whether the combination improves average precision or calibration is an empirical comparison, not a default property:

from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(
    n_estimators=200, max_depth=12, min_samples_leaf=5,
    class_weight='balanced',  # Adjusts loss to penalize minority-class errors more
    random_state=42, n_jobs=-1
)

The article reports preliminary comparisons of 1:1, 2:1, 3:1, and 5:1 undersampling ratios and says average precision peaked at 3:1 for random forest and XGBoost. No experiment table, implementation file, or run output is public, so that selection is an article-reported result rather than a reproduced benchmark. The general tradeoff remains: stronger undersampling discards more majority-class information.

Step 4: Train Three Models

A natural question: why not just use XGBoost? In fraud detection, interpretability carries weight. Logistic regression coefficients can be directly explained to auditors. Random forests offer feature importance that maps to investigative priorities. XGBoost often wins on raw metrics, yet its explanations require SHAP or similar post-hoc tools. Running all three lets us see whether the performance gap justifies the interpretability cost.

Let's compare three classifiers. Logistic regression is the workhorse most economists already know. A random forest aggregates hundreds of decision trees (each one a series of if/then splits on different features) and averages their predictions, which tends to reduce overfitting. XGBoost builds trees sequentially, where each new tree focuses on the cases the previous trees got wrong. Both tree-based methods capture nonlinear relationships and feature interactions that logistic regression misses.

from sklearn.linear_model import LogisticRegression
from xgboost import XGBClassifier

# Logistic Regression - the familiar baseline
lr = LogisticRegression(
    class_weight='balanced', max_iter=1000, random_state=42
)

# Random Forest - averages many decision trees to reduce overfitting
rf = RandomForestClassifier(
    n_estimators=200, max_depth=12, min_samples_leaf=5,
    class_weight='balanced', random_state=42, n_jobs=-1
)

# XGBoost - builds trees sequentially, each correcting prior errors
scale_pos = (y_train == 0).sum() / (y_train == 1).sum()
xgb = XGBClassifier(
    n_estimators=300, max_depth=6, learning_rate=0.1,
    scale_pos_weight=scale_pos, random_state=42,
    use_label_encoder=False, eval_metric='aucpr'
)

models = {'LogisticRegression': lr, 'RandomForest': rf, 'XGBoost': xgb}
for name, model in models.items():
    model.fit(X_train, y_train)

Step 5: Evaluate with Metrics That Matter

This is where the pipeline diverges from textbook ML. Why three metrics? Each tells a different story, and together they reveal whether the model is actually useful or just good at a narrow statistical game. Let's look at all three:

from sklearn.metrics import (
    roc_auc_score, average_precision_score, precision_score
)

results = {}
for name, model in models.items():
    y_score = model.predict_proba(X_val)[:, 1]

    auc_roc = roc_auc_score(y_val, y_score)
    average_precision = average_precision_score(y_val, y_score)

    # Precision at top k%
    prec_at_k = {}
    for pct in [1, 5, 10]:
        threshold = np.percentile(y_score, 100 - pct)
        y_pred = (y_score >= threshold).astype(int)
        prec = precision_score(y_val, y_pred, zero_division=0)
        prec_at_k[f'P@{pct}%'] = prec

    results[name] = {
        'AUC-ROC': auc_roc,
        'Average precision (AP)': average_precision,
        **prec_at_k
    }

pd.DataFrame(results).T.round(3)

Article-reported AUC-ROC: 0.92 to 0.96.^[1] AUC-ROC measures discrimination across all thresholds. It asks whether the model ranks positives above negatives, but it can look optimistic with class imbalance because the false-positive-rate denominator is large.
Article-reported average precision (AP): 0.85 to 0.91. The displayed average_precision_score computes non-interpolated AP as a recall-weighted mean of precision across thresholds. It is not the trapezoidal area under the precision-recall curve, sometimes called PR-AUC; the two summaries can differ.
Article-reported precision at the top 5%: approximately 60% to 70%. Precision@k connects ranking to a stated review capacity. These ranges are not backed here by public predictions or fold-level output.

As an arithmetic illustration, a top-5% review of 1,900 billers contains 95 cases. At 65% precision, about 62 of those reviewed cases would be positive. With a 10% base rate, random selection of 95 cases would yield about 10 positives on average. These values illustrate how to interpret the metric; they are not independently reproduced project counts.

Overall accuracy can therefore mislead. In the article's illustrative 91% majority-class example, always predicting the majority class produces 91% accuracy and zero recall for the positive class. Average precision and precision@k provide more relevant views when the operational task is finding rare cases.

Step 6: Compare Against a Domain-Knowledge Baseline

A model's metrics are uninterpretable without a baseline, and not a random baseline. We need a domain-knowledge baseline. What's the simplest rule an experienced auditor might use?

# Single-feature baseline: flag billers above 95th percentile
# on total_claims_amount
threshold_95 = X_val['total_claims_amount'].quantile(0.95)
y_baseline = (X_val['total_claims_amount'] >= threshold_95).astype(int)

baseline_prec = precision_score(y_val, y_baseline, zero_division=0)
print(f"Baseline precision (top 5% by claims): {baseline_prec:.3f}")

If the random forest's precision@5% is 65% and the single-feature baseline hits 40%, we can say the model adds 25 percentage points of precision: a concrete, defensible improvement. If the baseline hits 60%, the model's marginal value is slim and the complexity may not be justified. Without this comparison, reporting "65% precision" floats in a vacuum.

So far, Steps 2 through 6 used stratified k-fold cross-validation: the data is split into folds randomly, with stratification ensuring each fold preserves the original class balance. This approach tells us the model works on a representative sample of the data. But it doesn't tell us whether the model will keep working as billing patterns change year over year, because random splits allow the model to train on future observations and predict past ones. For temporal data, we need a fundamentally different splitting strategy.

Step 7: Test Whether the Model Holds Up Over Time

The stratified k-fold cross-validation in Step 2 splits data randomly while preserving class balance -- it's a strong approach for estimating general predictive performance. But for data with a time dimension, random splits have a fatal flaw: they can train on future observations to predict past ones. If billing patterns shift year over year (and they do, given policy changes, new fraud schemes, and pandemic disruptions), random CV will overestimate performance.

Forward-chaining temporal CV takes a different approach entirely. Instead of random splits preserving class balance, it splits strictly by time: each fold trains only on earlier years and validates on the next year forward. The training set grows with each fold, mimicking how a production model would actually be retrained. This means the model never sees the future during training, which gives us a more honest estimate of how well it will generalize to new data.

temporal_folds = [
    {'train': (2018, 2019), 'val': 2020},  # Fold 1
    {'train': (2018, 2020), 'val': 2021},  # Fold 2
    {'train': (2018, 2021), 'val': 2022},  # Fold 3
    {'train': (2018, 2022), 'val': 2023},  # Fold 4
    {'train': (2018, 2023), 'val': 2024},  # Fold 5
]

temporal_results = []
for fold in temporal_folds:
    train_mask = df['year'].between(*fold['train'])
    val_mask = df['year'] == fold['val']

    X_train_t, y_train_t = X[train_mask], y[train_mask]
    X_val_t, y_val_t = X[val_mask], y[val_mask]

    # Apply per-fold undersampling (same logic as Step 2)
    # Train models, collect metrics...
    temporal_results.append(fold_metrics)

If AUC-ROC is stable across folds (say, 0.93 +/- 0.02), the model seems to generalize well over time. If it degrades in later folds, something may be shifting: concept drift, policy changes, or data quality issues. That's valuable to know before deployment.

What Can Go Wrong

Having worked through the pipeline, let's catalog the failure modes. Each one can destroy, undetected, a model's real-world utility.

A shared global random-number stream makes fold samples sensitive to operation order. Calling np.random.seed(42) once does not repeat identical controls in every fold; successive draws advance the generator. Per-fold RandomState(42 + fold_idx) instances instead keep each fold's sample stable when unrelated random operations change.

Label leakage. Features like months_since_exclusion or exclusion_year encode the outcome directly. Any feature that could only be known after the label was assigned must be removed. This seems obvious but is easy to miss in wide feature sets assembled by different teams.

No domain-knowledge baseline. Without one, there is no way to assess whether the model provides value beyond a simple rule. Stakeholders will ask "couldn't we just flag the biggest billers?" We need a quantitative answer.

Random CV splits can leak future information. When evaluation is meant to mimic forward deployment, use a time-aware split that trains on earlier observations and tests on later ones. Expanding-window forward chaining is one option; a bounded rolling window can also be appropriate when it matches the deployment design.

Undersampling before splitting. A single global undersample changes both the training and validation populations before cross-validation begins. Each fold then evaluates performance on the selected class distribution rather than the untouched target population. Put the sampler inside the training-fold pipeline and evaluate on the original validation fold.

When to Use This Approach

This pipeline fits well when:

The positive class is rare (1-15% prevalence)
Data has a temporal dimension that matters
The operational question is "who should we investigate first?" (ranked retrieval)
Interpretability matters alongside performance
The dataset is moderate-sized (thousands to low millions of records)

Less suitable when:

Classes are roughly balanced (standard CV and accuracy work fine)
The problem is purely predictive with no ranking/prioritization need
Data volume exceeds millions of records and training time matters. Gradient boosting on the full dataset with scale_pos_weight may outperform undersampling

Alternatives Worth Exploring

This pipeline is one way to handle class imbalance. Several other approaches are worth considering, depending on the dataset and operational constraints.

LightGBM and CatBoost often match or exceed XGBoost with faster training. CatBoost handles categorical features natively, which avoids one-hot encoding overhead.
SMOTE and its variants (Borderline-SMOTE, ADASYN) generate synthetic minority examples instead of discarding majority ones. Results are mixed in the literature; undersampling tends to be more robust for fraud-like problems where the minority class is heterogeneous.
Deep learning (tabular transformers, TabNet) can work for very large datasets but rarely outperforms well-tuned gradient boosting on structured data below ~1M records.

Limitations

A few caveats worth noting:

Undersampling discards information. With five-fold training on about 80% of the article-reported 3,500 positives, each training fold would use approximately 2,800 positives and 8,400 sampled negatives at a 3:1 ratio. The general concern is that discarded negatives may contain patterns the model never sees.
Temporal CV reduces effective training data. Fold 1 trains on only two years. If the signal is noisy, early folds may underperform simply due to sample size, not model quality.
Precision@k depends on the k. Reporting precision@5% assumes the investigation unit can handle 5% of the population. If capacity is 1% or 15%, the metric needs to shift accordingly. Always align k with operational reality.
Feature engineering is not covered here. The pipeline assumes features are already constructed. In practice, feature engineering (interaction terms, rolling averages, provider-network features) often matters more than model choice.

Public-Material Status

No complete implementation is currently available in the public or reviewed local materials. The article names phase3_classifiers_v2.py, but that file was not found. The page's code blocks are instructional excerpts and do not establish that preprocessing, temporal cross-validation, model training, and evaluation run together as one reproducible pipeline.

A reproducible release would need the analytical dataset or a documented substitute, feature definitions, label construction, environment specification, complete training code, fold assignments, predictions, and fold-level metrics.

References and Notes

[1] The article reports AUC-ROC values of 0.92 to 0.96 and average-precision values of 0.85 to 0.91 across 5 temporal folds. The dataset, implementation, fold assignments, predictions, and output tables are not public here, so the ranges are not independently reproduced. ↩