The article reports a fraud-detection classifier with AUC between 92% and 96%, depending on the training sample. The underlying model, predictions, and evaluation output are not public here, so the range is not independently reproduced. The instructional question is still useful: what drives a classifier's predictions?
Feature importance from tree-based models gives us a ranking, sure. But a ranking doesn't tell us whether a feature pushes predictions toward fraud or away from it, how strong that push is, or whether the effect depends on other features. SHAP values (SHapley Additive exPlanations) answer all three questions. They decompose every prediction into per-feature contributions, grounded in cooperative game theory. The catch: interpreting SHAP correctly requires understanding what it measures and what it does not. A note on causal interpretation appears in Limitations.
Several interpretation methods exist for this kind of problem. LIME fits local linear models around individual predictions. Permutation importance measures how much test-set performance drops when a feature is shuffled. Partial dependence plots show the marginal effect of a feature across its range. SHAP is worth focusing on because it uniquely provides both local and global interpretability, with theoretical guarantees from cooperative game theory that the other methods lack. Let's walk through the workflow.
The interpretation workflow has seven steps, but they all depend on two core tools: a tree-based classifier and the TreeSHAP explainer. Here is the foundation.
Tool Stack: Random Forest + TreeSHAP
| Component | Tool | Purpose |
|---|---|---|
| Classifier | RandomForestClassifier | 200 trees, max_depth=12 |
| Explainer | shap.TreeExplainer | Fast exact SHAP for tree-based models |
| Visualization | SHAP summary plot | Feature importance with directional effects |
| Validation | Category aggregation | Group features into volume / intensity / behavioral / peer-relative |
The article describes 26 engineered features across four conceptual categories, Medicare provider billing records, and a binary fraud label. The analytical dataset and trained model are not public here, so those details are article-reported inputs for the teaching example.
Step 1: Fit the Random Forest
Let's start with a straightforward classifier. The key choices here are class_weight='balanced' (fraud is rare, so we need the model to pay attention to minority-class examples) and max_depth=12 (deep enough to capture interactions, shallow enough to avoid memorizing noise).
from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(
n_estimators=200,
max_depth=12,
min_samples_leaf=5,
class_weight='balanced',
random_state=42,
n_jobs=-1
)
rf.fit(X_train, y_train)The model is a means to an end; what we really want is the explanation layer on top.
Step 2: Initialize TreeSHAP
TreeSHAP computes SHAP values for supported tree ensembles much more efficiently than brute-force enumeration.2 Runtime depends on the model, SHAP version, explainer configuration, hardware, and sample size. This page does not publish a benchmark log for its 200-tree example.
The idea behind SHAP values comes from cooperative game theory. Shapley values were originally designed to fairly allocate credit among players in a coalition game. Here, the "players" are features and the "game" is the model's prediction. Each feature's SHAP value represents its average marginal contribution across all possible feature combinations, which gives us a principled way to decompose any single prediction into per-feature effects.1
Computing SHAP values on the test set tells us how the model explains new, unseen cases. This matters more for generalization than explaining training data the model may have memorized. Training-set SHAP values can reflect overfitting patterns, so test-set explanations tend to be a more honest picture of what the model has actually learned.
TreeSHAP's dependence assumption must be explicit. The current TreeExplainer documentation uses feature_perturbation="auto" by default: supplying background data selects the interventional approach, while omitting background data selects tree_path_dependent. Neither is a universal independence default. Correlated engineered features can divide attribution in ways that depend on this choice, so the SHAP version, background data, perturbation mode, and output scale all belong in the analysis record.
import shap
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
# Use a recorded background sample and request probability-scale explanations.
background = shap.sample(X_train, 200, random_state=42)
explainer = shap.TreeExplainer(
rf,
data=background,
feature_perturbation="interventional",
model_output="probability",
)
explanation = explainer(X_test) # Explain test data, not training dataStep 3: Handle the Binary Classification Array Shape
The first version-dependent issue appears here. With the current Explanation API, a multiclass tree model commonly returns an array shaped (n_samples, n_features, n_classes); some binary model integrations return a two-dimensional array for one output. Pin the SHAP version, inspect the returned shape, and map the positive label through rf.classes_ rather than assuming every model uses the same layout.
values = np.asarray(explanation.values)
positive_class_index = list(rf.classes_).index(1)
if values.ndim == 3:
sv = values[:, :, positive_class_index]
elif values.ndim == 2:
# This integration exposes a single modeled output. Confirm from the
# pinned model/library documentation that it is the positive class.
sv = values
else:
raise ValueError(f"Unexpected SHAP array shape: {values.shape}")For a three-dimensional result, the code selects the output associated with label 1. Do not assume that class-0 values are always the negation of class-1 values: that relationship depends on the model and output representation. Verify additivity against the requested probability output before interpreting or plotting the array.
For the article's reported 26-feature example, sv would have shape (n_samples, 26): one SHAP value per feature per test observation. Always verify the actual shape before continuing.
Step 4: Aggregate to Mean Absolute SHAP
Individual SHAP values are signed: positive means the feature pushed toward a fraud prediction, negative means it pushed away. To get overall importance, we take the mean of absolute values across all test samples.
It's worth pausing on what "importance" means here. Mean |SHAP| captures the average magnitude of a feature's contribution, but a feature with high mean |SHAP| does not necessarily have a consistent directional effect. It might push some predictions strongly toward fraud and others strongly away. The summary plot in Step 5 helps disambiguate this, so we should interpret the importance ranking and the directional plot together rather than treating the ranking alone as conclusive.
mean_abs_shap = np.abs(sv).mean(axis=0)
importance = pd.Series(mean_abs_shap, index=X_test.columns).sort_values(ascending=False)This gives us a global importance ranking (keeping in mind that SHAP values reflect association, not causation — see Limitations).
Step 5: Summary Plot (Top 15 Features)
The SHAP summary plot is where things get genuinely informative. Each dot is one test observation. Because Step 2 explicitly requests model_output="probability", the x-axis shows contributions to predicted fraud probability. With model_output="raw", the units would instead depend on the fitted model and should not be labeled log-odds without checking that model's documentation. Color encodes the feature's actual value (red = high, blue = low).
shap.summary_plot(sv, X_test, max_display=15, show=False)
plt.tight_layout()
plt.savefig("shap_summary_top15.png", dpi=150, bbox_inches="tight")
plt.close()What to look for: features where red dots cluster on the right (high feature value pushes toward fraud) versus features where the relationship is mixed or reversed. For instance, if avg_paid_per_claim shows red dots on the right, that means providers with high per-claim billing amounts are being flagged, consistent with domain knowledge about upcoding.
Reading a Single Provider's SHAP Profile
To make the probability-scale arithmetic concrete, consider a hypothetical provider. Suppose the explainer's positive-class expected value is 0.12. If billing_intensity contributes +0.15, monthly_spending_volatility +0.07, years_in_practice -0.08, and unique_hcpcs -0.03, those four contributions sum to +0.11. Adding +0.11 to the 0.12 expected value gives a predicted probability of 0.23. A real waterfall must include every remaining contribution and reproduce the model probability within numerical tolerance before being used in an audit.
Step 6: Group Features into Conceptual Categories
The article's example contains 26 features. Grouping a feature set into conceptual categories can make it easier to inspect which types of signal a model relies on.
The article's reported model assigns the largest category share to intensity features such as per-claim amounts, per-beneficiary costs, and spending volatility. Without the trained model and SHAP output, that ranking is not independently reproduced and should not be generalized to fraud classifiers as a class. The aggregation method below shows how to test the pattern in a supplied model.
categories = {
'Volume': ['total_paid', 'total_claims', 'total_beneficiaries',
'months_active', 'claims_per_month', 'avg_monthly_paid',
'entity_type'],
'Intensity': ['avg_paid_per_claim', 'avg_paid_per_beneficiary',
'claims_per_beneficiary', 'max_single_month_paid',
'monthly_spending_volatility', 'cv_monthly_paid'],
'Behavioral': ['share_top_code', 'hcpcs_hhi', 'hcpcs_entropy',
'unique_hcpcs', 'share_em_codes', 'share_high_reimburse',
'telehealth_share', 'rbcs_category_diversity',
'billing_gap_ratio'],
'Peer-Relative': ['z_cpm', 'z_ppb', 'z_entropy', 'z_paid']
}
for cat, feats in categories.items():
cat_importance = importance[feats].sum()
cat_share = cat_importance / importance.sum()
print(f"{cat:15s} {cat_share:.1%}")The article reports the following approximate breakdown. No public SHAP array or generated table currently reproduces it:3
| Category | Share of Total SHAP |
|---|---|
| Intensity | ~40% |
| Volume | ~35% |
| Behavioral | ~15% |
| Peer-Relative | ~10% |
In the article-reported breakdown, peer-relative features contribute approximately 10%. One possible explanation is that correlated raw and normalized features divide SHAP credit. That interpretation is a hypothesis, not a reproduced finding. Category aggregation can help organize the comparison, but it does not eliminate dependence among features.
Step 7: Does Importance Align with Domain Knowledge?
This is the step that separates mechanical SHAP computation from actual interpretation. Let's pose some questions and see whether the evidence supports sensible answers.
Do the top features match known fraud patterns? If avg_paid_per_claim and monthly_spending_volatility rank high, that's consistent with upcoding and burst-billing, both well-documented fraud schemes. If entity_type ranks high, we should check whether the model is picking up a real signal (organizations vs. individuals bill differently) or a data artifact.
Are any surprises genuine discoveries or artifacts? Suppose telehealth_share ranks unexpectedly high. Is that because telehealth genuinely correlates with fraud in the data (plausible for certain time periods), or because telehealth providers also tend to be smaller practices with different billing patterns? Disentangling association from mechanism requires domain investigation beyond what SHAP alone can provide.
Does the category breakdown shift across subgroups? Running the same aggregation on different provider specialties or entity types can reveal whether the model uses different signal types for different subpopulations. A model that relies on volume for one specialty and intensity for another might be picking up legitimate structural differences, or it might reflect label imbalance across groups.
What Can Go Wrong
SHAP is robust, but interpretation is fragile. Here are the failure modes we've encountered.
Array shape ambiguity. As noted in Step 3, the shap library's output format changed across versions. Code that works with shap==0.41 may break on shap==0.44. Always check type(shap_values) and shap_values.shape before proceeding.
Feature engineering artifacts. Z-scores, ratios, and log-transforms change what SHAP measures. If we feed the model z_paid (a provider's total billing normalized by specialty mean), SHAP tells us how much deviation from peers matters, not how much raw billing matters. These are different questions, and it's easy to conflate them.
Reweighting changes the SHAP landscape. If the training pipeline includes entropy balancing, inverse propensity weighting, or any sample reweighting, the SHAP values reflect the reweighted model rather than raw data relationships. Conditional importance under reweighting can look very different from unconditional importance. Both are valid; they answer different questions.
When to Use This Approach
TreeSHAP works best when we have a tree-based model (random forest, gradient boosting, XGBoost, LightGBM) and need per-prediction explanations. It's fast, exact, and well-supported. For global interpretation of a production classifier, it's hard to beat.
Less suitable when:
- The model is a neural network or SVM (use
KernelSHAPorDeepSHAPinstead, but expect slower computation) - Features number in the thousands (SHAP summary plots become unreadable; consider feature selection first)
- The goal is causal inference rather than predictive explanation (use causal methods instead)
- Stakeholders need a single importance number per feature and won't engage with distributional plots (permutation importance might communicate more clearly)
Limitations: SHAP Explains the Model, Not the World
Three constraints are worth keeping in mind. First, SHAP values explain model behavior, not data-generating processes. High SHAP importance does not mean a feature causes the outcome. If the model is biased or unstable, SHAP explains that model rather than correcting it. Second, computational cost scales with feature count and sample size. The article reports that its 26-feature, approximately 3,000-sample test set ran in under a minute, but it provides no public benchmark log. Larger feature sets and samples may require subsampling. Third, SHAP values are additive, but correlated and interacting features can make attribution difficult to interpret.
Public-Material Status
No complete SHAP implementation is currently available in the public or reviewed local materials. The article names 05_fraud_detection/phase4g_shap.py, but that file was not found. The code blocks on this page are instructional excerpts and do not establish an end-to-end reproducible pipeline.
A reproducible release would need the analytical data or a documented substitute, feature definitions, trained model, environment specification, complete interpretation script, SHAP arrays, and generated tables or plots.
References
[1] Lundberg, S.M. & Lee, S.-I. (2017). "A Unified Approach to Interpreting Model Predictions." Advances in Neural Information Processing Systems 30. The foundational paper connecting Shapley values to model interpretation. ↩
[2] Lundberg, S.M. et al. (2020). "From local explanations to global understanding with explainable AI for trees." Nature Machine Intelligence, 2, 56--67. Introduces the TreeSHAP algorithm used here. ↩
[3] The article reports category shares of approximately 35% for Volume, 40% for Intensity, 15% for Behavioral, and 10% for Peer-Relative features from a model run on Medicare Part B provider-level data. The model, SHAP arrays, and generated output are not public here, so the proportions are not independently reproduced. ↩