The fastest way to learn to read attributions is to plant a bug and watch the tool find it. This recipe trains a classifier on a dataset with a deliberately leaky feature, catches the leak with SHAP global importance in about four lines, then uses local explanations the way you would in a real error analysis session [1]. Everything is seeded, and the outputs on this page come from a real run of the script (library versions noted inline), so yours should match up to minor version drift.
The 45 minutes: ten to run the script and see the leak surface, fifteen to work through the local explanations, twenty to run the same screen on a model you actually own.
What you build
One script, explain.py, in three acts: make a credit-shaped synthetic dataset with a planted leak, train and explain globally (the leak detection), then explain three individual predictions (the debugging workflow). A closing section sketches the same audit for a PyTorch text model with Captum, connecting to the gradient attributions chapter.
Step 1: plant the leak
"""Catch a leaky feature with SHAP.
Run:
pip install shap scikit-learn pandas matplotlib
python explain.py
"""
import numpy as np
import pandas as pd
import shap
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(7)
n = 6000
df = pd.DataFrame({
"income": rng.normal(52, 18, n).clip(8, 160),
"debt_ratio": rng.beta(2, 5, n),
"years_employed": rng.integers(0, 30, n),
"late_payments": rng.poisson(1.2, n),
"utilization": rng.beta(2, 3, n),
})
logit = (0.035 * df.income - 2.2 * df.debt_ratio + 0.06 * df.years_employed
- 0.55 * df.late_payments - 1.4 * df.utilization - 0.5)
y = (logit + rng.logistic(0, 1, n) > 0).astype(int)
# The leak: "account_status_code" is written AFTER the default decision by a
# downstream system, then joined back in by a careless pipeline. It encodes
# the label with 5% noise, the way real leaks do.
noise = rng.random(n) < 0.05
df["account_status_code"] = np.where(noise, 1 - y, y) + rng.normal(0, 0.05, n)
X_tr, X_te, y_tr, y_te = train_test_split(df, y, test_size=0.3, random_state=7)
clf = RandomForestClassifier(n_estimators=300, random_state=7).fit(X_tr, y_tr)
print("test accuracy:", round(clf.score(X_te, y_te), 3)) # suspiciously high
Step 2: the four-line leak detector
explainer = shap.TreeExplainer(clf)
sv = explainer(X_te) # exact Shapley values for trees
global_importance = np.abs(sv.values[..., 1]).mean(axis=0)
print(pd.Series(global_importance, index=X_te.columns)
.sort_values(ascending=False).round(4))
Output from the seeded run (shap 0.51, scikit-learn 1.9):
account_status_code 0.4085
income 0.0356
years_employed 0.0337
late_payments 0.0256
debt_ratio 0.0199
utilization 0.0153
One feature carrying an order of magnitude more attribution than every legitimate signal combined is the leak signature. In a real pipeline you now ask the temporal question: could this value exist at prediction time? A status code written after the outcome cannot, so it goes, the model retrains, and accuracy falls back to honest. TreeExplainer makes this screen cheap enough to run at dataset onboarding, not just in postmortems; that pairing of exact tree Shapley values with ensemble models is the best-case use of the tool [2].
shap.plots.bar(sv[..., 1]) and shap.plots.beeswarm(sv[..., 1]) draw the same table; the beeswarm additionally shows direction, which is where you notice things like a feature that only matters at extreme values.
Figure: SHAP global feature importance from the seeded run: the planted leak account_status_code carries mean |SHAP| 0.4085, more than 11 times the strongest legitimate feature (income at 0.0356). One bar dominating every real signal is the label-leakage signature the four-line TreeExplainer screen is built to catch.
Step 3: local explanations as error analysis
wrong = np.flatnonzero(clf.predict(X_te) != y_te)[:3]
for i in wrong:
shap.plots.waterfall(sv[int(i), :, 1])
Read each waterfall the way you read a failing transcript: which features pushed toward the wrong decision, and does that push make business sense? On the retrained (leak-free) model, the misclassified cases typically show two legitimate signals fighting (high income against high utilization), which is the "model is reasonable, case is hard" verdict. What you are screening for is the other verdict: a single feature dominating an error for no defensible reason, which in production models is where proxies for protected attributes surface. That check, run per group on the MetricFrame audit's false negatives, connects explanations back to fairness.
Additivity is the property that makes waterfalls trustworthy: contributions plus the base value equal the model output exactly, so nothing is hidden in a residual [1].
Step 4: the same audit for deep models
For a PyTorch model the instrument changes but the workflow does not. Captum's LayerIntegratedGradients gives per-token attributions for a text classifier with the same additivity contract (completeness), and the same two screens apply: global attribution mass concentrated on a template artifact or metadata token is your leak; per-example attributions on misclassified inputs are your error analysis [3] [4]. The code pattern lives in the gradient attributions chapter; check the convergence delta each run, and run the model-randomization sanity check once before trusting any of the pictures.
Adapting to your product
Swap the synthetic frame for your feature table and keep the screens in this order: global importance for leaks and proxies at every dataset change, local explanations on a sample of errors during every error analysis pass, and attribution-profile comparison across demographic groups whenever the fairness audit flags a gap. If your model is a black-box API, accept KernelExplainer costs knowingly or fall back to input-ablation probes; the SHAP and LIME chapter covers that decision.