Explainability tools answer one question: which inputs moved this prediction. That answer has two uses in an eval program. The compliance use is the famous one, but the debugging use is the one that pays weekly: attributions are how you catch a model that is right for the wrong reason, before production traffic finds out. A credit model leaning on a zip-code proxy, a text classifier keying on a template artifact, a medical model reading the scanner watermark instead of the scan: all of these look fine in aggregate metrics and obvious in attributions.
LIME: the local surrogate
LIME explains one prediction by fitting a small interpretable model in the neighborhood of that instance: perturb the input, watch the black box, fit a weighted linear model to the local behavior, and read its coefficients as the explanation [1]. It is model-agnostic and works on tabular data, text, and images.
The idea is durable; the estimates are not. LIME's output depends on the perturbation sampling, the kernel width that defines "local", and the number of samples, so two runs on the same instance can rank features differently. Treat LIME as a fast, qualitative instrument: good for a first look during error analysis, wrong for anything a number gets attached to. If a LIME explanation goes in a report, the reproducibility caveat goes next to it.
SHAP: Shapley values with a contract
SHAP assigns each feature a contribution such that contributions sum exactly to the difference between this prediction and the average prediction [2]. That additivity contract is what makes SHAP the default: local explanations aggregate into honest global ones (mean absolute SHAP value per feature), and nothing is left unattributed.
Which explainer matters more than most tutorials admit [3]:
| Explainer | Scope | Cost | Notes |
|---|
| TreeExplainer | Tree ensembles (XGBoost, LightGBM, sklearn forests and GBMs) | Fast, exact | The best-case pairing; default for tabular work |
| LinearExplainer | Linear and logistic models | Trivial | Mostly a consistency check on coefficients |
| KernelExplainer | Any model | Very slow, approximate | Last resort; sample counts dominate quality |
| DeepExplainer / GradientExplainer | Neural networks | Moderate | Superseded in practice by Integrated Gradients, next chapter |
Two decisions silently shape every SHAP output. The background dataset defines the "average prediction" everything is measured against; explaining a loan denial against a background of all applicants versus approved applicants yields different attributions, and the choice should mirror the contrastive question you are actually asking. And correlated features split credit in ways that can mislead: when two features carry the same signal, SHAP divides the attribution between them, so "feature X has low SHAP" does not mean the information in X is unused. Cluster correlated features before reading global importance, or the report will understate a proxy.
Figure: SHAP vs LIME in one view: LIME perturbs the input and fits a weighted linear surrogate, so feature rankings can change between runs, while SHAP's additivity contract guarantees per-feature Shapley values sum exactly to this prediction minus the average prediction over the background dataset.
Attributions as evals
The highest-value habit: make attribution review a standing step in error analysis. For a sample of false positives and false negatives, compute SHAP values and open-code what the model relied on, exactly as you would open-code transcripts. Leakage announces itself here first: a feature with implausibly high global importance (an ID column, a timestamp, a field populated after the outcome) is the classic tell that the data integrity chapter formalizes. For fairness work, compare attribution profiles across demographic groups from the MetricFrame audit; a feature that dominates only in the disadvantaged group's false negatives is a proxy with a paper trail.
CAUTION
An explanation explains the model, not the world. SHAP on a model that learned a spurious correlation will faithfully report the spurious feature, and a plausible-looking attribution is not evidence the model is right. Attribution methods can also disagree with each other while all passing visual inspection; sanity checks exist for exactly this failure and many popular methods flunk them [4]. Faithfulness checks live in the next chapter.
SHAP and LIME assume a fixed feature space and a scalar output, which is why they shine on tabular classifiers and strain on generative models. For LLM products, the analogue of this chapter is reading traces and running counterfactual probes, not computing Shapley values over tokens. For deep models where gradients are available, Integrated Gradients is usually the better instrument. The SHAP explanations recipe runs the full tabular loop, including planting a leaky feature and catching it.