For models where you own the gradients, gradient attribution is faster and better-grounded than perturbation methods. Integrated Gradients is the method to know: it is the one with axioms instead of vibes, it scales to transformer embeddings, and Captum gives it a production-quality implementation for anything in PyTorch [1] [2].
Why plain gradients are not enough
The naive method, saliency, reads the input gradient as importance. It fails on saturated networks: once a feature has pushed a neuron into its flat region, the local gradient is near zero exactly where the feature mattered most. A model can be certain because of a feature whose gradient at the input is nothing.
Integrated Gradients
IG fixes saturation by integrating the gradient along a straight path from a baseline input to the actual input, and multiplying by the input-baseline difference. Two axioms make it defensible in a report. Sensitivity: if changing one feature changes the prediction, that feature gets nonzero attribution. Completeness: attributions sum exactly to the prediction difference from the baseline, the same additivity contract that makes SHAP auditable [1] [3].
Two knobs decide whether your IG numbers mean anything:
- The baseline is the counterfactual your attributions are measured against, and it is a modeling decision. All-zeros is conventional and often wrong: for text, use the pad or mask token embedding; for images, consider a blurred input rather than a black one; for tabular, the feature means. "Attribution versus black image" answers a different question than "attribution versus average input". Write the baseline choice into the report.
- The path integral is approximated with
n_steps discrete steps. Captum returns delta, the gap between the attribution sum and the actual prediction difference. Check it every run: a large delta means your step count is too low and the completeness axiom has quietly left the room. Raise n_steps until delta is a rounding error.
Figure: Integrated Gradients fixes gradient saturation by averaging gradients over n_steps points on the straight-line path from the baseline x' to the input x, so attributions sum to F(x) - F(x') under the completeness axiom even where the local gradient, and therefore plain saliency, is near zero. Raise n_steps until Captum's convergence delta is a rounding error.
Captum in practice
Captum wraps IG (plus DeepLift, GradientSHAP, occlusion, and conductance) behind one attribution API [2]. The pattern for a text classifier, the case that matters most for LLM-era teams evaluating fine-tuned classifiers, judges, and reward models:
import torch
from captum.attr import LayerIntegratedGradients
# forward_fn returns the logit being explained
lig = LayerIntegratedGradients(forward_fn, model.get_input_embeddings())
attributions, delta = lig.attribute(
inputs=input_ids,
baselines=pad_token_ids, # token-id baseline; Captum embeds it
n_steps=64,
return_convergence_delta=True,
)
token_attr = attributions.sum(dim=-1) # collapse embedding dim -> per-token scores
LayerIntegratedGradients targets the embedding layer because token ids are discrete and have no gradient; this is the standard trick for transformers. The per-token scores that come out are how you audit a safety classifier for keying on identity terms, a judge model for scoring the rubric header instead of the answer, or a moderation model for reacting to dialect features. Attribution on the classifier that gates your product is exactly the same discipline as judge bias auditing, one layer down.
The sanity checks most saliency maps fail
Adebayo et al. proposed two tests any attribution method must pass before you trust it: randomize the model weights and the explanations should degrade to noise; randomize the training labels and they should change. The results are humbling and specific: plain gradients and Grad-CAM pass; guided backprop and guided Grad-CAM produce nearly identical maps for trained and randomized models, behaving like edge detectors; and Integrated Gradients itself is only partially sensitive, because the input-minus-baseline multiplier stamps input structure into the maps regardless of what the model learned [4]. That last result is the reason to run the model-randomization check on your own pipeline rather than assuming your method is exempt. It costs a full attribution pass over a randomized copy of the model, and it is the difference between an instrument and a decoration.
TIP
Pretty maps are the failure mode. An attribution image that "looks reasonable" is confirming your prior, which is precisely what a broken method also does. Trust the checks, not the aesthetics: convergence delta near zero, randomization tests passed, and baseline documented. Then let the attributions argue with you.
Choosing between this chapter and the last one
Tree ensemble on tabular data: TreeExplainer SHAP, no contest. Deep model with gradient access: IG through Captum. Black-box API with neither: KernelExplainer SHAP or LIME, with their costs and caveats accepted in writing. The explanations recipe closes with a Captum port of its SHAP workflow so the two chapters meet in code.