Responsible AI has a reputation problem: too many decks, not enough numbers. This section treats it as an evals discipline. Fairness is a set of metrics you compute per demographic slice, with confidence intervals, the same way you compute pass rates. Explainability is a debugging instrument that tells you which features or tokens drove a prediction, so you can catch leakage and spurious correlations before an auditor does. Bias in generative models is a probe suite you run on a cadence, and data integrity is a checklist you enforce in CI.
The tooling here is mature and free. Fairlearn and AIF360 compute the standard group-fairness metrics and ship mitigation algorithms with documented trade-offs [1] [2]. SHAP, LIME, and Captum's Integrated Gradients attribute predictions to inputs, each with failure modes you need to know before trusting the pictures [3] [4] [5]. For LLMs, public probe sets like BBQ and RealToxicityPrompts anchor your bias reporting the way HarmBench anchors your safety reporting [6] [7].
The opinionated frame: measure before you mitigate, disaggregate before you average, and never publish a fairness number without the subgroup sample sizes next to it. A single aggregate fairness score is the same trap as a single accuracy number, and the impossibility results mean every metric choice is a product decision someone has to own [8].
Chapters:
- Fairness metrics you can defend. Demographic parity, equalized odds, predictive parity, the impossibility results, and how to pick a metric by harm type.
- Fairlearn in practice. MetricFrame disaggregation, parity gap metrics, and the ThresholdOptimizer and reductions mitigations, in scikit-learn idiom.
- AIF360 in practice. The metric zoo, the dataset classes that trip people up, and pre-, in-, and post-processing mitigation with honest guidance on when not to.
- SHAP and LIME. Local surrogates versus Shapley values, which explainer for which model, and using attributions to catch leakage.
- Integrated Gradients and Captum. Gradient attributions for deep models, baseline choice, convergence checks, and the sanity checks that many saliency methods fail.
- Bias and toxicity probes for LLMs. BBQ, BOLD, RealToxicityPrompts, counterfactual probes on your own traces, and how to report the numbers.
- Data integrity checks. Provenance, leakage, split hygiene, contamination, poisoning, and the PII scan, as a release checklist.