AI Fairness 360 started at IBM and now lives under the LF AI & Data foundation, and it is the encyclopedic option: more than 70 metrics and a dozen mitigation algorithms spanning the whole pipeline, in Python and R [1]. Where Fairlearn optimizes for the common sklearn case, AIF360 optimizes for coverage. That trade shows in the API, and knowing the shape of it in advance saves an afternoon.
The dataset classes are the API
Everything in AIF360 flows through its own dataset containers, chiefly BinaryLabelDataset and StandardDataset. You declare the label, the favorable label value, the protected attributes, and which values count as privileged, and the metrics read that declaration:
from aif360.datasets import StandardDataset
from aif360.metrics import BinaryLabelDatasetMetric, ClassificationMetric
ds = StandardDataset(
df, label_name="approved", favorable_classes=[1],
protected_attribute_names=["sex"], privileged_classes=[[1]],
)
priv, unpriv = [{"sex": 1}], [{"sex": 0}]
# Dataset-level: bias in the labels themselves, before any model
dm = BinaryLabelDatasetMetric(ds, unprivileged_groups=unpriv, privileged_groups=priv)
print(dm.statistical_parity_difference()) # base-rate gap in the data
print(dm.disparate_impact()) # four-fifths style ratio
# Model-level: bias in the predictions
cm = ClassificationMetric(ds_test, ds_pred, unprivileged_groups=unpriv, privileged_groups=priv)
print(cm.equal_opportunity_difference())
print(cm.average_odds_difference())
The wrapping is boilerplate, but it buys a distinction Fairlearn does not foreground: BinaryLabelDatasetMetric measures the labels before any model exists. Running it on your training data answers "how biased is the ground truth", which is the question that decides whether mitigation belongs in the data or in the model. A statistical parity difference in the raw labels means historical process bias baked into what you are about to teach the model to imitate.
Figure: AIF360's two measurement levels. Wrapping a DataFrame in StandardDataset lets BinaryLabelDatasetMetric quantify label bias before any model exists, while ClassificationMetric compares ds_test against ds_pred. Bias in the ground truth points to pre-processing fixes like Reweighing; bias in the predictions points to in- or post-processing mitigation.
The algorithm zoo, by pipeline stage
| Stage | Algorithm | What it does | Reach for it when |
|---|
| Pre | Reweighing | Reweights training examples per group-label cell | You can retrain, want the model untouched otherwise |
| Pre | DisparateImpactRemover | Repairs feature distributions toward group medians | Feature-level repair is defensible in your domain |
| In | AdversarialDebiasing | Trains against an adversary predicting the attribute | Deep model, TensorFlow stack, strong parity target |
| In | PrejudiceRemover | Adds a fairness regularizer to the objective | Logistic-regression class models, tunable trade-off |
| Post | RejectOptionClassification | Flips low-confidence predictions near the boundary | No retraining allowed, score access only |
| Post | CalibratedEqOddsPostprocessing | Equalizes error rates while preserving calibration | Calibration is a hard requirement downstream |
Reweighing is the workhorse: simple, transparent, model-agnostic, and easy to explain to a reviewer. Adversarial debiasing is the strongest in-processing option and the most operationally expensive. The post-processing pair mirrors the trade the metrics chapter warned about: you can equalize odds or preserve calibration, and the calibrated variant is explicit that it relaxes one to keep the other [2].
Honest comparison with Fairlearn
Choose Fairlearn when your model is an sklearn estimator and your need is a disaggregated audit table with one or two standard gaps; MetricFrame is a better audit surface than anything in AIF360, and the API fits pipelines you already have [3]. Choose AIF360 when you need a metric Fairlearn does not ship, dataset-level label bias measurement, mitigation stages beyond post-processing and reductions, or R. Many teams use both in one audit: AIF360's dataset metrics on the labels, Fairlearn's MetricFrame on the model.
Two cautions from production use. First, AIF360's index-based dataset alignment is unforgiving: ClassificationMetric trusts that ds_test and ds_pred rows correspond exactly, and a silently shuffled DataFrame produces plausible nonsense. Assert row identity before metric calls. Second, the project moves slower than its dependency stack; check the pinned scikit-learn and TensorFlow versions in the repo before adding it to an existing environment [4].
NOTE
Seventy metrics is not a feature for your report. The number you can defend is the number you chose in advance for a written reason. Use the zoo to find the one metric that matches your harm model, then report that metric per group with intervals, and resist decorating the report with every statistic the library can emit.
The fairness audit recipe demonstrates the combined workflow: label-bias measurement in the AIF360 style, model audit and mitigation in Fairlearn, one report at the end.