Classifier fairness has confusion matrices; generative models do not. An LLM's bias surface is behavioral: what it assumes, what it generates, and who it refuses. So the instrument changes from per-group error rates to probe suites, prompts constructed so that a biased response is detectable and countable. The good news is that the public suites are well built and the private ones are cheap to construct from traffic you already have.
The three public anchors
BBQ (Bias Benchmark for QA) probes stereotype reliance with paired contexts across nine social dimensions [1]. In the ambiguous condition, the context does not say who did what, and the correct answer is "unknown"; a model that picks the stereotype-consistent person instead is measurably importing the stereotype. In the disambiguated condition the context does say, and accuracy should recover. Report the two conditions separately: ambiguous-condition bias score is the number that moves when alignment changes, and disambiguated accuracy guards against a model that learned to answer "unknown" to everything, the same balance discipline as over-refusal testing [2].
BOLD covers open-ended generation: prompts drawn from Wikipedia across profession, gender, race, religion, and political ideology, scored for sentiment and toxicity differentials between groups [3]. It answers a question BBQ cannot: when the model is free to write anything, does the tone shift with the demographic?
RealToxicityPrompts measures degeneration under provocation: 100k naturally occurring sentence prefixes, including deceptively innocuous ones, continued by the model and scored for toxicity [4]. The headline metric is expected maximum toxicity over k continuations, which is a tail metric on purpose. Mean toxicity hides the failure that ends up in a screenshot; the max over 25 samples is much closer to what a determined user finds.
Figure: The three public LLM bias benchmarks compared: BBQ scores stereotype reliance with paired ambiguous and disambiguated QA contexts across 9 social dimensions, BOLD scores sentiment and toxicity differentials on open-ended Wikipedia prompts, and RealToxicityPrompts reports expected max toxicity over 25 continuations of 100k sentence prefixes.
Scoring the probes
Automated toxicity scoring means a classifier, usually Perspective API or an open-weights equivalent [5]. Two operational cautions. Toxicity classifiers have their own documented biases, flagging identity terms and dialect features at elevated rates, so a "toxicity drop" can be the scorer misfiring rather than the model improving; spot-check score distributions per identity term before trusting deltas. And classifier versions drift: pin the scorer version in your eval config like any other judge dependency, or your trend line measures the scorer's release schedule.
Counterfactual probes from your own traces
The public suites anchor comparisons; the probes that protect your product come from your traffic. The recipe is minimal pairs: take real prompts from your traces, produce variants that differ only in a demographic signal (names, dialect markers, stated identity, gendered role nouns), and diff the outputs. Score the pairs on whatever your product already measures: refusal, sentiment, verbosity, recommendation content, price quoted, tone. Aggregate as a flip rate (fraction of pairs where the outcome materially changed) per demographic axis.
flowchart LR
T["Sampled production prompts"] --> P["Generate minimal pairs<br/>(swap names, dialect, identity)"]
P --> R["Run both variants"]
R --> D{"Outputs materially differ?"}
D -->|no| OK["Count as stable"]
D -->|yes| REVIEW["Human reads the pair"]
REVIEW --> TAX["Tag axis and failure mode;<br/>add to regression set"]
This is dimensional sampling applied to fairness, and the output feeds the same machinery: confirmed failures become regression cases, flip rates become a dashboard slice, and the pairs corpus grows with every incident. A few hundred pairs across your top intents beats any public benchmark for detecting the regression your users will actually notice.
Reporting without theater
The report format mirrors the classifier fairness table: one row per demographic axis, the probe set and its size, the metric (BBQ bias score, sentiment differential, expected max toxicity, flip rate), the interval, and the trend versus last release. Resist the single blended "bias score"; the axes fail independently and a blend hides which one moved.
CAUTION
Probe suites go stale in two directions: models overfit to published sets that leak into training data, and the social assumptions a suite encodes age with it. Date-pin your public suite versions, hold out a private probe set that has never touched a vendor API, and treat a sudden unexplained improvement on a public bias benchmark with the same suspicion as any other contamination signal.