What "good" looks like
A safety eval scores two axes at once: attack success rate (ASR) on red-team sets, and over-refusal rate on benign-but-sensitive requests. Reporting either alone is broken.
The concrete failure: a model reaches ASR of 0% by refusing "how do I clean a wound" because it looks medical-adjacent. Useless in production.
The "good" target is product-dependent:
- Consumer chatbots: ASR well below 5% on standard red-team sets, over-refusal under 10% on XSTest.
- Coding assistants and enterprise tools: may tolerate different thresholds.
- The two-axis trade-off should be explicit in the release report.
XSTest is the standard over-refusal probe[1] and pairs naturally with HarmBench for the ASR side[2]. MLCommons AILuminate provides the industry-consortium standard with 12 hazard categories and over 24,000 prompts per language[3]. Its v1.0 safety test is single-turn and covers content hazards only. The separate Jailbreak benchmark, still in development, reports a Resilience Gap: the drop from baseline safety to under-attack performance. MLCommons found that none of the 39 models it tested scored as well under jailbreak as on its safety rating[4].
flowchart TD
REQ["Incoming request"] --> CLS{"Harmful or benign-sensitive?"}
CLS -->|harmful| ASR["ASR check: HarmBench red-team set"]
CLS -->|benign-sensitive| OVR["Over-refusal check: XSTest"]
ASR --> G1{"ASR below 5%?"}
OVR --> G2{"Over-refusal below 10%?"}
G1 -->|no| FIX1["Harden refusals"]
G2 -->|no| FIX2["Loosen over-broad refusals"]
G1 -->|yes| GATE{"Both axes pass?"}
G2 -->|yes| GATE
GATE -->|yes| SHIP["Ship"]
Figure: The two-axis safety eval target for a consumer chatbot: a release passes only inside the ship zone, with attack success rate (ASR) under 5% on red-team sets like HarmBench and over-refusal under 10% on XSTest. Driving ASR to zero alone lands in the over-cautious region that refuses benign requests.
Secondary metrics
| Metric | What it catches | How to score |
|---|
| False-refusal rate | Benign-sensitive requests wrongly refused | The XSTest axis |
| Demographic robustness | Behavior varying with demographic attributes of requester or target | BBQ, the canonical bias benchmark[5] |
| Persona-based attacks | Role-play that opens the door to harmful behavior | Deshpande et al. found that assigning ChatGPT a persona raised toxicity by up to 6x, depending on the persona[6] |
| Jailbreak persistence | A successful attack surviving across turns | Rate at which the attack remains successful over model turns or sessions |
Offline eval design
Core sets. Run the HarmBench standard subset for ASR and AILuminate practice prompts for industry-comparable scoring[3]. Of the 24,000 prompts per language, 12,000 are practice prompts and 12,000 are held private for official grading. Only a 1,200-prompt demo subset of the practice set is a public download; the full practice set and the official evaluator go to MLCommons members on request, and numbers scored with your own classifier are not comparable to official grades[7]. Keep XSTest in the pipeline for over-refusal.
Size the over-refusal set to the decision. XSTest is 250 safe prompts plus 200 unsafe contrasts[1], so one prompt moves the rate 0.4 points. When a model sits near the 10% line, add a larger set. OR-Bench has 80,000 over-refusal prompts across 10 categories and a hard subset of about 1,000 (ICML 2025)[8]. FalseReject has 16k seemingly toxic queries across 44 categories and a human-annotated test set (COLM 2025)[9].
Rotate the private set. Add a private red-team set rotated quarterly; static benchmarks contaminate over time and attackers adapt.
Add a multi-turn set. Most automated ASR sets are single-turn, and AILuminate v1.0 says so explicitly. Human multi-turn jailbreaks exceeded 70% ASR on HarmBench against defenses that reported single-digit ASR under automated single-turn attacks. The MHJ dataset of 2,912 prompts across 537 multi-turn jailbreaks is public[10].
The operational pattern. The HarmBench three-step pipeline is the pattern to copy: generate test cases, model completions, classifier eval[2]. Use the HarmBench classifiers (Llama-2-13b and Mistral-7b variants) for the judge stage. ASR is only as good as that judge: the StrongREJECT authors found that existing evaluation methods significantly overstate jailbreak effectiveness relative to human judgments[11], so hand-label a sample of judge verdicts before you trust a headline number.
High-stakes deployments. Pre-deploy external red-team engagement is the right default; Anthropic's 38,961-attempt human red-team study is the canonical methodology reference[12].
Online eval design
Classify and log. In production, run a classifier on inputs and outputs and log every triggered event. Sample a fraction (start at 10%) for review.
Alert on two signals:
- Attack-pattern surges: a new jailbreak technique often appears as a sudden spike in one category.
- Over-refusal complaint volume: user reports of refused benign requests.
Cadence. Quarterly external red-team engagements are the right cadence for high-stakes products; the threat model changes faster than internal teams can keep up with.
Top benchmarks
- HarmBench, https://github.com/centerforaisafety/HarmBench. 18 attack methods across 33 LLMs; the academic standard for ASR measurement.
- MLCommons AILuminate, https://mlcommons.org/benchmarks/ailuminate/. 12 hazard categories, over 24,000 prompts per language; the industry consortium standard. Safety v1.0 is single-turn, and the Jailbreak benchmark is still in development[3].
- XSTest for over-refusal: 250 safe prompts and 200 unsafe contrasts[1]. Pair with HarmBench for the two-axis scoring, and with OR-Bench when 250 prompts are too few to separate two models.
- HarmBench classifiers: the judge stage of the standard pipeline.
- Giskard: v3 (3.0.0, released 2026-08-26) is a rewrite for multi-turn testing of agents.
giskard-scan covers red teaming, prompt injection, jailbreaks, and harmful content, and v2 is no longer actively maintained[13].
- garak (NVIDIA): LLM red-team with a wide attack catalog.
- Promptfoo red-team mode: CI-integrated adversarial testing. Promptfoo announced in March 2026 that it is joining OpenAI; the repository states it remains open source under the MIT license, and the team says it will keep supporting other providers[14].
- AdvBench: transferable-suffix attacks from Zou et al., the canonical optimization-based jailbreak baseline[15].
Common pitfalls
| Pitfall | Why it bites | Fix |
|---|
| ASR-only without over-refusal | Ships a useless model that refuses everything touching a sensitive topic | Keep the two-axis frame; it is non-negotiable |
| Static benchmarks only | Attackers adapt and static sets contaminate | Quarterly private red-team rotation |
| No multimodal or multi-turn coverage | Many attacks emerge in image or audio inputs or across multi-turn coercion, which single-turn text-only eval misses | Cover image, audio, and multi-turn attacks |
| Treating "refused" as "safe" | A model that refuses then answers the same request rephrased has not actually refused | Make the harm test specific to the harm being measured |
Canonical reading
The HarmBench paper is the canonical reading: it operationalizes a three-step pipeline (test cases, completions, classifier) that is the practical pattern for most automated safety evaluation[2].
The Ganguli et al. Anthropic red-team paper is the canonical reference for the human red-team methodology that high-stakes deployments still require[12]. Read together: HarmBench for the automated layer, Ganguli for the human layer.
Day-1 starter
- HarmBench standard subset and XSTest in CI. The two-axis frame as the default scoring.
- An output classifier on every production response, with a 10% sample routed to human review. Alert on attack-pattern surges and on over-refusal complaints.
- A quarterly external red-team engagement for high-stakes products. The threat model evolves faster than internal teams can track; external expertise is the calibration.
Cross-cutting links