What "good" looks like
A conversational-quality eval asks two questions: does a human prefer the new model's response to the last release, and does each turn pass a domain rubric? Score both, and score turns as binary pass/fail rather than on a numeric scale.
The primary targets:
- Pairwise preference rate against the last released version, judged by whether a human prefers the new response to the baseline.
- Per-turn pass rate on a domain rubric, scored pass/fail per turn.
MT-Bench is the closest public reference, and it is saturated. It runs 80 multi-turn conversations across 8 categories with an LLM judge that reaches above 80% agreement with humans, the same level as inter-human agreement[1]. Frontier models now score near-perfect on it, so use it to check that your harness reproduces public numbers, not to separate candidate models[2].
MultiChallenge is the harder public multi-turn set. It tests four failure types: instruction retention, inference memory of user details, reliable versioned editing, and self-coherence (which includes avoiding sycophancy). Every conversation ends in a yes/no rubric question about the final response. A judge given that rubric agreed with human raters about 93% of the time, against 36% when it was prompted with the raw conversation[2]. The best model scored 41.4% at launch; the top entries on the live leaderboard are now in the mid-70s (fetched as of 2026-10-01)[3].
Chatbot Arena is the public gold standard for preference data but expensive to operate at scale. The LMSYS Bradley-Terry analysis is the citation-of-record for the methodology[4]. The project now runs as Arena at arena.ai, and lmarena.ai redirects there[5]. Read its rankings as a prior for model choice, not as a release gate: an audit of Arena data found that some providers privately test many variants and disclose only the best score, and that Google and OpenAI each received about 20% of all Arena data[6].
Use pass/fail, not a 1-to-5 scale. The framing that matters here, from Hamel, is binary pass/fail with a short critique[7]. Tracking scores on a 1-to-5 scale "is often a sign of a bad eval process": nobody can say what separates a 3 from a 4, so the numbers are not actionable and different evaluators read them differently. Annotators also park on 3, and the mean hides the disagreement.
Figure: Why conversational-quality evals score each turn as binary pass/fail instead of a 1-to-5 scale: annotators park on the middle rating so the mean hides disagreement, while pass/fail with a one-sentence critique and a judge calibrated to 80% expert agreement on 100 labeled traces keeps every disagreement visible.
flowchart TD
CONV["Production or test conversation"] --> TURN["Per-turn LaaJ"]
TURN --> RUBRIC{"Passes domain rubric?"}
RUBRIC -->|no| FAIL["Log turn failure: persona, retention, or factuality"]
RUBRIC -->|yes| PAIR["Pairwise A vs B, order randomized"]
PAIR --> PREF{"Preferred over last release?"}
CAL["PDE labels 100 traces"] --> AGREE{"LaaJ-PDE agreement at least 80%?"}
AGREE -->|no| ITER["Iterate the judge prompt"]
AGREE -->|yes| PREF
PREF -->|yes| SHIP["Ship"]
PREF -->|no| FIX["Fix the weaker axis"]
Secondary metrics
| Metric | What it catches | How to score |
|---|
| Persona-break rate | Assistant breaks character or admits it is an AI when the persona forbids it | Persona-break classifier on every response |
| Sycophancy | Model biases answers toward what the user appears to want to hear[8] | Compare answers under user-agreement pressure vs neutral framing; for sustained pressure across turns, count how soon the model flips (Turn of Flip) and how often (Number of Flip) as SYCON-Bench does[9] |
| Refusal false-positive rate | Assistant refuses a benign request | Fraction of benign prompts refused |
| Amnesia within window | Model forgets something it was told three turns ago | Per-turn retention check against earlier turns |
| Repeat-question rate | Model asks a question whose answer is in earlier turns | Fraction of turns re-asking answered questions |
Offline eval design
Start with MT-Bench for breadth. On top of it, build a domain dialogue set of 30 multi-turn scenarios with explicit per-turn rubrics covering:
- Persona adherence.
- Retention of earlier-turn information.
- No-hallucination on domain facts.
Score each turn binary pass/fail plus a one-sentence critique. Per Hamel's field guide and judge papers, that is the right scorer[7][10]. Avoid 1-to-5 scales.
Pair a calibration loop. A principal domain expert labels 100 traces, and the judge prompt is iterated until LaaJ-PDE agreement is at least 80%. Raw agreement flatters a judge when classes are imbalanced, so treat the expert labels as ground truth and report the judge's true positive rate and true negative rate separately. Hamel's current version of the guide also scopes the 100 as about 100 examples per failure mode, with enough passes and fails to measure both classes[7].
For pairwise preference, structure the eval as A vs B with the order randomized. LaaJ has documented position bias and verbosity bias, and both must be controlled.
Online eval design
Run a per-turn LaaJ on sampled production conversations. Sample conversations from production and score each turn.
Track explicit feedback by topic. Thumbs up and down broken down by topic tells you where the model is failing more than the global rate does.
Track the conversation-length distribution. A sudden shift usually means the model is being more verbose, less verbose, or losing context.
Run a persona-break classifier on every response. Alert on detected breaks, because they are visible to users in real time.
Top benchmarks
- MT-Bench, https://arxiv.org/abs/2306.05685. 80 multi-turn conversations, 8 categories; LaaJ-human agreement above 80%. Near-saturated at the frontier; keep it for harness parity.
- MultiChallenge, https://arxiv.org/abs/2501.17399. Four multi-turn failure types, one binary rubric question per conversation; best model 41.4% at launch.
- Chatbot Arena, https://arxiv.org/abs/2403.04132. Bradley-Terry Elo from human pairwise judgments; now Arena at arena.ai. A prior for model choice, not a release gate.
- Your domain set. The eval that predicts whether your assistant ships.
- Langfuse: live scoring, annotation queues, and per-turn LaaJ; the Langfuse cookbook on multi-turn evals is a useful template[11]. Langfuse v4 deprecates trace-level evaluators in favor of observation-level ones: on Cloud, existing trace-level evaluators stop producing results at the v4 cutover on 2026-11-16, and its docs describe multi-span evaluations as still to come on the new data model[12]. Score each assistant turn as its own observation and roll up per conversation yourself.
- Braintrust: hosted eval observability with strong pairwise primitives.
- MT-Bench harness: parity with public scores.
- Inspect AI: sandboxed deterministic runs across releases.
Common pitfalls
| Pitfall | Why it bites | Fix |
|---|
| Single-turn proxy benchmarks | MMLU and HumanEval predict almost nothing about multi-turn conversational quality | Build a multi-turn domain set instead |
| LaaJ position and verbosity biases | Untreated, they make pairwise comparisons unreliable | Randomize order and length-normalize where possible |
| 1-to-5 scale evaluation | Hamel calls it "often a sign of a bad eval process" | Use pass/fail with critique[7] |
| Skipping the calibration loop | A judge not calibrated against a domain expert on 100 hand-labeled traces is decoration, not a measurement | Calibrate on 100 traces before trusting the judge |
Canonical reading
Hamel's "Creating an LLM-as-a-Judge That Drives Results" is the canonical reading because it operationalizes the calibration loop that makes the judge trustworthy[7]. The "critique shadowing" pattern (one principal domain expert per judge prompt) is the most useful single technique in this category. Read it with the MT-Bench paper for the LaaJ-human agreement methodology[1].
Day-1 starter
- 30 multi-turn scenarios with per-turn rubrics. Persona, retention, factuality. The rubric is the eval.
- Pairwise A/B against the last release. One principal domain expert labels 100 traces; iterate the judge prompt until agreement is at least 80%.
- Track agreement percent, plus true positive and true negative rate, between LaaJ and the PDE in CI. When agreement drops, the judge has drifted from the rubric, which is the prior bug, not a regression in the model.
Cross-cutting links