A judge is a second LLM call that grades the first one. You give it the input, the candidate output, a rubric, and (often) a reference. It returns a label and a critique. That is the whole technique 1. Everything that follows is discipline around that one call: how to write the prompt, how to calibrate the score against humans, which biases will derail you, and which classes of task you should never put a judge anywhere near.
When the technique works
Use a judge when the criterion is real but expensive to write code for. The cases that consistently calibrate above 0.90 TPR and 0.90 TNR against a principal expert are open-ended generation with no clean reference, RAG faithfulness against a retrieved context window, tone and style adherence, and dimension-by-dimension summarization quality 2 3. Strong frontier judges agree with human raters above eighty percent on MT-Bench style preference data (GPT-4 at 85 percent against 81 percent between two humans, ties excluded), which is the empirical bound on the upper end 4. Check cheaper options first: the Evals FAQ says to build judges only for failures that survive a prompt fix, to use code assertions wherever a rule can catch the failure, and to budget a judge at 100 or more labeled examples plus weekly maintenance 5.
The cases where judges break: hard reasoning, math correctness, code correctness when the task is non-trivial. JudgeBench, the Tan et al. benchmark built to test judges on hard correctness with ground-truth labels, scores GPT-4o at 56.6 percent overall and 54.1 percent on the reasoning slice, where 50 percent is chance 6. Reasoning models do better under the same judge prompt: o1-preview reaches 75.4 percent, DeepSeek-R1 73.1 percent, and o3-mini at high effort 80.9 percent overall (89.8 percent on reasoning pairs). That table is the paper's April 2025 revision (fetched as of 2026-10-01) and does not cover 2026 models. Even the best row misjudges about one pair in five, well short of the 0.90 bar below. The intuition is mechanical: if the judge model cannot solve the underlying problem, it cannot reliably tell you whether the candidate solved it, and the judges that spend tokens thinking are the ones that improve. Use executable verification, not a judge, whenever the task admits a programmatic checker. Reserve the judge for the second signal.
The four-part prompt
The structure that consistently calibrates uses four parts. Role and definitions to kill ambiguity, binary criteria to force a falsifiable cutoff, few-shot examples lifted from the train split to carry the boundary cases prose cannot articulate, and a strict JSON schema so the parser fails loudly when the judge does. The pattern is treated in depth on the judge prompt structure page; the headline rule is that all four parts are non-negotiable 1.
# Part 1. Role and definitions.
You are an expert {DOMAIN_ROLE} evaluating whether responses
adhere to {DOMAIN_CRITERION}.
DEFINITIONS: ...
# Part 2. Binary criteria.
PASS: response adheres including method and content.
FAIL: response violates {DOMAIN_CRITERION} in any element.
When in doubt, FAIL and explain why.
# Part 3. Three few-shot examples from the TRAIN split.
...
# Part 4. Strict output schema.
Return JSON: {"label": "PASS" or "FAIL", "critique": "..."}
Temperature zero where the API lets you set it. Claude Opus 4.7 and later return a 400 error for any non-default temperature, so on those models pin the model ID and the prompt instead 7. One judge per failure mode: the FAQ names a judge asked to catch many kinds of error at once as the most common mistake, and Anthropic's agent-evals guide says to grade each dimension with its own judge and to let the judge return Unknown when it lacks the evidence 5 8. Show each judge only the part of the trace its failure mode needs, since extra context can make it worse 5. Strict parse on the way out. Few-shot examples come from the train split, never the dev or test split, or your calibration numbers stop meaning anything.
Calibration against humans
The judge is a classifier you wrote in English; you measure it like any classifier. Label 100 to 200 traces per failure mode with the principal domain expert, then split them. The FAQ's ranges are 10 to 20 percent train, 40 to 45 percent dev, and 40 to 45 percent test, so the 15, 40, 45 split in the figure fits; stratify so dev and test each hold 30 to 50 PASS and 30 to 50 FAIL where the data allows 5. Hamel's rule of thumb is that below about 60 examples the confidence intervals are too wide to support a conclusion, and the figure draws the 100-trace floor 1. The train split exists to pick few-shot examples. The dev split is your iteration target; you measure TPR and TNR against it and rewrite the prompt until both clear 0.90. That bar is this site's default; the FAQ says to set targets from the cost of a missed failure against the cost of a false alarm in your product. The test split is the unbiased number you cite to your team, and you look at it exactly once before deploy.
You can hand the dev loop to a prompt optimizer. GEPA, which reflects on failures in natural language and rewrites the prompt, is the example the FAQ names, and it was an ICLR 2026 oral 9. The FAQ's conditions still apply: read a few disagreements yourself first, because tuning too early hides missing-context and bad-label problems, and keep the test split outside the optimization loop 5.
Agreement percent is a trap on imbalanced data. A judge that says PASS for every trace agrees with the human reviewer ninety percent of the time when the base rate is ninety percent PASS, and it has captured zero failure signal. TPR (the rate at which the judge correctly flags failures the human flagged) and TNR (the rate at which the judge correctly clears passes the human cleared) are the metrics that survive imbalance. A June 2026 preprint that ran 21 judges through about 541,000 judgments found exact-match agreement overstates discrimination by 33 to 41 points against Cohen's kappa on MT-Bench, and that test-retest reliability above 0.95 coexisted with position bias above 0.10 in two production-deployed judges 10. A stable judge is not a valid one.
The recalibration cadence is monthly minimum: rerun fifty fresh traces through the judge, compute agreement with the same expert, and alert if either rate drifts below 0.85. A judge that calibrated cleanly six months ago has often drifted with the underlying model API. A model swap restarts calibration from the dev split; Hamel's answer to keeping a judge consistent across model updates is to go through the process again and measure 1. Forced migrations count: as of October 2026, Anthropic lists claude-sonnet-4-5-20250929 as deprecated since 2026-09-30 with retirement on 2026-11-30 and names claude-sonnet-5-5 as the replacement, so a judge pinned to the old ID needs a recalibration run before the cutover 7.
Figure: LLM-as-judge calibration workflow: 100 human-labeled traces are split 15/40/45 into train, dev, and test, few-shot examples come from the train split, and the judge prompt is rewritten against dev until TPR and TNR both clear 0.90. The test split is read exactly once before deploy, then 50 fresh traces are re-scored monthly with an alert if either rate drops below 0.85.
The three biases
Every team's first judge hits these in order. The mechanics are covered in detail on the biases page; the operational headlines are below.
Position bias. On pairwise judgments, the judge favors whichever response sits first. On the default prompt in Zheng et al.'s Table 2, GPT-4 is biased toward the first position 30 percent of the time, GPT-3.5 50 percent, and Claude v1 75 percent, and only 23.8 percent of Claude v1's verdicts survive a position swap 4. The mitigation is to swap positions and only count agreement when both orderings agree.
Verbosity bias. The judge prefers longer responses even when concision is closer to the rubric. MT-Bench's repetitive-list attack pads an answer with rephrased copies of its own content and fools Claude v1 and GPT-3.5 in 91.3 percent of cases, against 8.7 percent for GPT-4 4. The mitigation is rubric language that scores conciseness explicitly, or a length-normalized pairwise setup.
Self-preference bias. A judge favors outputs from its own model family; Zheng et al. call it self-enhancement bias, and G-Eval flags that LLM-based evaluators can favor LLM-generated text 4 3. For open-ended pairwise preference, the safer default is a judge from a different family than the generator, and the most capable judge your cost and latency budget allows 1. For a scoped binary judge the FAQ is looser: using the same model as the application is usually fine, because what matters is TPR and TNR on a held-out labeled set, and you switch providers only when alignment will not come 5. Measure it rather than assume it.
The MT-Bench figures above come from 2023 judges, and the bias profile has moved. Norman et al. audited 21 judges including the April 2026 frontier and found verbosity bias below 0.011 under a single pairwise rubric, but position bias above 0.10 in two production-deployed judges 10. Soumik (TMLR 2026) tested nine debiasing strategies on five judges and found style bias, a preference for markdown over plain prose, of 0.10 to 0.76 against position bias of 0.04 or less, with verbosity bias ranging from +0.24 to +0.44 for the Gemini and Llama judges, -0.12 for Claude, and -0.04 for GPT-4o 11. The biases are judge-specific, so measure them on the judge you deploy: swap order on a sample, compare length-matched pairs, and score the same content with and without markdown. Normalize formatting before judging.
Format sensitivity is a separate problem worth flagging. Sclar et al. demonstrate up to seventy-six accuracy-point swings on LLaMA-2-13B from meaning-preserving format changes 12. For judges, the relevance is that schema or option-label changes can move the judge's calls by enough to ruin a calibration. Pin the schema. Pin the option labels. Re-run calibration if either changes.
Direct versus pairwise
Two operational modes; pick by task. Direct scoring asks the judge to label one response against a rubric (PASS or FAIL). Pairwise asks the judge to pick between two responses on the same query. Yan's rule: pairwise for subjective preference where humans cannot calibrate a scalar, direct for objective binary criteria where the rubric has a clean cutoff 2. Direct binary is cheaper (one judge call per trace, no position swap), gives a confusion matrix you can report, and forces the rubric author to write a falsifiable criterion. Pairwise scales when ranking many systems without re-labeling each, at two judge calls per pair and the position-swap discipline 4. See pairwise and Elo for the Bradley-Terry mechanics and Chatbot Arena precedent.
Distillation in production
A frontier judge at one call per trace gets expensive at production volume. Distilled judges fine-tuned for the rubric are the practical answer. Prometheus trains a 13B open model on 1K rubrics and 100K GPT-4 feedback responses and reaches 0.897 Pearson with human raters on 45 customized rubrics, against 0.882 for GPT-4 13. Prometheus 2 handles both direct assessment and pairwise ranking and reports the highest agreement with humans among the open evaluator models it tested 14. JudgeLM ships 7B, 13B, and 33B judges; the 7B scores 5K samples in three minutes on 8 A100s, using swap augmentation for position bias, reference support for knowledge bias, and reference drop for format bias 15. Auto-J is a 13B generative judge that handles pairwise comparison and single-response evaluation and writes a natural-language critique 16. The pattern is the same in every case: a small judge for hot-path inference, a frontier judge as periodic calibration, a human gold set as the truth signal. Vendor judges follow it: Galileo's Luna-2 (announced June 2025) is a 3B and 8B pair with lightweight per-metric adapters on a shared core, listed at about $0.02 per million tokens 17; Vectara's open HHEM-2.1 has about 0.1B parameters 18; Patronus Lynx targets RAG hallucination 19.
Small judges generalize less well than the frontier judge they were distilled from. Yan's review of fine-tuned evaluators finds high in-domain scores but weaker generalization, fairness, and aspect-specific performance than GPT-4 2. On JudgeBench's hard pairs, Prometheus2-7b scores 34.9 percent and Prometheus2-8x7b 40.3 percent overall, below the 50 percent chance line 6. Validate a distilled judge on your own rubric before it takes the hot path.
Where judges still fail
Four failure modes are worth tracking even when calibration looks clean.
JudgeBench reasoning gap. GPT-4o is near chance on hard reasoning pairs and the best reasoning judge in the paper's table still misses about one pair in five; multi-step proofs, non-trivial code correctness, and multi-step math correctness are cases where the judge often lacks the capability to grade 6. Use a verifier.
Alignment gap and leniency. Thakur et al. ran 13 judge models over the answers of 9 exam-taker models on a task where humans agree with each other closely. Only the best and largest judges reached reasonable alignment with humans, their scores still differed from human scores by up to 5 points, and judges with high percent agreement could assign very different scores. They also found sensitivity to prompt length and complexity and a tendency toward leniency 20. Smaller judges ranked the exam-takers about as well, so ranking systems and scoring them are different jobs. Do not grade a judge with another judge; the human gold set is the only ground truth in the chain.
Noisy rubric checks on agent outputs. RuVerBench holds 2,458 deep-research and agentic-coding instances with human labels, and it finds substantial noise even in the strongest judges, with majority voting helping at diminishing returns and batched verification trading accuracy for cost 21. Budget for repeated judge calls on agent traces and measure the variance.
Criteria drift. Shankar et al. demonstrate that the human's notion of pass shifts as they look at more outputs, which means the rubric you wrote on day one does not match the rubric you would write on day thirty 22. The fix is a strict schema (so parser failures are visible) plus rubric versioning (so old labels stay legible).
What good looks like
A production judge program has all five of the following, or it is not production. A frozen rubric versioned with the code that uses it. A held-out human-labeled dev and test set of at least one hundred traces per failure mode, 30 to 50 of each label in dev and in test where the data allows (at the 100-trace floor a 40-trace dev split cannot hold 30 of each, so label toward 200), Cohen's kappa or Krippendorff's alpha at least 0.80 between two raters. TPR and TNR both above 0.90 on the test set, measured before deploy. Position-swap discipline on every pairwise comparison. A monthly recalibration cadence with an alert if judge-versus-human drift exceeds five points.
The cheapest first artifact is a fifty-trace labeled set, a one-page rubric, and a half-day of expert labeling. From there you walk the rubric to 100 to 200 labels per failure mode, split train, dev, and test, write the judge prompt, calibrate, then go look at the test number exactly once.