The community headline is "GPT-4 reaches above eighty percent agreement with humans on MT-Bench" 1. The community quietly does not repeat the JudgeBench headline: GPT-4o performed roughly at chance on hard correctness judgments 2. Both are true. Generalizing one to the other will cost you a release.
This page is the honest take. Strong judges work where they work, and the place they work is preference data over open-ended generation. The place they fail is hard correctness: reasoning, math, code, science. If you cannot solve the problem, you cannot reliably grade the solution.
What JudgeBench measures
JudgeBench (Tan et al. 2024) is a benchmark specifically constructed to test judges on hard correctness judgments. The construction takes existing benchmarks where ground truth is known (knowledge, reasoning, math, code) and turns each item into a pairwise judgment task: given a query and two candidate responses, pick the correct one. The dataset has 620 pairs, 350 written by GPT-4o and 270 by Claude-3.5-Sonnet. Each pair is judged twice with the order swapped, and an inconsistent verdict or a double tie counts as wrong 2.
The result in the original paper: many strong models, GPT-4o among them, score "just slightly better than random." GPT-4o reached 50.9 percent overall with the vanilla prompt and 56.6 with the Arena-Hard judge prompt, against 50 for guessing. The categories where you would most want a verifier were the categories where the verifier was least reliable.
The April 2025 revision adds reasoning models, and they change the picture. With the Arena-Hard prompt held fixed, o3-mini (high) scores 80.9 percent overall, o1-preview 75.4, DeepSeek-R1 73.1, and Claude-3.5-Sonnet 64.3; o3-mini on its low setting drops to 70.6 2. Nvidia's model card reports 81.4 for Qwen3-Nemotron-32B-GenRM-Principle, an open 32B generative reward model 3. The gap narrows as the judge gets better at solving the problem, which is the mechanism below. It does not close: at 81 percent, about one pair in five is still judged wrong. These figures are from the paper revision and Nvidia's September 2025 snapshot, fetched as of 2026-10-01; check the live leaderboard before you quote a number for a newer model.
Thakur et al.'s "Judging the Judges" sharpens the point from a different angle. In a clean setup (TriviaQA answers, thirteen judge models, human raters who agree with each other), only GPT-4 Turbo, Llama-3.1-70B, and Llama-3-70B aligned strongly with humans, and the best judge still sat 8 points behind human judgment on chance-corrected agreement. Percent agreement hid the gap: it was high for almost every judge while the chance-corrected score was low for most, and the judges leaned lenient 4. "Agreement with humans" can look fine while the judge is wrong in a consistent direction.
Figure: LLM judge reliability follows whether the judge can solve the task. GPT-4 exceeds 80 percent agreement with humans on MT-Bench preference judgments. On JudgeBench hard correctness (knowledge, reasoning, math, code), GPT-4o with the vanilla prompt lands at about 51 percent, no better than chance, while the reasoning model o3-mini (high) reaches about 81 percent.
Why this happens
The mechanism is straightforward. To grade whether a candidate solution is correct, the judge has to internally check the solution. If the underlying problem is beyond the judge's capabilities, the judge cannot do the check. It falls back on heuristics: which response sounds more confident, which is longer, which uses more authoritative vocabulary. Those heuristics are exactly the biases we cover on the biases page, and they are why the position-swap and verbosity controls do not save you here. The bias is downstream of an upstream failure: the judge does not know which answer is right. The reasoning-model results above fit the same story: a judge that thinks the problem through before ruling is, in effect, solving it first.
This connects directly to a related result. Lightman et al. ("Let's Verify Step by Step") show that on hard math reasoning, a process-reward model trained on per-step labels significantly outperforms outcome-only supervision; the verifier needs to check the steps, not just the answer 5. A general-purpose LLM judge is doing outcome supervision in the harder direction (no step labels, no domain training). It loses badly to a small specialized verifier.
flowchart TD
A[Task category] --> B{Judge model<br/>can solve it?}
B -->|no| C[Judge cannot verify.<br/>Use executable check,<br/>process-reward model,<br/>or human grader.]
B -->|yes, easily| D{Reference<br/>solution available?}
B -->|barely| E[Risky. Use LLM judge as<br/>secondary signal only.<br/>Required: human spot-check.]
D -->|yes| F[Reference-anchored<br/>LLM judge; cheap, reliable.]
D -->|no| G[Direct rubric judge;<br/>calibrate TPR/TNR.]
What this means for your pipeline
Three concrete rules, applied at the task level:
Hard reasoning, math, code correctness: never LLM judge alone. If you can run the candidate's code against unit tests, do that. If you can execute the SQL against a known schema, do that. If you can run the proof through a checker, do that. The LLM judge is a secondary signal at best, useful for catching style or process violations that the executable check ignores. The primary signal is the executable check 2. A reasoning-model judge narrows the gap but does not remove it, and an executable check is right by construction.
Open-ended generation without executable ground truth: LLM judge fine, calibrated. Chat replies, summaries, brainstorming, tone, RAG faithfulness. This is the regime MT-Bench-style results apply to. Use the four-part prompt, do the train/dev/test discipline, hit 0.90 TPR and TNR before deploying.
The mixed case: route by task type. If your product spans both regimes (a coding assistant that also writes a friendly summary), do not run a single rubric. Route the coding output to executable verification and the summary output to the calibrated judge. They are different evaluators because the underlying tasks have different verifiability properties.
A worked failure: ScienceAgentBench
Chen et al.'s ScienceAgentBench is a case study in why execution beats reading 6. It has 102 data-driven science tasks drawn from 44 peer-reviewed publications, and the target output is a self-contained Python program. The primary score is a success rate computed by running the program and checking its output against task-specific criteria. A GPT-4o judge grades figure outputs only, and the authors say plainly that it and the CodeBERTScore similarity metric are not perfect.
The failure profile is the useful part. Most failed programs had correct high-level structure but implementation-level errors, such as missing steps or wrong API usage. Code like that reads right. Without executing its code, Claude-3.5-Sonnet, the strongest of the non-reasoning models compared, solved 16.7 percent of tasks; with execution feedback (self-debug) it reached 32.4 percent, and 34.3 percent with expert-provided knowledge. OpenAI o1-preview, which the authors list for reference only because it spends extra inference-time compute, went from 34.3 percent without execution to 42.2 percent with it, so the reasoning model also gained from running its code.
The lesson: a judge that reads code and says "looks right" is approximately useless for correctness, because plausible structure is exactly what the failing programs have. The fix is the same as the JudgeBench takeaway: run the code and grade by the result of execution, not by the appearance of the code.
The MT-Bench result is not contradicted
To be precise about what JudgeBench does and does not show: it does not contradict the MT-Bench finding that strong judges reach >80 percent agreement with humans on preference tasks. The two benchmarks measure different things. MT-Bench asks "which response do humans prefer." JudgeBench asks "which response is correct." Those are not the same question, and the judge's reliability differs sharply between them 1 2.
The community confusion comes from collapsing the two. Practitioners read the MT-Bench headline ("GPT-4 agrees with humans on most pairs") and infer "GPT-4 is a good judge." It is a good judge for preference. It is not a good judge for correctness.
CITATION
Tan et al. 2024, "JudgeBench: A Benchmark for Evaluating LLM-Based Judges," ICLR 2025. The empirical result that GPT-4o performs at roughly chance on hard reasoning, coding, and math judgments. The April 2025 revision adds reasoning-model judges (o3-mini high at 80.9 percent). The single most useful corrective to "just use an LLM judge." arxiv.org/abs/2410.12784
What to use instead, by task
When the judge fails, you have other tools.
- Math, formal logic. Lean, Isabelle, sympy. A theorem prover or a CAS verifies the answer.
- Code. Unit tests. The model writes the code; you run it. HumanEval, MBPP, LiveCodeBench all work this way. For trajectory-style scoring on agent code, add tool-call F1 separately.
- SQL. Execute against a sandbox schema. Spider, BIRD all work this way.
- Structured output. JSON schema validation. Regex on extractive fields.
- Math step-by-step. A process-reward model trained on step labels 5. Out of reach for most teams; the next best fallback is "run the candidate's working, check the intermediate values."
- Reasoning over a closed corpus. Atomic-fact decomposition plus per-fact retrieval (FActScore-style). The judge grades atomic claims it can check, not the whole answer.
What still requires human review
Some categories no automation reaches. Legal advice, medical decisions, novel research claims, creative work where the criterion is "good." For these, the eval pipeline is: code-based filter for obvious failures, LLM judge for the next layer of cheap signal, human expert review for the final call. The judge buys you scale on the second layer; it does not replace the expert on the third 7.
How to read your own results honestly
The JudgeBench finding produces a specific failure mode in practitioner reporting: teams report "judge agreement with humans" as a single number across all task categories in their product, and the number averages to something respectable like seventy-eight percent. The aggregate hides the directional truth. The judge agrees with humans ninety percent of the time on the preference and tone slices and forty-five percent of the time on the reasoning slice; the average means nothing because the reasoning slice is the one you care about.
The discipline is to report agreement (or TPR and TNR) sliced by task category. If your product spans categories, slice by category. The slice where the judge is at chance is the slice that needs an executable verifier or a human grader, not a recalibration.
A useful field test on your own pipeline: take a sample of fifty traces where the judge said PASS and ground truth said FAIL (false positives). If most of them are in the hard-reasoning category, you have a JudgeBench-style problem; mitigate by routing those traces to a different verifier. If they spread across categories, you have a calibration problem; revise the prompt.
What this means for benchmark consumption
Practitioners consume benchmarks too. When a vendor or a paper reports "our system reaches X percent on LLM-judged benchmark Y," check whether Y is a preference benchmark (MT-Bench, Arena-Hard, AlpacaEval) or a correctness benchmark (anything where ground truth is verifiable by execution or formal check). For preference benchmarks, the number is meaningful with caveats; for correctness benchmarks judged by an LLM, the number is approximately uninformative about whether the system actually got the answer right. The Tan et al. result is the citation you need when pushing back on an over-claim 2.
The takeaway
A strong LLM judge is a useful instrument with a real working range. Inside the range (preference, faithfulness, tone, calibrated binary rubrics on open-ended generation) it scales human review by ten or twenty times for a small fraction of the cost. Outside the range (hard reasoning, math, code, science correctness) a non-reasoning judge is close to a coin flip and still writes a confident critique, and a reasoning judge still misses about one pair in five 2. Liu et al. flag a related risk: LLM evaluators may favor LLM-generated text 8.
The discipline is to know which side of the line your task sits on before you ship. JudgeBench gave us the numbers; the rest is reading them honestly.
Once you know where the judge can be trusted, the remaining question is what you feed it, which is the business of Datasets and Golden Sets.