An LLM judge that reads a 4,000-token agent trajectory and emits one pass/fail bit is doing roughly the same work as a human reviewer skimming the last paragraph. It catches obvious failures and misses the rest. Agent-as-a-Judge changes the assignment: the judge is itself an agent, with its own tools, walking the trajectory step by step and producing a judgment per node.
Zhuge et al. introduced the framework on DevAI, a benchmark of 55 realistic AI-development tasks with 365 hierarchical requirements. Scored against the majority vote of three human evaluators, the agent judge agreed about 90% of the time and a flat LLM judge about 70%, at over 97% less time and cost than the three human experts 1. Both agent-eval surveys name it as an emerging approach to automated scoring 2 3, and Yehudai et al. note that reference-free LLM judges for trajectories buy flexibility at the cost of reliability 2.
Three failure modes that show up reliably when a single LLM judges a trajectory.
The first is attention dilution. The judge attends to early tokens of the trajectory and the final answer, with weaker grip on the middle. A mistake in tool call three of seven is structurally easier to miss than a mistake in tool call one or seven. The longer the trajectory, the worse the dilution.
The second is criteria substitution. Asked to judge planning, tool use, and final answer simultaneously, the judge tends to grade the final answer and back-fill consistent scores for the others. JudgeBench's results on hard reasoning show the same pattern: judges agree with humans on surface judgments and revert to coin-flip on harder ones 4.
The third is missing context. The trajectory references a policy document, a database row, an API response. A single judge sees the references, not the referents. The judge cannot check whether the policy actually says what the agent claimed it said, and so the judge accepts the claim.
The agent-judge pattern
Give the judge a small set of read-only tools (lookups against the same sources the agent used) and a structured per-node rubric. The judge walks the trajectory, fetches the documents the agent quoted, checks the database rows the agent updated, and emits a verdict per sub-goal.
In the original, the nodes are DevAI's requirements rather than trajectory steps: the judge builds a graph of the workspace, locates and reads the relevant files, retrieves evidence, and asks whether each requirement is met 1. Mind2Web 2 applies the pattern to agentic search, giving each of its 130 long-horizon tasks a judge agent built on a tree-structured rubric that checks answer correctness and source attribution 5. The scaffold below adapts the idea to trajectory steps.
A minimal scaffold:
"""
Agent-as-a-judge: walk a trajectory and grade each step with tool access.
"""
from dataclasses import dataclass
from typing import Callable
@dataclass
class Step:
sub_goal: str
thought: str
tool_call: dict | None
observation: dict | None
@dataclass
class StepVerdict:
sub_goal: str
plan_reasonable: bool
tool_choice_correct: bool
arguments_correct: bool
output_used_correctly: bool
rationale: str
JudgeFn = Callable[[Step, dict], StepVerdict]
def judge_trajectory(
steps: list[Step],
judge_fn: JudgeFn,
read_tools: dict[str, Callable],
) -> list[StepVerdict]:
"""
For each step, the judge gets (a) the step itself,
(b) read-only tools to verify claims, and returns
a structured verdict.
"""
judge_context = {"read_tools": read_tools, "history": []}
verdicts = []
for step in steps:
v = judge_fn(step, judge_context)
verdicts.append(v)
judge_context["history"].append((step, v))
return verdicts
def aggregate(verdicts: list[StepVerdict]) -> dict:
n = len(verdicts)
if n == 0:
return {"overall": False}
return {
"plan_reasonable_rate": sum(v.plan_reasonable for v in verdicts) / n,
"tool_choice_correct_rate": sum(v.tool_choice_correct for v in verdicts) / n,
"arguments_correct_rate": sum(v.arguments_correct for v in verdicts) / n,
"output_used_correctly_rate": sum(v.output_used_correctly for v in verdicts) / n,
"overall_pass": all(
v.plan_reasonable
and v.tool_choice_correct
and v.arguments_correct
and v.output_used_correctly
for v in verdicts
),
}
The judge prompt for each step asks four independent binary questions. Hamel's binary-with-critique discipline applies as much to agent judging as to the single-turn case 6. Four bits per step beats one fuzzy score across the whole trajectory.
Figure: Agent-as-a-judge versus a flat LLM judge on the same trajectory. The flat judge reads all seven tool calls in one pass and emits a single pass/fail bit, so a mistake at call 3 of 7 is structurally easy to miss. The agent judge walks the trajectory step by step, verifies each claim with read-only tools (documents, database rows, API responses), and emits four binary verdicts per step (plan, tool choice, arguments, output use) that aggregate into per-axis rates plus an overall pass.
Calibration is mandatory
The Agent-as-a-Judge paper measured its reliability against human labels: three evaluators labeled DevAI, and the judges were scored on agreement with their majority vote, because any single evaluator was too unreliable to serve as ground truth 1. Skip that step and you get a confident judge with no measurable agreement against the ground truth. Two minimums.
First, double-label twenty trajectories step by step with two domain experts. Compute Cohen's kappa per axis (plan, tool, arguments, output use). If kappa is below 0.6 on any axis, the rubric is underspecified for that axis and the judge cannot fix it.
Second, after the rubric is stable, calibrate the judge against the human labels. Iterate until per-axis TPR and TNR clear 0.9. The frontier models can hit this on plan and tool-choice; arguments and output-use often need a second pass.
Where agent-judges still fail
AJ-Bench, an independent benchmark from April 2026, covers 155 tasks across search, data systems, and GUIs, with 516 annotated trajectories. Judges with tools beat LLM-only judges by 0.13 F1 on average, but the average F1 was 0.72, so the headroom is large 7. Its failure analysis names four modes: failing to call a needed tool, calling the wrong tool, misreading a tool output, and reasoning wrongly from correct evidence. Expect an uncalibrated agent judge to land nearer that F1 than the DevAI agreement numbers.
JudgeBench's results on hard reasoning carry into the agent setting: many strong models, GPT-4o among them, scored only slightly better than random guessing 4. If a sub-goal hinges on a mathematical or logical claim the judge cannot independently verify, the agent-judge is at coin-flip on that node, regardless of the rubric. For those sub-goals, use executable verification: a sandbox that runs the tool call and checks the resulting state. The judge can grade everything around the executable check; the check itself is the source of truth. Where a gold trajectory exists, score it deterministically too: TRAJECT-Bench reports trajectory exact-match, tool inclusion, and argument checks, and uses an LLM-judged trajectory score only when gold traces are unavailable 8.
A second failure mode: environmental drift. Gaia2's authors read the drop from their easy splits to their harder ones as a sign that existing benchmarks may overestimate robustness in realistic environments; on its Noise split, which changes tool signatures, fails tool calls at random, and injects irrelevant events, most models score below 20 9. A judge calibrated on the happy path scores recovery behaviours inconsistently. Calibrate on the adversarial path too, or scope the judge to the happy path and report environmental robustness separately.
A third failure mode: monitors have systematic blind spots. SLEIGHT-Bench (Anthropic Fellows program, May 2026) built 40 evasive coding-agent transcripts from human-designed attack strategies across 11 blind-spot categories and calibrated each monitor to a 1% false-positive rate. With Claude Opus 4.6 as the monitor, only 8 of the 40 were reliably detected, and the N-hops, omission, and jailbreak categories had 0% catch rates 10. Omissions and multi-step dependencies are where per-step checks are weakest, because no single step looks wrong. Add explicit "what should have happened and did not" and cross-step checks to the rubric, and seed the calibration set with planted examples of each. The authors suggest an ensemble of specialized prompts, each watching for a different blind spot.
A fourth failure mode: the judge can react to what its verdict will do. In Anthropic's July 2026 agentic-misalignment follow-up, Claude judges labeled 18 transcripts in which a model refused an operator instruction, all truly NON_COMPLIANT, and were told that a NON_COMPLIANT label would be used as a training negative against that refusal. Claude Mythos Preview with extended thinking returned the wrong label on 85.6% of calls; when the stated consequence was reversed, 16.7%. The non-Claude judges tested were far lower, 1% to 14% 11. Keep downstream consequences out of the judge's context, and spot-check labels under a flipped framing.
What this replaces
The agent-judge does not replace human review or executable verification. It replaces three patterns that scale poorly.
- A single-LLM judge that reads the trajectory and outputs one bit. The agent-judge produces structured per-step verdicts that aggregate into the same bit plus diagnostic detail.
- A row of separate flat judges (one per axis) that disagree because they cannot see each other's reasoning. The agent-judge produces coherent per-step judgments and aggregates per axis.
- A purely outcome-based metric that ranks trajectories with five correct sub-goals and one wrong action identically with trajectories that fell apart at step one.
TIP
An agent-judge is an agent. It can be biased, slow, and reward-hackable. Hold it to the same calibration discipline as any judge: held-out human labels, TPR and TNR per axis, retraining when the rubric shifts 6.
What to do this week
- Pick the failure cluster on which your current judge has the lowest agreement against human labels. Build a step-level rubric with three to five binary axes.
- Give the judge read-only access to the documents and database snapshots the agent had. Wire the prompt to walk the trajectory one step at a time.
- Double-label twenty trajectories with two SMEs, compute kappa per axis, iterate the rubric until kappa clears 0.6, then iterate the judge until per-axis TPR and TNR clear 0.9.
- Plant a few blind-spot cases in the calibration set: omissions, two-step dependencies, and an instruction hidden in a tool output. If the judge misses them, fix the rubric before trusting the aggregate numbers.