An agent run is a sequence: thought, tool call, observation, response, repeat. A correct final answer can ride on a broken plan that hit the right cell of the search space by luck. A wrong final answer can come from a perfect plan whose last tool call returned a 503. Scoring only the last token in that sequence is the easiest way to ship an agent that looks fine on the leaderboard and collapses the moment your tools behave differently in production.
The argument of this page is that you need two grades, not one: outcome success and trajectory quality. They disagree often enough that picking the wrong one can rank your candidate models in the opposite order from the one your users would prefer. TRAJECT-Bench grades 5,670 queries over 1,228 executable tools and reports tool selection, argument correctness, and dependency order next to final accuracy, which is how it surfaces failure modes a final-answer score cannot localize: confusing tools with overlapping capabilities, and picking a tool from its description without checking the parameters it needs 1. The two main agent-eval surveys point the same way. The Yehudai survey (revised April 2026) describes final-response, stepwise, and trajectory-based assessment as the three levels evaluation frameworks support and names cost-efficiency, safety, robustness, and fine-grained evaluation as the open gaps 2; the Mohammadi survey organizes the field along evaluation objectives (behavior, capabilities, reliability, safety) and evaluation process, not a single score 3.
What "trajectory" means
The unit you score is the span tree: the root request, the model's plan, every tool call with its arguments, every tool response, every model turn in between, and the final response. In OpenTelemetry terms, the trajectory is the trace; each step is a span; the per-step attributes are the arguments, the result, the latency, the token count. If you cannot reconstruct the trajectory from your traces, you cannot do trajectory-level eval; instrumentation is the prerequisite. OpenTelemetry's GenAI conventions (status: Development) now name these spans: invoke_agent, plan, and execute_tool, with tool-call arguments and results as opt-in attributes (gen_ai.tool.call.arguments, gen_ai.tool.call.result) 4. Opt-in matters here: default traces carry the tool name but not the arguments that argument-level scoring needs, so switch them on in the environment you eval in. The trace schema page covers what else to capture.
A useful mental split, in line with the stepwise and trajectory-based assessment the Yehudai survey describes 2 and exercised by multi-environment suites like AgentBench 5 and tool-heavy ones like GAIA 6:
- Plan quality. Did the agent decompose the task correctly before acting? Are there obvious missing sub-goals or wrong dependencies?
- Tool selection. Did it pick a reasonable tool at each step, including the decision not to call a tool when answering from memory was correct?
- Tool argument correctness. When it called the tool, were the arguments grounded in the user's request and the prior context?
- Error recovery. When a tool returned an error, did the agent retry, replan, or escalate, instead of silently fabricating a result?
- Final answer. Did the response actually solve the user's task?
You score these independently because they fail independently. A model that picks the right tools but supplies hallucinated arguments fails differently from a model that picks the wrong tool every time but happens to get the right answer through compensating tries.
TIP
If the trajectory-level score and the outcome score never disagree on your set, your trajectory scorer is too loose. Real agents pass for the wrong reasons and fail for the wrong reasons. Rubric tightening, not metric removal, is the response.
Why outcome-only ranks models in the wrong order
Three concrete reasons.
First, many agent tasks have several routes to the same end state and a policy that rules out some of them. τ-bench grades by comparing the database at the end of the conversation with an annotated goal state, and its authors note that a matching state is necessary but not sufficient: an agent can process a return without explicit user confirmation, violate policy, and still land on the goal state 7. Outcome scoring rewards "the database ended in the right state". Operationally, you also need "no policy violations along the way". Two agents with identical outcome scores can have very different policy-violation rates; only the trajectory score distinguishes them.
Second, contamination and overfitting hide in outcome metrics. GAIA-style benchmarks contain widely-known reference answers 6. An agent that has memorized the final answer will hit it from a plan that no domain expert would endorse. A judge that walks the trajectory catches the gap between "answer is right" and "answer was earned"; an outcome metric cannot.
Third, error recovery matters more in production than in static evals because production tools fail. TRAJECT-Bench discards tools that error out during curation and scores selection, arguments, and order over the ones that execute cleanly 1, so recovery rate is a number you have to produce in your own harness (see environmental drift). If your model is excellent at first-try success but never recovers from a 429, you will see "win" on a clean benchmark and "loss" on a real workload.
Figure: Outcome success versus trajectory quality as a 2x2 matrix. Outcome-only agent evaluation cannot tell an earned pass from a lucky pass (right answer off a memorized or policy-violating path) or an earned fail from an unlucky fail (sound plan that hit a 503), which is why agents need both an outcome grade and a trajectory grade.
State-check functions: the minimum viable trajectory scorer
Before you reach for an agent-as-judge, write deterministic state checks. They are cheaper, faster, and catch the largest single class of failure: the agent claimed success but the world is in the wrong state. The Anthropic guide to building effective agents argues for the simplest harness that does the job; for evaluation, the simplest harness is a sandbox and an assertion 8.
The pattern is: run the agent against a sandboxed environment, snapshot the relevant state at the end, and check it against an expected state. The check returns a boolean for outcome success plus a structured record of which sub-goals fired, in what order, and which post-conditions hold.
"""
Minimum-viable trajectory scorer for a retail-support agent.
Run the agent in a sandbox, then check the resulting state.
"""
from dataclasses import dataclass, field
from typing import Callable
@dataclass
class Trajectory:
user_request: str
steps: list[dict] # each step: {tool, args, result, latency_ms}
final_response: str
@dataclass
class TaskCase:
name: str
user_request: str
setup: Callable[["Sandbox"], None]
check_outcome: Callable[["Sandbox"], bool]
expected_subgoals: list[str] # ordered sub-goals
forbidden_actions: list[str] = field(default_factory=list)
@dataclass
class TrajectoryScore:
outcome_pass: bool
subgoal_hits: list[str]
subgoal_misses: list[str]
policy_violations: list[str]
redundant_calls: int
recovered_from_error: bool
def score(traj: Trajectory, case: TaskCase, sandbox) -> TrajectoryScore:
tools_called = [s["tool"] for s in traj.steps]
subgoal_hits, subgoal_misses = [], []
for goal in case.expected_subgoals:
if goal in tools_called:
subgoal_hits.append(goal)
else:
subgoal_misses.append(goal)
policy_violations = [
a for a in case.forbidden_actions if a in tools_called
]
# redundant: same tool called with same args twice in a row.
redundant_calls = 0
for i in range(1, len(traj.steps)):
prev, cur = traj.steps[i - 1], traj.steps[i]
if prev["tool"] == cur["tool"] and prev["args"] == cur["args"]:
redundant_calls += 1
# recovery: any step whose result is an error followed by a non-error
# call to a different tool or same tool with changed args.
recovered_from_error = False
for i in range(len(traj.steps) - 1):
cur, nxt = traj.steps[i], traj.steps[i + 1]
is_error = isinstance(cur["result"], dict) and cur["result"].get("error")
recovered = (
nxt["tool"] != cur["tool"]
or nxt["args"] != cur["args"]
)
if is_error and recovered:
recovered_from_error = True
break
return TrajectoryScore(
outcome_pass=case.check_outcome(sandbox),
subgoal_hits=subgoal_hits,
subgoal_misses=subgoal_misses,
policy_violations=policy_violations,
redundant_calls=redundant_calls,
recovered_from_error=recovered_from_error,
)
That gives you five numbers per run instead of one. Aggregate across your set and report the joint distribution: outcome pass-rate by sub-goal completion bucket, by policy-violation count, by recovery success. The headline is the outcome, but the diagnostic is the trajectory.
Three cautions before you trust those numbers.
First, a state check is only as good as its goal state. When Sierra audited τ-bench for the τ³ release, it corrected 27 of the 50 airline tasks and 26 of the 114 retail tasks, and airline pass^1 rose by 14 to 20 points depending on the model once agents stopped being penalized for following the domain policy 9. Read the failing trajectories before you believe a failing score. Tool-call evaluation covers a grader bug from the same family.
Second, do not weight every step equally. SABER's analysis of τ-bench airline and retail traces found that each additional deviation in a mutating (environment-changing) action cut the odds of success by up to 92 percent on airline and 96 percent on retail, while deviations in non-mutating steps had little to no effect 10. Grade write calls strictly (exact arguments, preconditions met) and count extra reads as cost, not failure. Treat the redundant-call counter above as a cost metric for reads, not a pass/fail gate; a repeated write belongs with the policy violations.
Third, do not over-correct into grading the route. Anthropic's January 2026 guidance on agent evals says that asserting a fixed sequence of tool calls produces brittle tests, because agents regularly find valid approaches the designer did not anticipate, and recommends grading what the agent produced, with partial credit for multi-part tasks 11. The same post describes Opus 4.5 finding a loophole in a τ2-bench flight-booking policy and failing the task as written with what Anthropic judged a better solution for the user. Whether a loophole is a better solution or a violation is a policy decision; make it, then encode it, in the forbidden-actions list if it is a violation and in the accepted end states if it is not. Grade the destination first, then grade the path only on properties that hold on every acceptable route: no forbidden tool, a lookup before each write, no repeated mutation. The scorer above matches sub-goals by membership, not order. The platforms the Yehudai survey catalogs (LangSmith, Vertex AI, AgentEvals) offer exact, partial, unordered, and subset matching of reference trajectories, and the survey notes that reference-based methods are limited because multiple valid paths typically exist 2. Use exact-order matching only where the order is the policy.
When you need an agent-as-judge
State checks cover the deterministic-environment case. They do not cover open-ended tasks like "research this topic and summarize sources", or trajectories long enough that hand-writing sub-goal lists scales badly. For those, you need a judge that walks the trajectory and grades each step against a rubric.
The Zhuge et al. Agent-as-a-Judge work shows that a judge with its own tools, evaluating step by step on 55 realistic AI development tasks, approaches human-evaluator reliability on tasks where a single-pass LLM judge breaks down 12. Their finding is that single-LLM judges cannot reliably reason about long trajectories; the judge needs to act like an agent itself, opening files, running code, checking intermediate state.
In practice, three patterns work:
- State-check first, judge second. If the state check passes, you do not need the judge. If it fails, run a judge to attribute the failure to plan / tool selection / argument / recovery.
- Independent per-step grades. Score plan quality, tool selection, argument grounding, and recovery separately. Aggregate at the end. Joint grading of "was the whole trajectory good" produces vague labels.
- The judge gets the tools too. Without tool access, a judge cannot verify "the file the agent claimed to write actually has the right contents". Read-only sandbox access to the same tool set as the agent is the minimum.
Inter-rater agreement is the first sanity check
Before any of this is automated, two domain experts should grade the same 20 trajectories. The Yehudai survey notes that final-response evaluation is fast, inexpensive, and easy to automate but cannot assess intermediate decisions, and it flags the cost of running expensive LLM judges over large numbers of traces as a gap current frameworks overlook 2. Step-level grading is where that cost lands, for human annotators as much as for judges, and you should expect inter-rater κ at step granularity to come in lower than at the outcome level on a first pass. That is not a rater problem; it is a rubric problem. If two trained humans cannot agree on whether the plan was reasonable, no automated judge can either, and the right move is to sharpen the rubric, not to swap models.
The cheapest version of this: label 20 trajectories with two annotators, compute κ on (a) outcome success and (b) overall trajectory quality, then on each per-step axis. Step-level κ below 0.5 means the rubric for that step is under-specified. Fix the rubric before automating anything downstream.
What to report
For every model and every dataset, report the vector, not the scalar:
- Outcome pass-rate, with a 95% confidence interval.
- Sub-goal completion rate.
- Mean number of redundant tool calls per run.
- Recovery rate on injected errors.
- Policy-violation rate (any forbidden tool / argument).
- Median and p95 step count.
- Median and p95 cost per run.
Anyone asking for a single number is asking you to hide one of these. Refuse the question or, at minimum, name the trade you are making.
What this section covers next
The trajectory split is the frame. The chapters that follow drill into the parts. Tool-call evaluation gets its own page; pass^k handles the inconsistency problem that even a perfect trajectory scorer cannot detect from a single attempt; reward hacking is the failure mode trajectory scoring is most likely to miss when the judge itself is the target of the optimization. Read those next if you are designing a harness from scratch.