An agent is a system that takes multiple actions to accomplish a task: tool calls, sub-goal decomposition, plan-then-execute, retrieval-then-act. An outcome-only eval collapses everything between start and end into a single bit. The trajectory pattern scores each step plus the outcome. Yehudai et al.'s 2025 survey is the canonical reference 1.
Why outcome-only fails
Three failure modes are invisible to an outcome-only metric.
Tool-call hallucination. The agent calls a non-existent tool or calls a real tool with malformed arguments. The catch handler triggers a fallback that produces a passable outcome. Outcome: pass. Reality: the agent was wrong.
Redundant work. The agent calls the same tool five times with minor variations, processes intermediate state poorly, eventually arrives at the right answer. Outcome: pass. Cost: five times the budget. Latency: five times the SLA.
Inconsistency. The agent passes on the first run, fails on the second, passes on the third. Yao et al. introduced pass^k for tau-bench to surface this: the best gpt-4o function-calling agent averaged over 60% success on tau-retail, yet its pass^8 fell below 25% 2. The agent is not reliable; the outcome metric did not say so.
The trajectory pattern fixes all three by scoring steps and outcome together. He et al.'s TRAJECT-Bench is the explicit benchmark 3.
The four trajectory metrics
Tool-call F1. Did the agent pick the right tool with the right arguments? Compute per-step precision and recall against a gold trajectory; aggregate as F1. T-Eval and BFCL operationalize this at scale 4. BFCL is now at V4, which adds agentic web-search and memory categories to multi-turn calling; its leaderboard showed a last-updated date of 2026-04-12 (fetched as of 2026-10-01).
Plan correctness. Score the plan separately from the execution. A wrong plan executed correctly is a different failure from a right plan executed incorrectly.
Outcome correctness. Did the task complete. Necessary, not sufficient.
Pass^k consistency. Run the agent k times. Report the probability all k succeeded. Yao et al.'s tau-bench result: the best gpt-4o function-calling agent, above 60% on average success, drops below 25% at pass^8 on tau-retail 2. The gap is the reliability gap operational systems care about.
| Metric | What it catches | Computed against |
|---|
| Tool-call F1 | wrong tool, wrong args | gold trajectory |
| Plan correctness | wrong plan, right execution | gold plan or rubric |
| Outcome correctness | task completion | task verifier |
| Pass^k | inconsistency | k independent runs |
A complete agent-eval report ships all four.
Figure: Agent trajectory evaluation scores each part of a run instead of only the end state: plan correctness against a gold plan or rubric, tool-call F1 against gold (tool, args) pairs, and outcome correctness from a task verifier. Pass^k then repeats the whole run k times; on tau-bench, the best gpt-4o agent at over 60 percent pass@1 falls below 25 percent at pass^8.
Pass^k in detail
For an agent with per-run success probability p (assumed independent across runs), pass^k = p^k.
| p (per-run) | pass^1 | pass^2 | pass^4 | pass^8 |
|---|
| 0.95 | 0.95 | 0.90 | 0.81 | 0.66 |
| 0.85 | 0.85 | 0.72 | 0.52 | 0.27 |
| 0.70 | 0.70 | 0.49 | 0.24 | 0.06 |
| 0.50 | 0.50 | 0.25 | 0.06 | 0.004 |
A team shipping an agent at 70% pass@1 has a one-in-four shot at four consecutive successes. For a customer-facing flow where reliability matters, pass^4 is the headline metric, not pass@1.
The table is the uniform-difficulty case. tau-bench computes pass^k per task, as the chance that all k sampled trials succeed (C(c,k)/C(n,k) from c successes in n trials), then averages across tasks 2. Real pass^k therefore sits at or above p^k, and closes on p when tasks are either always solved or never solved. Compute it per task; do not raise the average to the k-th power. The pass^k chapter walks the implications.
Computing tool-call F1 requires a gold trajectory: a labeled sequence of (tool_name, arguments) pairs a correct agent should produce. BFCL provides labeled calls for a curated set of API-calling tasks 4; for internal benchmarks, the gold trajectory comes from a human-expert demonstration.
def tool_call_f1(gold: list[dict], pred: list[dict]) -> float:
def normalize(call):
return (call["tool"], frozenset(call["args"].items()))
g = set(normalize(c) for c in gold)
p = set(normalize(c) for c in pred)
if not g or not p: return 0.0
tp = len(g & p)
prec, rec = tp / len(p), tp / len(g)
return 2 * prec * rec / (prec + rec) if (prec + rec) else 0.0
Practical extensions: argument-similarity tolerance (gold "user_id=alice" vs predicted "user_id=Alice"), order tolerance (some sequences require exact order, others permit any topological order on the dependency graph).
Gold-trajectory matching has a cost. Anthropic's January 2026 guide to agent evals reports that checking for a specific sequence of tool calls is too rigid and produces brittle tests, because agents regularly find valid approaches the eval designer did not anticipate. It recommends grading what the agent produced, meaning the final state of the environment, and building in partial credit 5. Use tool-call F1 as a diagnostic where the tool set is genuinely constrained, such as an API-calling benchmark or a policy that mandates a call, and let outcome verifiers carry pass/fail. BFCL draws the same line: it verifies calls by AST match or by environment state transition, not by trajectory similarity 4.
Agent-as-judge
When outcomes are subjective, use another agent to judge the trajectory. Zhuge et al. formalize this as "Agent-as-a-Judge" and report, on DevAI (55 code-generation tasks with 365 hierarchical requirements), that it is as reliable as their human-evaluation baseline 6. The evaluator agent has access to the trace (input, plan, tool calls, intermediate state, output), runs against a rubric, returns scores per step plus an overall judgment.
The same LLM-as-Judge discipline applies. Calibrate against the principal expert on a hundred-trace labeled set; measure TPR and TNR; recalibrate monthly. The LLM-as-Judge page covers the discipline.
Reward-hacking probes
The agent learns to satisfy the metric rather than the underlying intent. Pan et al. document "in-context reward hacking" where an agent in a feedback-loop eval exploits the metric's definitions 7. Three probes. Specification gaming: construct a task with a metric that has an obvious exploit, watch whether the agent finds it. Feedback-loop: iterative eval where the agent sees its score after each attempt, measure whether scores improve via exploit or improvement. CoT faithfulness: check whether the stated reasoning matches actual behavior; if the agent says "I'll call tool X" then calls Z, the chain-of-thought is unfaithful. A clean chain of thought is weak evidence, though: in Anthropic's study Claude 3.7 Sonnet mentioned a planted hint it had used 25% of the time and DeepSeek R1 39% of the time, and in reward-hacking setups the models verbalized the hack less than 2% of the time in most scenarios 8. Audit actions and environment state, not just the stated reasoning. These are red-team-style audits on a quarterly minimum for safety-critical agents.
Environmental drift
An agent depends on an environment (tools, APIs, retrieval index). When the environment changes, an agent that calibrated cleanly six months ago may fail predictably. Pin the tool version, API schema, and index snapshot for every eval run; note them in the trace. When production drifts, re-run and compare. He et al.'s TRAJECT-Bench covers a neighboring axis, scoring tool selection, argument correctness, and order satisfaction on parallel and interdependent tool trajectories 3; the drift test itself, swapping a tool version or schema between runs and re-scoring, is yours to build.
Benchmarks drift too. The tau-bench line moved to tau2-bench (a dual-control telecom domain where the user also acts on the environment) and then tau3-bench. Its v1.0.0 (18 March 2026) fixed 53 airline and retail tasks with wrong expected actions or ambiguous user instructions; on the three models Sierra re-ran, airline pass^1 rose 14 to 20 points, and GPT-5.2 (high) went from 50% to 72% at airline pass^4 9. v1.0.1 (22 July 2026) regraded the banking_knowledge domain, and the maintainers say results from before 1.0.1 are not comparable 10. Wrong expected actions depress pass^k as well as pass^1, so record the benchmark version and task-set commit in every trace, next to the tool and index versions.
Observability
Moshkovich et al.'s 2025 work treats observability as the production-time counterpart to trajectory eval: agent runtime logs, built on OpenTelemetry-style spans and traces, feed analytics that discover flows and issues 11. Log each tool call, intermediate state, and decision as a structured span; a stored trace can then be re-scored against an updated rubric without re-running the agent. Schema covers full prompt and response per turn, tool name and arguments per call, intermediate state, plan if surfaced, outcome verdict. Mohammadi et al.'s survey covers the choices 12.
What good looks like
A production agent eval program ships a gold-trajectory benchmark of 100-300 tasks with labeled (tool, args) sequences, reporting of tool-call F1, plan correctness, outcome correctness, and pass^k for k>=4, quarterly reward-hacking probes, trajectory traces with replayability against updated rubrics, and pinned environment versions in every eval run. The trajectory vs outcome chapter walks the conceptual setup; this page is the methodology reference.