What "good" looks like
A long-horizon agent eval scores whether the agent succeeds consistently, not once. The right primary metric is pass^k: the probability that the agent succeeds on every one of k independent trials of the same task. Pass@1 is a vanity metric here, because real users do not retry forever.
The ship targets on a task set:
- Report pass^k with k chosen to match the user experience. If a real user gives up after four attempts at the same goal, evaluate at k=4 or higher.
- Report pass@1 alongside pass^k for the same task set. If pass^k is significantly below pass@1^k, the agent is non-deterministic in production-meaningful ways.
Why pass@1 misleads. Two agents can both hit 60% pass@1, and one can be production-ready while the other is not. The single number hides how often the agent repeats a success.
The evidence. The Sierra tau-bench paper introduces pass^k precisely because it surfaces inconsistency that pass@1 hides. On retail tasks, the headline result is that GPT-4o pass^8 is below 25%, despite a much higher pass@1[1]. That gap between a strong single-shot score and a weak repeated-success score is the signal to fix.
Time horizons show the same gap. METR's time horizon is the human-expert task duration at which an agent is predicted to succeed at a stated reliability, so the reliability level is part of the claim[2]. In METR's Time Horizon 1.1 data (page last updated 2026-05-08, fetched as of 2026-10-01), the 80% horizon sits far below the 50% horizon: about 17.4 hours against 3.1 hours for Claude Mythos Preview (early), and about 5.7 hours against 54 minutes for GPT-5.4[3]. METR also flags measurements above 16 hours as unreliable on its current task suite, and its data file gives the 17.4-hour estimate an interval of roughly 8.5 to 55 hours[3]. When you quote a horizon, quote the reliability level with it, and size your own tasks to the 80% figure if your users will not tolerate a coin flip.
flowchart TD
TASK["Multi-step user goal"] --> RUNS["Run k=5 or k=8 trials"]
RUNS --> STATE["Deterministic state check per trial"]
STATE --> P1["pass@1: any single trial"]
STATE --> PK["pass^k: every trial succeeds"]
P1 --> GATE{"pass^k far below pass@1^k?"}
PK --> GATE
GATE -->|yes| FIX["Fix inconsistency: the bug"]
GATE -->|no| SHIP["Ship candidate"]
Figure: Pass^k versus pass@1 for long-horizon agent evals: two agents with the same 60% pass@1 diverge as trials grow, and fully independent trials collapse to 1.7% by k=8. tau-bench reports GPT-4o pass^8 below 25% on retail tasks despite a much higher pass@1, which is why pass^k is the primary metric.
Secondary metrics
| Metric | What it catches | How to score |
|---|
| Steps to completion | Bloated or looping trajectories | Median and 90th percentile turns per successful task |
| Replanning frequency | Instability in the plan | How often the agent changes its declared plan mid-execution |
| Error attribution | Where the failures come from | Tag every failed trajectory with a cause. tau-bench's own breakdown of GPT-4o function-calling failures in retail is wrong argument or information (about 55% of failures), wrong decisions against the policy (25%), and partial resolution of compound requests (19%)[1] |
| Cost per successful task | Economically different products at the same pass rate | A 90% pass^k at 0.40 |
Offline eval design
Standard benchmarks first. The tau-bench retail and airline domains are the closest things to a standard for this category, and they now live in the successor repo: sierra-research/tau-bench is frozen with a maintainer warning that its tasks are outdated, so run both domains from sierra-research/tau2-bench (currently shipping tau3-bench; pin a tag)[4]. Its v1.0.0 release (March 2026) made 75+ task fixes across airline, retail, and banking, and v1.0.1 (July 2026) corrected banking_knowledge grading: the maintainers say results from before 1.0.1 are not comparable on that domain, and the other domains are unaffected (README fetched as of 2026-10-01)[4]. The leaderboard submission rules strongly prefer at least 4 trials per domain, while the landing page ranks by pass^1, so compute pass^k from your own multi-trial runs with --num-trials 4 or higher[4]. Run both. AgentBench is broader and covers 8 environments including a SQL agent, an OS agent, and a web shopping agent[5]; pick the subset that matches your product. Its main branch now ships AgentBench FC (October 2025), a function-calling variant with containerized runs for five of those tasks; the original eight-environment version sits on the v0.1 and v0.2 branches, so record which one you ran[6].
Then a domain harness. Build 20 to 50 of the most common multi-step user goals, each encoded as a deterministic state-check function (for example, did the row appear in CRM with the expected fields). State checks are what make the eval reusable; LaaJ on the final agent message is too noisy at this scope.
Run and report. Run k=5 or k=8 trials per task and report pass@1 and pass^k together. The variance across trials is the eval; the headline is the summary.
Online eval design
Capture full trajectories. Log every tool call, observation, and intermediate plan snippet, with state snapshots at known checkpoints.
Sample for review by three stratifications:
- Outcome bucket (success, partial, failure).
- Steps-decile (anomalously long trajectories often hide loops).
- Any tool error, which is a leading indicator of pipeline drift.
Reserve judges for samples. LaaJ on the plan or critique is expensive; use it for sample reviews, not full traffic.
Alert on drift. Flag cost-per-task drift greater than 20% week over week, and retry-loop patterns (the same tool with the same arguments three times in a row).
Top benchmarks
- tau2-bench (successor line), https://github.com/sierra-research/tau2-bench. Retail and airline with maintained task fixes, plus telecom and banking_knowledge; latest release v1.0.1 (July 22, 2026). The original tau-bench introduced the pass^k metric and the user-simulator strategy, and is now frozen[4].
- AgentBench, https://github.com/THUDM/AgentBench. 8 environments in the paper; the current branch is the function-calling AgentBench FC (October 2025). Useful for breadth[6].
- GAIA, https://arxiv.org/abs/2311.12983. 466 questions designed for general assistants; humans hit 92%, GPT-4 with plugins around 15% in the original paper[7]. The public test leaderboard has since passed the human figure: the best entry scores 93.69% (submitted 2026-09-22) and the top ten fall between 92.0% and 93.7% (fetched as of 2026-10-01)[8]. Use it as a smoke test, not a ranking.
- tau2-bench harness: pass^k on retail and airline tasks, computed by the CLI.
- AgentBench: cross-environment breadth.
- Inspect AI: research-grade trace capture with strong sandbox primitives.
- Langfuse or Braintrust: production trajectory logging and trace search; both have annotation queues that work for the sample-review pattern.
- OSWorld: sometimes used in this category, though it is browser- and OS-focused enough to merit its own playbook[9].
Common pitfalls
| Pitfall | Why it bites | Fix |
|---|
| Reporting only pass@1 | The most common error in this category; it masks the inconsistency that makes agents fragile in production | Report pass^k; tau-bench's whole framing exists because of this gap[1] |
| No state checkpointing | If you cannot replay a failed trajectory from a known mid-state, your eval is a black box | Snapshot state at known checkpoints so a failed run is replayable |
| Conflating "completed" with "completed correctly" | Many agents end with confident success messages that are wrong | Use a deterministic state check as the only pass signal |
| Letting LaaJ-on-trajectory replace deterministic verifiers | Judges of long multi-step traces have high variance and unstable cost | Use them on samples only |
Canonical reading
tau-bench is the most useful single read because it both defines the metric this task needs and instruments a credible user simulator alongside it[1]. The abstract is direct about the takeaway: the findings point to the need for methods that make agents act consistently and follow rules reliably. That is the operational thesis of this whole playbook.
Day-1 starter
- Encode 20 of your most common multi-step user goals as state-check functions. The function takes the agent's final state and returns pass or fail; no LLM, no judge.
- Log every tool call, observation, and plan snippet to one trajectory row per attempt. State snapshots at known checkpoints so a failed run is replayable.
- Run k=5 trials per task on every release candidate; report pass@1 and pass^5 side by side. If they diverge, the agent's inconsistency is the bug to fix.
Cross-cutting links