CAUTION
Run the successor, not this repo. sierra-research/tau-bench is frozen: its last functional commit was August 2025, and in March 2026 its maintainers added a README warning that its airline and retail tasks are outdated, pointing users to the successor line at sierra-research/tau2-bench 1. That repository now ships τ³-bench (release v1.0.0, March 2026; v1.0.1, July 2026, still the latest tag as of 2026-10-01), which adds a banking_knowledge domain, full-duplex voice evaluation, and 75-plus task-quality fixes across airline, retail, and banking 2. There is no tau3-bench repository; the URL is unchanged. The method below is unchanged and the metric is the same, but point your clone at the successor, pin a tag, and record it: the maintainers state that banking_knowledge results before and after v1.0.1 are not comparable.
τ-bench is the cleanest available benchmark for "does the agent actually get the job done across multiple trials" because it ships two production-shaped domains (retail and airline), a deterministic state checker, and a user-simulator strategy that makes the conversation reproducible 3. The reason this recipe exists is the consistency story: the paper shows that agents that solve a task once routinely fail it when you ask for the same result across several trials. That is pass^k, and pass^k is the headline metric for any product where consistency matters more than peak capability.
This recipe runs the retail subset against your agent (any agent that conforms to the τ-bench interface). It reports pass@1, pass^4, average steps, and the cost per trajectory. The goal is to make the consistency drop visible: if your agent posts 70% pass@1 but 20% pass^4, you have a flaky agent, not a low-skill agent.
Background: what pass^k catches
Pass@1 is the fraction of times the agent solves the task in one trial. Pass^k is the fraction of times the agent solves the task in all k trials.
If the agent is a fair coin per attempt, pass^k decays geometrically as p^k. A 0.70 pass@1 collapses to 0.24 at pass^4. The τ-bench paper documents this for the strongest agent it tested: GPT-4o with function calling averages above 60% task success on retail, and its pass^8 falls below 25% 3. The original repository's leaderboard shows the same shape at k=4: GPT-4o function calling on retail posts 0.604 pass^1 and 0.383 pass^4 1. The implication is operational: for any product that runs the same agent more than once per user, the metric that predicts user experience is pass^k, not pass@1. Anthropic's agent-eval guide draws the same line, pass@k for tools where one success is enough and pass^k for customer-facing agents where users expect reliable behavior every time 4. The leaderboard tables on taubench.com list pass^1 (as of 2026-10-01), so the consistency number has to come from your own multi-trial runs 5.
Figure: pass@1 vs pass^k on the same 8 trials: the agent runs each of two tasks 4 times, pass@1 averages per-trial success (7/8 = 0.88), while pass^4 requires all 4 trials of a task to pass, so Task A's single failed trial drops the score to 1/2 = 0.50.
Step 1: install the successor
# The successor line. tau-bench itself is frozen; this is where the maintained tasks live.
git clone https://github.com/sierra-research/tau2-bench.git
cd tau2-bench
git checkout v1.0.1 # pin the tag and record it in your run metadata
uv sync # the documented install path; needs Python 3.12 or 3.13
export OPENAI_API_KEY=...
# or: export ANTHROPIC_API_KEY=...
The package installs as tau2 with a tau2 CLI. Domains today are mock, airline, retail, telecom, and banking_knowledge; this recipe uses retail. The original two domains were retail and airline. Each ships a tool spec, a user simulator, and a set of tasks with deterministic state-check functions 6.
Step 2: pick a subset
The full retail set is 114 tasks at v1.0.1. At pass^4 you run each task four times, so the full set is 456 trajectories. That is too many for a smoke test. Start with the first 20 tasks (80 trajectories) for a couple of dollars of API budget.
# 20 retail tasks, 4 trials each, seeded. The CLI computes pass^k for you.
tau2 run --domain retail \
--agent-llm gpt-4o-2024-08-06 --user-llm gpt-4o-2024-08-06 \
--num-tasks 20 --num-trials 4 --seed 42
Pass --user-llm explicitly: τ-bench conversations are agent against user simulator, the simulator model is part of your experimental setup, and the CLI otherwise falls back to gpt-4.1-2025-04-14 for both sides 6. Pin it and record it with the tag; changing the user simulator changes the numbers as surely as changing the agent. The run writes per-trial result records to data/simulations/<run_name>/results.json plus a metrics summary that includes pass_hat_ks, the pass^k estimates, so on the successor the metric arrives computed. Step 3 exists anyway, because you should be able to defend the definition when someone asks what the number means, and because the same aggregation applies to any per-trial records your own harness produces.
Step 3: compute pass@1 and pass^k
The original harness wrote one flat record per (task, trial), and the script below assumes that shape. The successor's results.json instead holds a simulations list whose entries carry task_id, trial, reward_info.reward, and agent_cost, so adapt the loading loop to those fields, or read the pass_hat_ks the CLI already prints 6. We aggregate:
"""Compute pass@1 and pass^k from a τ-bench results file."""
import json
from collections import defaultdict
from pathlib import Path
import sys
import pandas as pd
results_path = Path(sys.argv[1])
records = json.loads(results_path.read_text())
# Group by task_id; each task should have k trials.
by_task: dict[int, list[bool]] = defaultdict(list)
for r in records:
by_task[r["task_id"]].append(bool(r["reward"]))
k = max(len(v) for v in by_task.values())
n_tasks = len(by_task)
# pass@1: average over all trials of all tasks.
pass_at_1 = sum(any(v[:1]) for v in by_task.values()) / n_tasks
# Actually pass@1 typically averages the per-trial success rate.
trial_successes = [t for v in by_task.values() for t in v]
pass_at_1 = sum(trial_successes) / len(trial_successes)
# pass^k: fraction of tasks where ALL trials succeeded.
pass_k = sum(all(v) for v in by_task.values() if len(v) == k) / n_tasks
print(f"Tasks: {n_tasks}, trials per task: {k}")
print(f"pass@1 = {pass_at_1:.3f}")
print(f"pass^{k} = {pass_k:.3f}")
print(f"consistency gap = {pass_at_1 - pass_k:.3f}")
# Cost summary if logged.
df = pd.DataFrame(records)
if "total_cost" in df.columns:
print(f"total cost USD = {df['total_cost'].sum():.2f}")
print(f"cost/trial USD = {df['total_cost'].mean():.3f}")
if "num_steps" in df.columns:
print(f"mean steps = {df['num_steps'].mean():.1f}")
Run it:
python compute_metrics.py tau_runs/retail/gpt-4o-2024-08-06/raw_results.json
Output:
Tasks: 20, trials per task: 4
pass@1 = 0.488
pass^4 = 0.150
consistency gap = 0.338
total cost USD = 12.40
cost/trial USD = 0.155
mean steps = 11.3
The consistency gap is the headline. A 34-point drop from pass@1 to pass^4 says the agent solves the task slightly better than half the time on any single attempt but reliably solves it across four attempts only 15% of the time. That gap is the cost of inconsistency the τ-bench paper warns about 3.
Step 4: read the failure modes
The per-task trial logs are in tau_runs/retail/<model>/. Open the ones where pass@1 was high but pass^k was zero. These are the tasks the agent could solve sometimes but not reliably. The paper's failure analysis of 115 GPT-4o function-calling retail trajectories names three patterns that recur 3:
- Wrong argument or information. The agent calls the right tool with slightly wrong arguments (truncated order id, wrong shipping address format). Symptom: tool calls error or return mismatched results.
- Incorrect decision-making and rule following. The agent ignores a domain rule (return windows, refund limits) on some trials but not others. Symptom: tool calls succeed but the final state violates a rule.
- Partial resolution of compound requests. The user asks for more than one thing, the agent finishes one and says the task is complete. Symptom: trajectory ends short, state check fails.
Each pattern has a different fix and you only see the pattern if you read the trajectories. The trace viewer for τ-bench output is roughly the structure described in the data viewer chapter.
Step 5: compare two agent scaffolds
The τ-bench paper's method comparison makes the point that "the agent" is the model plus the scaffolding (system prompt, tool spec, retry logic): the same models score differently under native function calling, ReAct, and Act-only, with function calling ahead of the text-formatted methods 3. Use the harness to compare your own variants:
# Baseline: bare tool-calling agent.
python run_subset.py --model gpt-4o-2024-08-06 --trials 4
# Variant: same model, with an extra "rule-check" planning step.
# Implement in tau_bench/agents/your_agent.py and register.
python run_subset.py --model gpt-4o-2024-08-06 --trials 4 \
--agent-strategy planning-rule-check
Read the metrics for both, and judge the variant by whether the consistency gap shrinks, not by pass@1 alone. Do not assume an extra reasoning step helps: the paper found that adding a "think" function to function-calling agents did not boost performance, although reasoning traces did help text-formatted agents 3. Anthropic's "Building Effective Agents" makes the tooling-side point: keep the design simple until added complexity demonstrably improves outcomes, and put as much effort into the tool interface as into the prompt 7. On the successor, register your agent class and select it with --agent <name> instead of using the script and path shown above; the Agent Developer Guide in the repository covers registration 6.
Cost control
A 20-task pass^4 on a strong model runs roughly 15. A 114-task pass^4 runs 100. Two ways to keep cost bounded.
- Pin a "smoke 10" subset for every PR. Run the full 114 nightly.
- Cap steps per trial, and know what a step is: the original harness stopped a task at 30 agent actions, while the successor's
--max-steps defaults to 200 on the v1.0.1 CLI 6. Read a few successful trajectories to calibrate a sane cap for your model before trusting any stuck-run heuristic.
If you are cost-bound, run k=2 in CI and k=4 nightly. Pass^2 is a weaker but cheaper consistency signal.
What this skips
This recipe runs the public benchmark. It does not build a domain-specific harness for your own tools, which is the bigger payoff and which is covered in the long-horizon planning task type. It does not compute trajectory-level rubrics (does the plan justify the calls?), which is covered in trajectory-vs-outcome. For tool-call argument correctness specifically, BFCL is a better choice than τ-bench because it scores AST equivalence directly 8. The Moshkovich observability paper is the right reading for putting trajectory-level evals into production 9.
TIP
Once you have a domain-shaped tool spec, write 20 of your most-common multi-step user flows as τ-bench-style tasks. Encode each completion check as a Python function. That set is the highest-impact golden set you can own.
What to do next
The pass^k chapter covers the math. The trajectory-vs-outcome chapter covers why outcome-only scoring misses scaffolding bugs. The SWE-bench Verified walkthrough is the autonomous-coding analog of this recipe.