CAUTION
Run the successor, not this repo. sierra-research/tau-bench is frozen: its last functional commit was August 2025, and in March 2026 its maintainers added a README warning that its airline and retail tasks are outdated, pointing users to the successor line at sierra-research/tau2-bench 1. That repository now ships τ³-bench (release v1.0.0, March 2026; current v1.0.1, July 2026), which adds a banking_knowledge domain, full-duplex voice evaluation, and 75-plus task-quality fixes to airline and retail 2. There is no tau3-bench repository; the URL is unchanged. The method below is unchanged and the metric is the same, but point your clone at the successor, pin a tag, and record it: the maintainers state that banking_knowledge results before and after v1.0.1 are not comparable.
τ-bench is the cleanest available benchmark for "does the agent actually get the job done across multiple trials" because it ships two production-shaped domains (retail and airline), a deterministic state checker, and a user-simulator strategy that makes the conversation reproducible 3. The reason this recipe exists is the consistency story: the paper shows that agents that pass once at high rates collapse when you ask them to do the same task four times in a row. That is pass^k, and pass^k is the headline metric for any product where consistency matters more than peak capability.
This recipe runs the retail subset against your agent (any agent that conforms to the τ-bench interface). It reports pass@1, pass^4, average steps, and the cost per trajectory. The goal is to make the consistency drop visible: if your agent posts 70% pass@1 but 20% pass^4, you have a flaky agent, not a low-skill agent.
Background: what pass^k catches
Pass@1 is the fraction of times the agent solves the task in one trial. Pass^k is the fraction of times the agent solves the task in all k trials.
If the agent is a fair coin per attempt, pass^k decays geometrically as p^k. A 0.70 pass@1 collapses to 0.24 at pass^4. The τ-bench paper documents this for the strongest agents tested: pass^4 on retail at roughly 20% even when pass@1 is 50% or higher 3. The implication is operational. for any product that runs the same agent more than once per user, the metric that predicts user experience is pass^k, not pass@1.
Figure: pass@1 vs pass^k on the same 8 trials: the agent runs each of two tasks 4 times, pass@1 averages per-trial success (7/8 = 0.88), while pass^4 requires all 4 trials of a task to pass, so Task A's single failed trial drops the score to 1/2 = 0.50.
Step 1: install the successor
# The successor line. tau-bench itself is frozen; this is where the maintained tasks live.
git clone https://github.com/sierra-research/tau2-bench.git
cd tau2-bench
git checkout v1.0.1 # pin the tag and record it in your run metadata
uv sync # the documented install path; needs Python 3.12 or 3.13
export OPENAI_API_KEY=...
# or: export ANTHROPIC_API_KEY=...
The package installs as tau2 with a tau2 CLI. Domains today are mock, airline, retail, telecom, and banking_knowledge; this recipe uses retail. The original two domains were retail and airline. Each ships a tool spec, a user simulator, and a set of tasks with deterministic state-check functions 4.
Step 2: pick a subset
The full retail set is 114 tasks at v1.0.1. At pass^4 you run each task four times, so the full set is 456 trajectories. That is too many for a smoke test. Start with the first 20 tasks (80 trajectories) for a couple of dollars of API budget.
# 20 retail tasks, 4 trials each, seeded. The CLI computes pass^k for you.
tau2 run --domain retail \
--agent-llm gpt-4o-2024-08-06 --user-llm gpt-4o-2024-08-06 \
--num-tasks 20 --num-trials 4 --seed 42
The --user-llm flag is not optional: τ-bench conversations are agent against user simulator, and the simulator model is part of your experimental setup. Pin it and record it with the tag; changing the user simulator changes the numbers as surely as changing the agent. The run writes per-trial result records plus a metrics summary that includes pass_hat_ks, the pass^k estimates, so on the successor the metric arrives computed. Step 3 exists anyway, because you should be able to defend the definition when someone asks what the number means, and because the same aggregation applies to any per-trial records your own harness produces.
Step 3: compute pass@1 and pass^k
The repo's results file lists one record per (task, trial). We aggregate:
"""Compute pass@1 and pass^k from a τ-bench results file."""
import json
from collections import defaultdict
from pathlib import Path
import sys
import pandas as pd
results_path = Path(sys.argv[1])
records = json.loads(results_path.read_text())
# Group by task_id; each task should have k trials.
by_task: dict[int, list[bool]] = defaultdict(list)
for r in records:
by_task[r["task_id"]].append(bool(r["reward"]))
k = max(len(v) for v in by_task.values())
n_tasks = len(by_task)
# pass@1: average over all trials of all tasks.
pass_at_1 = sum(any(v[:1]) for v in by_task.values()) / n_tasks
# Actually pass@1 typically averages the per-trial success rate.
trial_successes = [t for v in by_task.values() for t in v]
pass_at_1 = sum(trial_successes) / len(trial_successes)
# pass^k: fraction of tasks where ALL trials succeeded.
pass_k = sum(all(v) for v in by_task.values() if len(v) == k) / n_tasks
print(f"Tasks: {n_tasks}, trials per task: {k}")
print(f"pass@1 = {pass_at_1:.3f}")
print(f"pass^{k} = {pass_k:.3f}")
print(f"consistency gap = {pass_at_1 - pass_k:.3f}")
# Cost summary if logged.
df = pd.DataFrame(records)
if "total_cost" in df.columns:
print(f"total cost USD = {df['total_cost'].sum():.2f}")
print(f"cost/trial USD = {df['total_cost'].mean():.3f}")
if "num_steps" in df.columns:
print(f"mean steps = {df['num_steps'].mean():.1f}")
Run it:
python compute_metrics.py tau_runs/retail/gpt-4o-2024-08-06/raw_results.json
Output:
Tasks: 20, trials per task: 4
pass@1 = 0.488
pass^4 = 0.150
consistency gap = 0.338
total cost USD = 12.40
cost/trial USD = 0.155
mean steps = 11.3
The consistency gap is the headline. A 34-point drop from pass@1 to pass^4 says the agent solves the task slightly better than half the time on any single attempt but reliably solves it across four attempts only 15% of the time. That gap is the cost of inconsistency the τ-bench paper warns about 3.
Step 4: read the failure modes
The per-task trial logs are in tau_runs/retail/<model>/. Open the ones where pass@1 was high but pass^k was zero. these are the tasks the agent could solve sometimes but not reliably. The τ-bench paper identifies three patterns that recur:
- Rule misapplication. The agent ignores a domain rule (return windows, refund limits) on some trials but not others. Symptom: tool calls succeed but the final state violates a rule.
- Tool-call drift. The agent calls the right tool with slightly wrong arguments (truncated order id, wrong shipping address format). Symptom: tool calls error or return mismatched results.
- Premature termination. The agent says "the task is complete" before the state check passes. Symptom: trajectory ends short, state check fails.
Each pattern has a different fix and you only see the pattern if you read the trajectories. The trace viewer for τ-bench output is roughly the structure described in the data viewer chapter.
Step 5: compare two agent scaffolds
The τ-bench paper's strongest lesson is that "the agent" is the model plus the scaffolding (system prompt, tool spec, retry logic). The same model with two different scaffolds posts wildly different pass^k. Use the harness to compare:
# Baseline: bare tool-calling agent.
python run_subset.py --model gpt-4o-2024-08-06 --trials 4
# Variant: same model, with an extra "rule-check" planning step.
# Implement in tau_bench/agents/your_agent.py and register.
python run_subset.py --model gpt-4o-2024-08-06 --trials 4 \
--agent-strategy planning-rule-check
Read the metrics for both. The τ-bench paper's finding is that adding a planning step typically raises pass^k more than it raises pass@1, which is the right direction; you want the consistency gap to shrink 3. The Anthropic "Building Effective Agents" writeup makes the same point: scaffolding matters more than model swaps for agentic workflows past a certain capability threshold 5.
Cost control
A 20-task pass^4 on a strong model runs roughly 15. A 114-task pass^4 runs 100. Two ways to keep cost bounded.
- Pin a "smoke 10" subset for every PR. Run the full 114 nightly.
- Cap
max_steps per trial, and know what a step is: the successor counts each message as a step and defaults far higher than the original harness did (100 at the run layer in v1.0.1). Read a few successful trajectories to calibrate a sane cap for your model before trusting any stuck-run heuristic.
If you are cost-bound, run k=2 in CI and k=4 nightly. Pass^2 is a weaker but cheaper consistency signal.
What this skips
This recipe runs the public benchmark. It does not build a domain-specific harness for your own tools, which is the bigger payoff and which is covered in the long-horizon planning task type. It does not compute trajectory-level rubrics (does the plan justify the calls?), which is covered in trajectory-vs-outcome. For tool-call argument correctness specifically, BFCL is a better choice than τ-bench because it scores AST equivalence directly 6. The Moshkovich observability paper is the right reading for putting trajectory-level evals into production 7.
TIP
Once you have a domain-shaped tool spec, write 20 of your most-common multi-step user flows as τ-bench-style tasks. Encode each completion check as a Python function. That set is the highest-impact golden set you can own.
What to do next
The pass^k chapter covers the math. The trajectory-vs-outcome chapter covers why outcome-only scoring misses scaffolding bugs. The SWE-bench Verified walkthrough is the autonomous-coding analog of this recipe.