Pass@1 measures the probability that a single attempt succeeds. Pass^k measures the probability that all k attempts on the same task succeed. They sound similar and they are emphatically not. The first is the headline number on every agent leaderboard; the second is the number that predicts whether you can ship the agent to a real user. The gap between them, for the same model on the same task, can be enormous.
τ-bench introduced pass^k as a first-class metric for exactly this reason 1. A customer-support agent at 90% pass@1 on a retail-task benchmark drops to roughly 43% at pass^8 and around 19% at pass^16 if the failures are independent. The arithmetic is unforgiving: pass^k = p^k under independence, and even with positive correlation between attempts the decay is steep enough to matter operationally. A 1-in-10 chance of going off-policy per multi-turn conversation is the difference between an internal tool and a production launch. The paper's own numbers show the same shape: GPT-4o function calling solved fewer than half of the tasks, and its pass^8 in the retail domain was under 25% 1.
Check the benchmark version before you reuse any τ-bench number. The original repo now warns that its tasks are not updated and points to the τ³-bench line 2. That line's v1.0.0 (March 2026) made 75+ task-quality fixes in the airline and retail domains, and v1.0.1 (July 2026) changed grading in the banking_knowledge domain, with the maintainers saying earlier banking_knowledge scores are not comparable 2. The public leaderboard at taubench.com shows pass^1 only (fetched as of 2026-10-01), so pass^k has to come from your own multi-trial runs; the CLI takes a --num-trials flag 3 2. Record the benchmark version next to every pass^k you publish.
The math, in two lines
For a single task with per-attempt success probability p:
- pass@1 = p
- pass^k (independent attempts) = p^k
When attempts are correlated, the decay is slower but still strictly worse than pass@1. The τ-bench definition is the strict one: pass^k is the fraction of tasks for which all k independent attempts succeed 1. You can compute it from raw per-attempt results without any modeling assumption.
For a benchmark with n tasks and k attempts each, the unbiased per-task estimator is:
- task_i_pass^k = 1 if all k attempts succeed, else 0
- benchmark_pass^k = mean over tasks
If you ran more than k attempts per task and want a stable estimate of pass^k from m total attempts (m >= k), do not just take the first k. Average C(c, k) / C(m, k) over tasks, where c is the number of successes on the task. That is the combinatorial form the τ-bench paper uses for pass^k 1; it is the all-must-succeed counterpart of the HumanEval pass@k estimator, and at k=1 it reduces to pass@1.
One trap from coding benchmarks: m and c count independent rollouts of the same task, not unit tests inside one submission. An August 2026 preprint found that using the number of unit tests as the attempt count reported 0.96 to 0.98 where the rollout-based count gave 0.00 to 0.12 4.
Figure: Pass^k decay curve for independent attempts, where pass^k = p^k. An agent with 90% pass@1 drops to 66% at pass^4, 43% at pass^8, and 19% at pass^16, while a 99% pass@1 agent still holds 85% at pass^16. Per-attempt consistency, not the pass@1 headline, determines repeated-run reliability.
Why pass@1 hides the inconsistency you ship with
Three failure modes that pass@1 cannot see and pass^k can.
First, near-misses that become real misses. An agent that succeeds 9 times out of 10 may be failing on a near-deterministic class: the same kind of input fails the same way. Pass@1 averages over the class; pass^k surfaces it because k independent attempts have k chances to hit the failure class. If the class is 5% of the input space and it is hit on every attempt, pass^k decays much faster than a 95% pass@1 model with random failures would suggest.
Second, judge gaming. An agent that wins pass@1 by gaming the judge prompt (more later, in reward hacking) often wins by a narrow margin that is sensitive to small perturbations. Pass^k repeated under temperature greater than zero will catch the agent that wins one attempt and fails the next because the gaming is fragile.
Third, environmental variance. Production tools fail intermittently. An agent that is brittle to a single 429 response will pass at k=1 because most attempts do not see the 429, and fail at k=8 because at least one of eight does. A happy-path pass^k will not show this; run the environmental drift perturbations at k>=4 and compare.
TIP
A model that gains pass@1 and loses pass^4 is a regression for production even if the leaderboard moves up. Pass@1 measures peak capability; pass^k measures reliability. Reliability is what ships.
Pareto curves, not single numbers
Once you accept that pass^k matters, you stop reporting a single benchmark number and start reporting a curve. The shape of the curve tells you what kind of agent you have.
- Steep decay (pass@1 high, pass^4 collapses): a strong but brittle model. Capability is real; consistency is not. Useful for capability ceilings; bad for production gates.
- Shallow decay: a model whose failures are concentrated on a small set of tasks. The capability ceiling is lower but the operating ceiling is higher.
- Plateau then drop: a model that handles common cases reliably and fails identically on a hard subset. The hard subset is where to invest annotation effort.
xychart-beta
title "pass@1 vs pass^4 vs pass^8 for three agents"
x-axis ["Agent A", "Agent B", "Agent C"]
y-axis "Success rate (%)" 0 --> 100
bar [90, 65, 75]
bar [66, 45, 65]
bar [43, 30, 56]
Agent A has the highest pass@1 (90%) but its independence-implied pass^8 (about 43%) means about 57% of tasks hit at least one failure across eight attempts. Agent B is weaker everywhere. Agent C trails A on pass@1 (75%) but holds 56% at pass^8 because its failures are concentrated and not lottery-style; in a production setting that needs consistent multi-step behaviour, Agent C is the better deploy.
The point of the chart is not that lower decay is always better; it is that you cannot tell A from C from pass@1 alone, and the choice between them depends on whether your application is one-shot or repeated.
A runnable pass^k computation
The naive version of pass^k under k=1 attempts gives you nothing; you need k>=4 trials at temperature greater than zero per task. Then compute the per-task fraction and the benchmark-level pass^k. Here is a runnable harness skeleton you can drop into a notebook.
"""
Compute pass@1, pass^k, and the HumanEval-style unbiased pass@k from
raw per-attempt results. No external dependencies.
"""
from math import comb
from statistics import mean
from typing import Iterable
Attempt = bool # True if the attempt succeeded
Task = list[Attempt] # k attempts on one task
Benchmark = list[Task] # n tasks, each with k attempts
def pass_at_1(benchmark: Benchmark) -> float:
"""Average per-attempt success across all (task, attempt) pairs."""
flat = [a for task in benchmark for a in task]
return mean(flat)
def strict_pass_pow_k(benchmark: Benchmark, k: int) -> float:
"""
τ-bench-style pass^k: fraction of tasks for which the first k
attempts ALL succeed. Requires at least k attempts per task.
"""
bad = [t for t in benchmark if len(t) < k]
if bad:
raise ValueError(f"{len(bad)} task(s) have fewer than {k} attempts")
return mean(1 if all(t[:k]) else 0 for t in benchmark)
def unbiased_pass_at_k(benchmark: Benchmark, k: int) -> float:
"""
HumanEval-style estimator for pass@k from m>=k attempts per task.
pass@k_hat = E_task [1 - C(m-c, k) / C(m, k)] where c is the count
of successful attempts on the task and m is the number of attempts.
This estimates "probability that at least one of k samples is correct"
using all m samples without bias.
"""
estimates = []
for task in benchmark:
m = len(task)
c = sum(1 for a in task if a)
if m - c < k:
estimates.append(1.0)
else:
estimates.append(1.0 - comb(m - c, k) / comb(m, k))
return mean(estimates)
def report(name: str, benchmark: Benchmark, ks: Iterable[int] = (1, 4, 8)):
print(f"=== {name} ===")
print(f" pass@1 = {pass_at_1(benchmark):.3f}")
for k in ks:
try:
print(f" pass^{k} = {strict_pass_pow_k(benchmark, k):.3f}")
except ValueError as e:
print(f" pass^{k} = skipped ({e})")
# Toy example: 5 tasks, k=8 attempts each.
# Agent A: 90% pass@1 but failures are independent (Bernoulli with p=0.9).
# Agent C: 60% pass@1 (two of five tasks fail every attempt; the other
# three always succeed).
agent_a = [
[True, True, True, True, True, True, True, False],
[True, True, False, True, True, True, True, True],
[True, True, True, True, True, True, False, True],
[True, True, True, True, False, True, True, True],
[True, True, True, True, True, True, True, True],
]
agent_c = [
[True] * 8,
[True] * 8,
[True] * 8,
[False] * 8,
[False] * 8,
]
report("Agent A (brittle)", agent_a)
report("Agent C (concentrated)", agent_c)
Running this on the toy data: Agent A reports pass@1 = 0.90 and pass^8 around 0.20; Agent C reports pass@1 = 0.60 and pass^8 = 0.60. The numbers are toy, but the asymmetry is real: outcome on a single attempt does not constrain joint success across attempts, and a leaderboard that reports only the former is hiding the latter.
The same run prints pass^1 = 1.000 for Agent A against pass@1 = 0.900, because strict_pass_pow_k only looks at the first k attempts and every task's first attempt succeeded. That is the order-dependence of the naive estimator. With m > k attempts per task, use the C(c, k) / C(m, k) form above, which gives pass^1 = pass@1 by construction.
How to report pass^k in a model card
Three rules, lifted from how the τ-bench paper presents results 1:
- Always pair pass^k with pass@1. The delta is the information.
- Pick k from the deployment, not the benchmark. If your product is a 6-turn customer support flow, pass^6 is the meaningful one. If your product is a one-shot search summarizer, pass^1 is the meaningful one. Pick the direction too: Anthropic's agent-eval guidance reaches for pass@k where one success among k attempts is enough, and pass^k where consistency is essential, as in customer-facing agents. In its illustration, at k=10 the two tell opposite stories, with pass@k near 100% and pass^k near 0% 5.
- Use temperature greater than zero for at least k attempts per task, with the random seed varied per attempt. Temperature zero with the same seed makes the metric meaningless.
The agent-eval surveys both cover repeated-trial consistency under reliability, through pass@k variants and τ-bench, and note that single-run scores mislead when agent behaviour is non-deterministic 6 7. Adopting it is cheap: it needs only repeated runs of the harness you already have.
Pass^k as defined above resamples the same task; it never changes the input. A May 2026 preprint argues consistency should also be tested under semantic-preserving rewrites of the task, and reports on three agentic benchmarks that trajectory-level consistency metrics were far more diagnostically sensitive than pass@1, with agents that had the needed knowledge still breaking down on minor task variations 8. If your tasks have natural paraphrases, vary the wording across the k attempts and report the seed-only and wording gaps separately.
Operational implications
Three implications for how you build agent products.
First, the workflow-vs-agent decision sharpens. Anthropic's December 2024 guide to building effective agents argues for the simplest architecture that solves the problem, with workflows preferred over agents wherever they suffice 9. Pass^k is the metric that surfaces when an agentic loop is buying you peak capability at the cost of reliability. If a workflow has pass^k = 0.95 and your agent has pass@1 = 0.97 with pass^k = 0.50, the workflow wins for a production gate.
Second, replanning behaviour shows up in pass^k. An agent that detects its own failures and retries with a corrected plan has correlated successes across attempts and a flatter pass^k curve. An agent that hits the right answer on lucky tries has uncorrelated successes and a steeper curve. The shape of the curve is a diagnostic for replanning quality.
Third, contamination can show up too. An agent that has memorized answers to a public benchmark, GAIA's validation answers being public 10, can post a high pass@1 and a much lower pass^k, because memorized answers tend to be brittle to seed perturbation. A widening pass@1 to pass^k gap on a public benchmark, where it does not appear on a private regression set, is a contamination signal.
Budget is a hidden variable in every pass^k number
Everything above assumes each attempt runs to completion. In practice each attempt runs to a budget: a step cap, a token cap, a wall-clock timeout. That budget is not a neutral implementation detail, it is a variable that changes the number, and a pass^k reported without it is underspecified.
The UK AI Security Institute measured this directly in a July 2026 post. Capability is a curve over test-time compute, not a point: raising the per-task budget from one million to ten million tokens lifted scores on TerminalBench 2.0 and SWE-Bench Pro by roughly 25%, and about 8% of their cyber tasks were solved only once budgets reached ten million tokens or more, some needing fifty million 11. A fixed-budget eval therefore does two things you did not intend. It understates a frontier model that would have succeeded with more room, and it distorts cross-model comparison, because a token-efficient model and a token-hungry one meet at different points on their curves. The same post found the cyber time-horizon trend over the past year is about 60% steeper measured at fifty-million-token budgets than at 2.5-million, and put the current frontier's horizon on its cyber CTF suite at roughly 2 hours at 2.5 million tokens against about 14 hours at 50 million. The budget you pick changes both the level and the slope of the trend you report.
This compounds the reproducibility problem the statistics chapters warn about. METR's time-horizon work makes the point from the other side: when the Time Horizon 1.1 update grew the task suite from 170 to 228 tasks and moved the harness from Vivaria to Inspect, re-estimates for existing models shifted by between -57% and +55%, and the estimated capability-doubling time since 2023 moved from 165 days to 131, with no change in the models themselves 12. The methodology itself is the long-tasks paper 13. METR's live page still lists Time Horizon 1.1 as current (last updated May 8, 2026, fetched as of 2026-10-01) and warns that measurements above 16 hours are unreliable with the current task suite 14. Point estimates drift when the measurement moves, and budget is one of the largest things that moves.
Three practical rules for reporting pass^k, extending the three above:
- State the budget with the number. "Pass^4 at a 50-step, 200k-token cap" is a claim; "pass^4" alone is not reproducible.
- Prefer a budget curve to a point. Report pass^k at two or three budgets so a reader can see whether the agent is saturated or still climbing.
- Establish a minimum informative budget before you compare models: raise the budget until pass^k stops improving, and compare there. AISI recommends the same, using that term 11. Comparing two agents at a budget where one is still climbing measures the budget, not the agents.
The deeper framing is that accuracy without cost is an incomplete metric, the argument of AI Agents That Matter: benchmarks that optimize accuracy alone reward needlessly expensive agents and produce mistaken conclusions about where the gains came from 15. Pass^k is a reliability metric, and reliability at unbounded cost is not the thing your product ships. Report the consistency and the budget it was bought at, together, the same way the cost and latency chapter reports quality against its price.
What to do this week
- For one workflow you already evaluate, run k=4 attempts per task at temperature 0.7. Compute pass@1 and pass^4. Compare to your current single-run number.
- Pick k by deployment. Six-turn flows: k=6. One-shot tasks: k=1. Long-horizon research agents: k=8 or 16.
- Add pass^k to your model card or release note alongside pass@1. Track it across releases. A pass^k drop is a release gate, not a footnote.