Asha v3.2 shipped after one green run of 60 scenarios, and the dispute segment in that run held 4 scenarios. A month later the dispute flow failed 1 call in 5. Nothing in the release run was wrong. It was one trial per scenario, four scenarios where it mattered most, and a gate that read "all passed" as "safe". This chapter covers the statistics that would have stopped it.
What happened at release?
The release gate ran each of 60 scenarios once, saw 60 passes and shipped. The dispute segment contributed 4 of those passes. A single all-green run on 4 scenarios is weak evidence: an agent that fails 1 dispute call in 5 still passes all 4 about 41% of the time.
release gate run Asha v3.2 candidate
scenarios 60
trials per scenario 1
result 60 / 60 pass -> shipped
dispute segment 4 scenarios, 4 / 4 pass
one month later dispute flow fails 1 call in 5 in production
Diagnosis: the gate measured whether one draw succeeded, on a sample too small to bound the rate, and never asked how often every call succeeds.
The arithmetic behind "41%" is 0.8^4 = 0.41. The release run's own data allowed for this: the Wilson 95% interval on 4 of 4 is 0.51 to 1.00, so the run could not rule out a dispute success rate of 51%. Clearing a 90% bar on the Wilson lower bound with zero failures takes 35 of 35. A gate that ignores its own uncertainty does not have a bar at all, only a ritual.
Why does every-trial success fit collections?
pass^k is the probability that all k independent trials of the same task succeed, averaged over tasks. Collections needs it because the same flow runs thousands of times a day and one borrower can hear several calls. At 80% per call, 8 in a row all succeed 0.8^8 = 16.8% of the time.
tau-bench (Yao et al., Sierra, 2024) defines pass^k this way and reports what it does to a strong agent. In its main results the best agent, gpt-4o with function calling, scored pass^1 61.2% on retail and 35.2% on airline, and its retail pass^8 fell below 25% 1. The headline pass@1 and the every-time rate are different numbers about different promises.
Anthropic's engineering guidance makes the same point with a small example: an agent at 75% per trial passes all 3 trials about 42% of the time, and the post recommends pass^k for agents where consistency is essential, especially customer-facing ones 2. That arithmetic holds for one task at a fixed rate. Across a suite of mixed tasks, pass^k is the average of each task's p^k, which is at least (mean p)^k. Two tasks at p = 1 and p = 0.5 average 0.75 per trial, yet their pass^3 is 0.5625, not 0.42. Heterogeneity matters in both directions, which is why the estimator below works per task.
Figure: pass^k = p^k for one task with independent trials. A 5-point gap in per-trial success (95% against 90%) becomes a 23-point gap at k = 8; at 80% per trial, 8 in a row succeed 16.8% of the time. Computed curves, not measurements.
Choose k from the deployment, not from habit. Ask how many times one borrower meets the same flow before the outcome is settled. Tarazu allows at most 2 connected recovery calls per account a day and 6 a week (C11), and a disputing borrower can hear Asha on several of them; one bad call on that path is a complaint. So k should cover a realistic run of exposures, and k = 4 is the nightly choice at Tarazu. Two limits bound k from above. The estimator needs at least k trials per scenario, and each audio trial costs a simulated call, which chapter 14 prices. Report pass^1 next to pass^k always: pass^1 tells you about the average call, pass^k about the borrower who gets four of them. The general treatment, including why pass@k is the wrong direction for this question, is in pass^k and consistency.
How do you estimate every-trial success from repeated runs?
Run each task n >= k times, count successes c, compute C(c,k)/C(n,k) per task and average across tasks. That is the tau-bench estimator, unbiased for each task's p^k. Raising an observed pass rate to the power k is biased: upward per task, downward when pooled across mixed tasks.
The estimator counts the fraction of size-k subsets of the n trials in which every trial passed. For c = 6 successes in n = 8 trials at k = 4, it gives C(6,4)/C(8,4) = 15/70 = 0.214. The shortcut (6/8)^4 gives 0.316, half as large again.
The direction of that error is predictable. The Codex paper (Chen et al., OpenAI, 2021) generated n = 200 samples per task and showed that the plug-in shortcut 1 - (1 - p_hat)^k consistently underestimates pass@k, with a gap that does not fully close even when n > 5k; it uses an unbiased per-task estimator instead 3. Our reasoning, not that paper's claim: for pass^k the plug-in p_hat^k is biased upward per task, the mirror of the pass@k underestimate, because the average of p_hat^k is at least p^k. For a scenario at a true 80% per trial, the expected plug-in pass^4 is 0.549 with n = 4 trials and 0.483 with n = 8, against a true 0.410. The unbiased estimator averages 0.410 at every n.
Unbiased is not the same as precise. With n = k = 4, C(c,4)/C(4,4) is 1 if all four passed and 0 otherwise, so one scenario's estimate is a coin flip with a standard deviation of 0.49 at p = 0.8. At n = 8 that falls to 0.31, at 12 to 0.25 and at 16 to 0.21. Averaging over many scenarios fixes most of it, which is why Tarazu runs k = 4 on 240 scenarios rather than k = 16 on 60.
Worked example, from the seeded synthetic suite in the pass^k planner recipe: 60 scenarios with a hard tail, 8 trials each. Synthetic Tarazu data.
| k | Unbiased pass^k | Per-task (c/n)^k | Pooled (mean p)^k | pass@k |
|---|
| 1 | 0.969 | 0.969 | 0.969 | 0.969 |
| 4 | 0.886 | 0.904 | 0.881 | 1.000 |
| 8 | 0.800 | 0.857 | 0.776 | 1.000 |
At k = 1 all three agree, since there is nothing to be biased about. At k = 8 the per-task shortcut overstates by 5.7 points and the pooled shortcut understates by 2.4. The true suite pass^8 from the generating rates is 0.751, so even the unbiased number is 4.9 points high on this one draw: unbiased means right on average, and a single suite of 60 scenarios still carries sampling error. pass@k sits at 1.000 from k = 4, which is why it never belongs on a collections release report.
Two rules follow. Store per-scenario counts (c, n), never only the suite's pass rate, because pass^k cannot be recovered from the mean. And never compute pass^k from fewer than k trials per scenario; the estimator is undefined there, and any number you produce is a guess dressed as a metric.
Why do runs vary at temperature 0?
Even at temperature 0, LLM outputs and benchmark scores differ between identical runs; Thinking Machines Lab traces most of it to kernels that are not batch-invariant, so output depends on server load. A voice stack adds streaming STT, VAD and turn detection, TTS and the caller simulator.
The LLM part is measured. Atil et al. found min-to-max accuracy gaps of up to 15% across 10 runs of five models at temperature 0, top-p 1 and a fixed seed 4. Thinking Machines Lab got 80 distinct completions from 1,000 temperature-0 runs of one prompt on Qwen3-235B in vLLM and made all 1,000 completions identical with batch-invariant kernels 5. Load-dependent output matters more for a voice agent than for a batch eval, since the nightly run and the 17:00 peak load do not share a batch profile. The non-determinism chapter covers the general case.
The voice stack contributes four more sources, each with a mechanism you can test:
- STT. Streaming recognition finalizes on chunk boundaries. A different alignment of 20 ms frames can turn "dhai" into "do" on one run and not the next, which is T01's failure mode as a probability, not a one-off.
- VAD and the turn detector. End-of-turn decisions compare a measured silence with a budget. A hesitation near the silence budget commits on one run and holds on another; T05's false cutoff is the case where the commit landed inside a pause.
- TTS. A model-based voice can render the same text with different durations, which moves where a caller's barge-in lands relative to the agent's words.
- The simulator. The caller model samples, and Tarazu's interruption scripts are content-triggered, so a barge-in fires at a different point when the agent's wording changes.
The design decision is to separate these, not to suppress them. Run a sample of scenarios two ways: frozen caller audio replayed through the agent (agent-side variance only) and live simulation (agent plus caller). If live-simulation variance dwarfs replay variance, the simulator dominates and pinning seeds in the agent buys nothing; the simulation chapter covers validating the simulator itself. Setting a seed and running once does not remove variance. It hides it inside a single draw and reports that draw as the system.
How many runs do you need?
Start from the minimum detectable effect, the smallest drop you must not ship, then solve for runs at a chosen power and alpha. Detecting 90% falling to 85% at 80% power, one-sided alpha 0.05, takes about 540 scenarios per arm unpaired, and about 220 when both versions run on the same scenarios.
The paired number depends on the discordant cells, the share of scenarios where the versions disagree. With 7% of scenarios passing on the old version and failing on the new, and 2% the other way, McNemar's test needs 221 paired scenarios 6. Concordant scenarios carry no information about the difference, so pairing spends the budget only where the versions disagree. The more correlated the two versions, the fewer discordant cells and the larger the saving. Miller's paper on error bars for evals makes the same case for paired differences when two models answer the same questions 7. For a voice agent, pairing means more than the same scenario card: the same audio conditions, the same codec chain and, for replay runs, the same caller audio.
Power is planned against the unit you resample. Trials inside a scenario are correlated, because the same hard scenario fails again and again, so 240 scenarios at k = 4 are not 960 independent trials. Plan on scenarios, and treat k as a way to reduce noise within each scenario. The site's power tables cover the unclustered case.
Planner output for three decisions, computed under stated assumptions: every scenario in a segment shares one per-trial rate, trials are independent, simulated cases use 4,000 releases at seed 20261120. The 95% old-version rate is an assumption for the arithmetic, not a Tarazu measurement.
| Decision | Design | Scenarios | Alpha | Power or hit rate |
|---|
| 1. Suite-wide drop, 90% -> 85% | Unpaired two-proportion test | 540 per arm | 0.05 | 0.80 (closed form) |
| 1. Suite-wide drop, 90% -> 85% | Paired McNemar, 7% / 2% discordant | 221 | 0.05 | 0.80 (closed form) |
| 2. One gated segment, 95% -> 80% | Paired delta on scenario means at k = 4, allowed drop 5 points | 20 | 0.0042 (0.05/12) | about 0.26 (simulated) |
| 2. One gated segment, 95% -> 80% | Same test without correction | 20 | 0.05 | about 0.63 (simulated) |
| 3. One flow group, 95% -> 80% | Paired delta pooled over 3 language segments | 60 | 0.0125 (0.05/4) | about 0.89 (simulated) |
Read row 2 slowly. Twenty scenarios per segment cannot reliably catch even a 15-point drop once the gate corrects for 12 segments. That is the price of control over false alarms, and the next sections decide how to pay it.
tau-Voice shows the scale at which benchmark authors resolve voice gaps. It tested its main gaps with paired permutation tests rather than trial-level confidence intervals: 2 independent runs per condition on the Retail domain (114 tasks, 228 simulations per condition), pairing by task ID, 100,000 two-sided permutations and Holm-Bonferroni correction within each provider. The Clean-to-Realistic drop was 13.2 points for Google (42.1% to 28.9%, adjusted p = 0.026) and for xAI (50.0% to 36.8%, p = 0.044), and 24.6 points for OpenAI (67.5% to 43.0%, p < 0.001; the paper's figure, from unrounded rates) 8. Our reading of those p-values, not the paper's claim: a 13-point drop on 114 paired tasks only just clears 0.05. Your 20-scenario segment will not resolve 5 points.
How do you put an interval on WER and entity error rate?
Resample speakers, not utterances, and recompute total errors over total reference words on each replicate. Utterances from one speaker share an accent, a phone and a room, so treating them as independent narrows the interval. For a version delta, score both versions on the same resampled speakers.
Corpus WER is a ratio metric: summed substitutions, deletions and insertions over summed reference words. It is not the average of per-utterance WERs, and its interval does not come from a binomial formula. Entity error rate has the same shape (errors over reference entities), with the call or speaker as the cluster, and its definition belongs to the entity chapter. Cameron and Miller's guide is the standard reference for why clustered data needs cluster-level resampling or a cluster-adjusted variance 9; the site's clustered standard errors page works the formula.
The script below uses the shape of Tarazu's gold-v7 STT set, 1,200 reference utterances from 180 speakers, with synthetic errors. Synthetic Tarazu data.
"""Speaker-clustered bootstrap for corpus WER and a paired WER delta.
Synthetic Tarazu-shaped data: 1,200 utterances from 180 speakers. Standard library only."""
import math
import random
rng = random.Random(20261120)
WORDS = 12 # reference words per utterance
# 120 speakers x 7 utterances + 60 x 6 = 1,200. Speaker difficulty is skewed.
rows = [] # (speaker, ref_words, errors_old, errors_new)
for spk in range(180):
rate = min(0.6, 0.08 * math.exp(rng.gauss(0, 0.7)))
for _ in range(7 if spk < 120 else 6):
old = sum(rng.random() < rate for _ in range(WORDS))
new = sum(rng.random() < rate * 0.92 for _ in range(WORDS))
rows.append((spk, WORDS, old, new))
def wer(sample, col):
return sum(r[col] for r in sample) / sum(r[1] for r in sample)
by_spk = {}
for r in rows:
by_spk.setdefault(r[0], []).append(r)
speakers = list(by_spk)
def ci(stats):
stats.sort()
return stats[int(0.025 * len(stats))], stats[int(0.975 * len(stats))]
B = 2000
naive, clustered, delta, unpaired = [], [], [], []
for _ in range(B):
utt = [rows[rng.randrange(len(rows))] for _ in rows]
naive.append(wer(utt, 2))
spk = [r for _ in speakers for r in by_spk[speakers[rng.randrange(len(speakers))]]]
clustered.append(wer(spk, 2))
delta.append(wer(spk, 3) - wer(spk, 2)) # same speakers, both versions: paired
other = [r for _ in speakers for r in by_spk[speakers[rng.randrange(len(speakers))]]]
unpaired.append(wer(other, 3) - wer(spk, 2)) # independent resamples: ignores pairing
lo_n, hi_n = ci(naive)
lo_c, hi_c = ci(clustered)
lo_d, hi_d = ci(delta)
lo_u, hi_u = ci(unpaired)
print(f"utterances {len(rows)}, speakers {len(speakers)}")
print(f"corpus WER old {wer(rows, 2):.4f}, new {wer(rows, 3):.4f}")
print(f"utterance bootstrap 95% CI [{lo_n:.4f}, {hi_n:.4f}] width {hi_n - lo_n:.4f}")
print(f"speaker bootstrap 95% CI [{lo_c:.4f}, {hi_c:.4f}] width {hi_c - lo_c:.4f}")
print(f"width ratio {(hi_c - lo_c) / (hi_n - lo_n):.2f}")
print(f"paired delta new-old {wer(rows, 3) - wer(rows, 2):+.4f}, speaker bootstrap 95% CI [{lo_d:+.4f}, {hi_d:+.4f}]")
print(f"same delta, resampled as if unpaired: [{lo_u:+.4f}, {hi_u:+.4f}]")
Running it with Python 3.11 prints:
utterances 1200, speakers 180
corpus WER old 0.0914, new 0.0856
utterance bootstrap 95% CI [0.0856, 0.0974] width 0.0118
speaker bootstrap 95% CI [0.0806, 0.1025] width 0.0219
width ratio 1.86
paired delta new-old -0.0058, speaker bootstrap 95% CI [-0.0119, +0.0006]
same delta, resampled as if unpaired: [-0.0208, +0.0105]
Three readings. The honest interval is 1.86 times wider than the utterance bootstrap, and with about 6.7 utterances per speaker that is ordinary, not extreme. The new version's 0.58-point improvement has a paired interval that still crosses zero, so it is not yet a claim. And the same delta resampled as if unpaired gives an interval 2.5 times wider than the paired one; throwing away the pairing throws away most of the evidence.
Use the right tool per metric type. Wilson is for a single binary proportion over independent units, such as the share of scenarios that pass 10. Differences of proportions on shared scenarios use McNemar or a paired bootstrap; independent samples use Newcombe's method. WER and entity error rate use the speaker- or call-clustered bootstrap, and percentiles such as P95 turn gap use a bootstrap clustered by call. A Wilson interval on WER, on clustered utterances or on a latency percentile looks rigorous and is wrong. The site's confidence intervals and bootstrap and paired tests pages carry the general methods.
How do you gate on segments without drowning in false alarms?
Gate each of the 12 segments on a paired delta between candidate and control, scenario as the unit, Holm-corrected across segments; block when the one-sided bound excludes the allowed drop. Add a pooled flow-group test so a regression spread across languages is not missed, and report the slices you do not gate.
Tarazu's 12 gated segments are 3 language groups (Hinglish, Hindi-dominant, Indian English) x 4 flow groups (promise; dispute and already paid; hardship and complaint; other parties and machines). DPD bucket, carrier, hour and code-switch density are monitored slices: reported on every release, never gated. The split is deliberate. Every gated cell needs enough scenarios to mean something, and every extra gated cell costs power through the correction.
Without correction, 12 segments each tested at 0.05 produce at least one false block on a healthy release with probability 1 - 0.95^12 = 46%, if the tests were independent. A gate that blocks every other release gets overridden, and an overridden gate is worse than none because it teaches the team to ignore it. Holm's step-down method holds the family-wise false-block rate at or below 5% under any dependence: sort the 12 p-values, compare the smallest to 0.05/12, the next to 0.05/11, and stop at the first that fails 11. It blocks everything Bonferroni would block, and sometimes more. The alternatives, including Benjamini-Hochberg when you can accept some false blocks for more power, are on the multiple comparisons page.
The gate's direction matters. Tarazu's test is H0: "the segment dropped no more than the allowed 5 points", and a block requires evidence against it. That keeps false blocks rare and puts the burden on the alarm, which is right for a nightly gate but means a segment with weak evidence passes. Row 2 of the planner shows the cost: about 26% power against a 15-point drop in one segment.
Two additions recover most of it without new scenarios:
- A flow-group family. Pool the three language segments of each flow group (60 scenarios) and run the same paired test, Holm-corrected across the 4 flow groups. Against the same 95% -> 80% drop it fires about 89% of the time in the planner. A dispute regression that hits every language shows up here even when no single segment blocks.
- pass^4 per segment, on the release report. At 95% per trial the scenario-level pass^4 is 0.81; at 80% it is 0.41. A floor at 0.60 would catch the 80% segment 93% of the time with a false alarm rate of 0.6% per segment at 95%, about 7% across 12. Floors are absolute, though: a segment that legitimately runs at 90% per trial trips that floor about 22% of nights. Set each floor from the segment's own history, or report pass^4 without gating on it.
In the recipe's seeded run, a regression planted in Hindi-dominant x dispute and already paid moved that segment from 0.912 to 0.613, a paired delta of -0.300 with a one-sided upper bound of -0.163 and a Holm-adjusted p of 0.006, and the gate blocked it. The pooled delta over all 240 scenarios was -0.022, which a suite-level gate would have read as a small dip. Pooling hides a concentrated regression; segmenting without correction invents them. The gate needs both levels.
What does a release report look like?
For every number, a release report gives the estimate, its interval and method, n, k and the versions of every model, judge and dataset behind it, then per-segment paired deltas with Holm-adjusted p-values and the decision. A number without n, k and versions cannot be reproduced or compared.
| Field | What goes in it |
|---|
| Versions | agent, prompt, STT, turn detector, LLM, TTS, policy pack, taxonomy; simulator version and replay or live mode |
| Dataset | gold set release (gold-v7), scenario count, how the nightly 240 were drawn |
| Judges | each judge id and version with its validation status per language; J1 is advisory and never gates |
| Run shape | n scenarios, k trials, temperature, kernel or serving settings that affect determinism |
| Task success | pass^1 with a scenario-clustered interval; pass^4 by the unbiased estimator |
| Segment table | per gated segment: control, candidate, paired delta, one-sided bound, raw p, Holm p, pass^4, decision |
| Flow-group table | the same for the 4 pooled flow groups |
| Monitored slices | DPD bucket, carrier, hour, code-switch density: estimates and intervals, no decision |
| Entity and STT | amount and date entity error rate with speaker-clustered intervals; WER as a diagnostic |
| Latency | P95 platform turn gap by turn type, with a call-clustered bootstrap interval |
| Decision | ship, block or ship with conditions; who signed; which tripwires are armed |
The judge row links to the judges chapter, which owns judge agreement statistics; this report only records which judge version scored the run and whether it was allowed to gate. Rare-event bounds for violations seen in production belong to the monitoring chapter, not to a pre-release report.
What gate would have caught it?
Nightly audio runs of 240 scenarios at k = 4, 20 per gated segment, with a per-segment paired delta against the current production version, Holm across the 12 segments and an allowed drop of 5 points. The flow-group line and per-segment pass^4 sit beside it, and no release ships on a single trial.
Replay the incident through that gate. The dispute and already-paid flow group now has 60 scenarios across three languages instead of 4, each run 4 times. At 80% per trial the chance that all 240 trials pass is 0.8^240, about 5.5 x 10^-24, so the "all green" that shipped v3.2 cannot recur. Each dispute segment's pass^4 lands near 0.41 instead of the 0.81 a 95% flow would show. If the old version ran at 95% (the planner's assumption), each language segment blocks on its own about 26% of the time, and at least one of the three blocks about 60% of the time, treating them as independent. The flow-group test blocks about 89% of the time.
That last gap is the lesson of this chapter in one line: the per-segment gate is a tripwire for large, concentrated regressions, the flow-group test catches broad ones, and pass^k on the report shows the reader what the borrower experiences. The pass^k planner recipe runs the gate end to end on seeded data.
What goes in the decision memo?
The memo turns the gate into a decision someone owns. Tarazu's release-gate memo:
Decision. No release ships on fewer than 240 nightly audio scenarios at k = 4. Each of the 12 gated segments gets a paired delta against production, Holm-corrected, blocking when the one-sided bound excludes a 5-point drop; the 4 flow groups get the same test as a second family.
Context. Asha v3.2 shipped on one green run of 60 scenarios with 4 in the dispute segment; the dispute flow then failed 1 call in 5 (illustrative).
Options considered. (a) One trial per scenario: what failed. (b) A pooled suite gate: hides concentrated regressions (pooled -0.022 against a segment -0.300 in the recipe). (c) Uncorrected per-segment gates: 46% false blocks on a healthy release. (d) Per-segment Holm gate plus a flow-group family: chosen.
Evidence. gold-v7 scenario cards; unbiased pass^4 per segment; planner power about 0.26 per segment and about 0.89 per flow group against a 95% -> 80% drop; speaker-clustered intervals for entity error rate.
Owner and cadence. Eval platform staff engineer owns the gate and report; nightly, and before every release.
Tripwires. Any flow group blocks while no segment does; live-simulation variance exceeds replay variance; a gating judge changes version.
Dated facts. tau-bench (June 2024); tau-Voice paper (March 2026); Thinking Machines (September 2025).
What to do this week
Pull last month's release runs and recompute every headline with n, k and an interval. Any segment whose interval spans more than your allowed drop was never gated.
- Store per-scenario success counts (c, n) for every run, not only suite pass rates, and compute pass^k with C(c,k)/C(n,k).
- Move the release gate to at least k trials per scenario and report pass^1 beside pass^k.
- Run one scenario sample twice, frozen-audio replay and live simulation, and compare the variance.
- Replace any utterance-level or Wilson interval on WER or entity error rate with a speaker-clustered bootstrap, and score version deltas on paired replicates.
- Write down your gated segments and monitored slices, then compute the gate's power against the drop you most fear before you trust a green run.
NOTE
What will date on this page. tau-bench figures (arXiv 2406.12045 v1, June 2024; tau2-bench and tau3 keep the same metric; checked 2026-11-20). The Codex paper's estimator results (arXiv 2107.03374 v2, July 2021). Anthropic's "Demystifying evals for AI agents" (published 9 January 2026, checked 2026-11-20). Atil et al. on temperature-0 variance (arXiv 2408.04667) and Thinking Machines Lab on batch invariance (September 2025); serving stacks change, so re-test determinism on yours. tau-Voice's paired permutation results (arXiv 2603.13686 v1, March 2026); Sierra's leaderboard has since moved on newer benchmark versions, and those scores are not comparable. Planner power figures are simulations under stated assumptions.