Steelyard Credit released Hazel v3.2 on the strength of a single all-green pass over 60 scenarios, only 4 of which exercised the dispute flow. Within a month, 1 dispute call in 5 was going wrong in production. The run reported honestly; the gate asked too little of it: one trial per scenario, four scenarios in the flow that later broke, and a rule that read 60 passes as safe. This chapter builds the statistics that would have stopped it.
What happened at release?
One trial per scenario, 60 passes, ship. Four of the passes came from dispute scenarios, and four passes say little: a dispute flow that fails 20% of calls still goes 4 for 4 with probability 0.8^4 = 0.41, so the gate had roughly a two-in-five chance of waving this exact regression through.
release gate run Hazel v3.2 candidate
scenarios 60
trials per scenario 1
result 60 / 60 pass -> shipped
dispute segment 4 scenarios, 4 / 4 pass
one month later dispute flow fails 1 call in 5 in production
Diagnosis: the gate checked that one draw came up green, on too few dispute scenarios to bound a rate, and never measured how often a borrower gets a good call every time.
The run carried the warning itself. Put a Wilson 95% interval on 4 successes out of 4 and you get 0.51 to 1.00: the data were consistent with a dispute flow that works half the time. To push the Wilson lower bound past 0.90 with no failures at all, you need 35 passes out of 35 (34 of 34 stops at 0.899). A gate that reports a point estimate and discards the interval is checking a box, not testing a hypothesis.
Why does every-trial success fit collections?
pass^k is the chance that k independent trials of one task all succeed, averaged over the tasks in a suite. It fits collections because one borrower meets the same flow several times within one days past due (DPD) bucket. A flow that works on 80% of calls works on 8 calls running only 0.8^8 = 16.8% of the time.
The metric comes from tau-bench (Yao et al., Sierra, 2024), which reports pass^k next to the usual single-trial score. In the paper's main results its best agent, gpt-4o with function calling, reached pass^1 of 61.2% on the retail domain and 35.2% on airline, and its retail pass^8 dropped below 25% 1. The single-trial score describes the average call; pass^k describes how often every call is good, which is what collections owes a borrower.
CAUTION
Vendor-reported. Treat it as a hypothesis to test on your own calls.
Anthropic's guidance on agent evals works a smaller case: an agent that succeeds on 75% of trials succeeds on all 3 of 3 trials about 42% of the time, and the post recommends pass^k where consistency matters, naming customer-facing agents 2. Raising one rate to a power is right for one scenario only. A suite's pass^k averages each scenario's own p^k, and that average is never below (mean p)^k. Take two scenarios, one at p = 1.0 and one at p = 0.6. Their mean rate is 0.80, so the shortcut predicts 0.8^3 = 0.512 for three in a row; the true suite pass^3 is (1 + 0.216) / 2 = 0.608. Spread in difficulty moves the answer, which is why every estimate below starts from per-scenario counts.
Figure 1: pass^k = p^k for one scenario with independent trials. The shaded band is the run of conversations a Steelyard borrower can have while the account sits in the 1-30 DPD bucket; at Steelyard's k = 4, a flow at 80% per call gives the borrower four good calls 41.0% of the time, and at k = 8 only 16.8%. Computed curves, not Steelyard measurements.
Pick k from how the deployment exposes a borrower, not from a benchmark convention. Steelyard collects its own debts in its own name, so the FDCPA and Regulation F largely do not bind it 3 4; it adopts their conduct rules as company policy, and state law (California's Rosenthal Act, Massachusetts 940 CMR 7.00), the TCPA and the Consumer Financial Protection Act's ban on unfair, deceptive or abusive practices still apply. One of the adopted rules sets k almost by itself. Under Regulation F's call-frequency presumption, which Steelyard adopts as C11, a telephone conversation about a debt starts 7 consecutive days with no further calls to that person about it, the conversation date counting as day one 5. A borrower who stays in the 1-30 DPD bucket can therefore talk to Hazel about four times before the account moves on (30 / 7 = 4.3; five at most, on days 1, 8, 15, 22 and 29), and each of those calls is rationed: a bad one is not retried tomorrow, it costs the account a week. Four exposures is the realistic run, so Steelyard runs k = 4 nightly. The UK book has no numeric anchor of this kind. CONC 7.9 bans contact at unreasonable times and requires due regard to a customer's reasonable requests about contact, but it sets no call-frequency cap 6, so a UK lender has to derive k from its own contact policy and the contact history of its accounts.
Two things cap k from above. The estimator needs at least k trials per scenario, and every audio trial is a simulated call that chapter 14 puts a price on. Whatever k you choose, publish pass^1 next to pass^k: the first describes the typical call, the second the borrower who receives four. pass^k and consistency gives the general treatment, including why pass@k points the wrong way for this question.
How do you estimate every-trial success from repeated runs?
Give each scenario n trials (n at least k), count its successes c, compute C(c,k)/C(n,k) and average over scenarios. This tau-bench estimator is unbiased for each scenario's p^k. Raising the observed rate to the k-th power is not: it runs high for one scenario and low when applied to a pooled mean.
Read C(c,k)/C(n,k) as a count of subsets. Of all the ways to choose k of the n trials, it is the share in which every chosen trial passed. A scenario with 6 passes in 8 trials, at k = 4, scores C(6,4)/C(8,4) = 15/70 = 0.214. The tempting shortcut, (6/8)^4, scores 0.316, which overstates by about half.
The bias has a known sign. The Codex paper (Chen et al., OpenAI, 2021) drew n = 200 samples per problem and showed that the plug-in estimate 1 - (1 - p_hat)^k consistently underestimates pass@k, with a gap that persists even for n > 5k, so it adopted an unbiased per-problem estimator 7. The extension to pass^k is our reasoning, not a claim of that paper: the plug-in p_hat^k is biased upward per scenario, the mirror image of the pass@k case, because the expected value of p_hat^k is at least p^k. Take a scenario whose true per-trial rate is 80%. Its true pass^4 is 0.410. The plug-in estimate averages 0.549 when n = 4 and 0.483 when n = 8; the unbiased estimator averages 0.410 at any n.
Unbiased does not mean tight. When n = k = 4, C(c,4)/C(4,4) can only be 1 (all four passed) or 0, so a single scenario at p = 0.8 yields an estimate with a standard deviation of 0.49. More trials help slowly: 0.31 at n = 8, 0.25 at n = 12, 0.21 at n = 16. Averaging across scenarios helps far more, which is why Steelyard spends its simulation budget on 240 scenarios at k = 4 instead of 60 scenarios at k = 16.
The pass^k planner recipe generates a seeded synthetic suite to show the estimators side by side: 60 scenarios, a few of them hard, 8 trials each (seed 20261120). Synthetic Steelyard data.
| k | Unbiased pass^k | Per-scenario (c/n)^k | Pooled (mean p)^k | pass@k |
|---|
| 1 | 0.969 | 0.969 | 0.969 | 0.969 |
| 4 | 0.886 | 0.904 | 0.881 | 1.000 |
| 8 | 0.800 | 0.857 | 0.776 | 1.000 |
The columns agree at k = 1, where no power is taken and there is no bias to show. By k = 8 the per-scenario shortcut is 5.7 points too high and the pooled shortcut 2.4 points too low. The generating rates give a true suite pass^8 of 0.751, so on this particular draw the unbiased estimate also lands 4.9 points high. That is sampling error, not bias: the estimator is right on average over many suites, and any one 60-scenario suite can miss. pass@k reads 1.000 from k = 4 onward because it rewards one success in k tries, so it has no place on a collections release report.
Two operating rules come out of this. Keep the per-scenario pair (c, n) for every run, because a suite pass rate cannot be turned back into pass^k. And refuse to report pass^k for any scenario with fewer than k trials: the estimator is undefined there, and a number produced anyway has no statistical meaning.
Why do runs vary at temperature 0?
Identical runs at temperature 0 still give different LLM outputs and scores, mostly because inference kernels are not batch-invariant, so a response depends on what else the server is batching. A voice stack adds four sampling points of its own: streaming STT, VAD and turn detection, TTS, and the caller simulator.
The LLM share has been measured. Atil et al. ran five models 10 times each at temperature 0, top-p 1 and a fixed seed and found gaps of up to 15% between the best and worst accuracy 8. Thinking Machines Lab ran one prompt 1,000 times at temperature 0 on Qwen3-235B in vLLM, got 80 distinct completions, attributes this mainly to kernels that are not batch-invariant, and made all 1,000 identical by switching to batch-invariant kernels 9. For Steelyard this matters more than for an offline benchmark: the nightly run and the 17:00-19:00 Central peak put very different batches on the same servers, so the nightly number describes a load profile production never sees. The site's non-determinism chapter covers the general case.
Each voice component adds variance through a mechanism you can test directly:
- STT. A streaming recognizer finalizes words at chunk boundaries, and those boundaries fall on 20 ms frames. Shift the alignment by one frame and "fifty" can come back as "fifteen" on one run and correctly on the next. That is T01's error, seen as a probability per call rather than a single bad day.
- VAD and the turn detector. End of turn is a comparison between measured silence and a budget. A pause that sits near the budget closes the turn on some runs and holds it on others; T05, where Ryan Mitchell's turn was cut before "fifth", is the run where the commit landed inside the pause.
- TTS. A neural voice can render the same sentence at slightly different lengths. That moves where a caller's interruption falls against the agent's words, for example whether Sofia Ramirez's "ya lo pague" [I already paid it] lands during the amount or after it, as in T02.
- The simulator. The caller model samples too, and Steelyard's interruption scripts trigger on content rather than on a timer, so a barge-in moves whenever the agent's wording moves.
Measure the sources separately rather than trying to switch them off. Take a sample of scenarios and run it twice: once by replaying frozen caller audio into the agent, which leaves only agent-side variance, and once with the live simulator, which adds the caller's. If live variance is much larger than replay variance, the simulator dominates and fixing seeds inside the agent changes nothing that matters; the simulation chapter shows how to validate the simulator itself. A fixed seed and a single run do not make a system deterministic. They pick one draw from the distribution and report it as if it were the whole.
How many runs do you need?
Pick the smallest drop you refuse to ship (the minimum detectable effect), then solve for run count at a chosen power and alpha. Detecting a fall from 90% to 85% with 80% power at one-sided alpha 0.05 takes about 540 scenarios per arm on separate scenarios, and about 220 when both versions run on the same ones.
The paired figure depends on the discordant cells: the scenarios where the two versions disagree. Suppose 7% of scenarios pass on the old version and fail on the new, and 2% go the other way. McNemar's test then needs 221 paired scenarios 10. Scenarios where both versions agree say nothing about the difference, so pairing spends the whole budget on the cases that do. The more alike the two versions are, the fewer discordant scenarios there are and the bigger the saving. Evan Miller's paper on error bars for evals recommends paired differences for the same reason when two models answer one question set 11. In a voice eval, a pair has to share more than the scenario card: the same audio condition, the same codec chain and, on replay runs, the same caller audio.
Plan power on the unit you resample. Trials within one scenario are correlated, since a hard scenario keeps failing, so 240 scenarios at k = 4 are not 960 independent observations. Count scenarios for power, and treat the k trials as a way to measure each scenario more precisely. The site's power tables handle the case without clustering.
The planner's output for three decisions follows. Assumptions: all scenarios in a segment share one per-trial rate, trials are independent, and the simulated rows use 4,000 simulated releases at seed 20261120. The 95% rate for the old version is an assumption chosen for the arithmetic, not a Steelyard measurement.
| Decision | Design | Scenarios | Alpha | Power or hit rate |
|---|
| 1. Suite-wide drop, 90% -> 85% | Unpaired two-proportion test | 540 per arm | 0.05 | 0.80 (closed form) |
| 1. Suite-wide drop, 90% -> 85% | Paired McNemar, 7% / 2% discordant | 221 | 0.05 | 0.80 (closed form) |
| 2. One gated segment, 95% -> 80% | Paired delta on scenario means at k = 4, allowed drop 5 points | 20 | 0.0042 (0.05/12) | about 0.26 (simulated) |
| 2. One gated segment, 95% -> 80% | Same test without correction | 20 | 0.05 | about 0.63 (simulated) |
| 3. One flow group, 95% -> 80% | Paired delta pooled over 3 language segments | 60 | 0.0125 (0.05/4) | about 0.89 (simulated) |
Row 2 is the uncomfortable one. With 20 scenarios in a segment and a correction for 12 segments, even a 15-point drop is caught only about a quarter of the time. That is what control of false alarms costs at this size, and the gate design below decides how to pay for it.
tau-Voice gives a sense of the scale benchmark authors need to settle a voice gap. Its main comparisons used paired permutation tests, not confidence intervals on individual trials: 2 independent runs per condition on the Retail domain (114 tasks, 228 simulations per condition), pairs matched by task ID, 100,000 two-sided permutations and Holm-Bonferroni correction within each provider. Moving from Clean to Realistic audio cost Google 13.2 points (42.1% to 28.9%, adjusted p = 0.026), xAI 13.2 points (50.0% to 36.8%, p = 0.044) and OpenAI 24.6 points (67.5% to 43.0%, p < 0.001; the paper's figure, from unrounded rates) 12. How we read those p-values, which the paper does not say: a 13-point gap over 114 paired tasks barely clears 0.05. A 20-scenario segment has no chance of resolving a 5-point one.
How do you put an interval on WER and entity error rate?
Resample speakers, not utterances, and on each replicate recompute total errors over total reference words. One speaker's utterances share a voice, a handset and a room, so treating them as independent makes the interval too narrow. To compare two versions, score both on the same resampled speakers.
Corpus WER is a ratio: all substitutions, deletions and insertions summed, over all reference words summed. It is not the mean of per-utterance WERs, and no binomial formula gives its interval. Entity error rate has the same form (entity errors over reference entities), clustered by speaker or by call; the entity chapter owns its definition. Cameron and Miller's practitioner's guide is the standard account of why clustered data needs either a cluster-level resample or a cluster-adjusted variance 13, and the site's clustered standard errors page works the formula.
The script copies the shape of Steelyard's gold-v7 STT set, 1,200 reference utterances from 180 speakers, and fills it with synthetic errors (seed 20261121). Synthetic Steelyard data.
"""Speaker-clustered bootstrap for corpus WER and a paired WER delta.
Synthetic Steelyard-shaped data: 1,200 utterances from 180 speakers. Standard library only."""
import math
import random
rng = random.Random(20261121)
WORDS = 12 # reference words per utterance
# 120 speakers x 7 utterances + 60 x 6 = 1,200. Speaker difficulty is skewed.
rows = [] # (speaker, ref_words, errors_old, errors_new)
for spk in range(180):
rate = min(0.6, 0.08 * math.exp(rng.gauss(0, 0.7)))
for _ in range(7 if spk < 120 else 6):
old = sum(rng.random() < rate for _ in range(WORDS))
new = sum(rng.random() < rate * 0.92 for _ in range(WORDS))
rows.append((spk, WORDS, old, new))
def wer(sample, col):
return sum(r[col] for r in sample) / sum(r[1] for r in sample)
by_spk = {}
for r in rows:
by_spk.setdefault(r[0], []).append(r)
speakers = list(by_spk)
def ci(stats):
stats.sort()
return stats[int(0.025 * len(stats))], stats[int(0.975 * len(stats))]
B = 2000
naive, clustered, delta, unpaired = [], [], [], []
for _ in range(B):
utt = [rows[rng.randrange(len(rows))] for _ in rows]
naive.append(wer(utt, 2))
spk = [r for _ in speakers for r in by_spk[speakers[rng.randrange(len(speakers))]]]
clustered.append(wer(spk, 2))
delta.append(wer(spk, 3) - wer(spk, 2)) # same speakers, both versions: paired
other = [r for _ in speakers for r in by_spk[speakers[rng.randrange(len(speakers))]]]
unpaired.append(wer(other, 3) - wer(spk, 2)) # independent resamples: ignores pairing
lo_n, hi_n = ci(naive)
lo_c, hi_c = ci(clustered)
lo_d, hi_d = ci(delta)
lo_u, hi_u = ci(unpaired)
print(f"utterances {len(rows)}, speakers {len(speakers)}")
print(f"corpus WER old {wer(rows, 2):.4f}, new {wer(rows, 3):.4f}")
print(f"utterance bootstrap 95% CI [{lo_n:.4f}, {hi_n:.4f}] width {hi_n - lo_n:.4f}")
print(f"speaker bootstrap 95% CI [{lo_c:.4f}, {hi_c:.4f}] width {hi_c - lo_c:.4f}")
print(f"width ratio {(hi_c - lo_c) / (hi_n - lo_n):.2f}")
print(f"paired delta new-old {wer(rows, 3) - wer(rows, 2):+.4f}, speaker bootstrap 95% CI [{lo_d:+.4f}, {hi_d:+.4f}]")
print(f"same delta, resampled as if unpaired: [{lo_u:+.4f}, {hi_u:+.4f}]")
Running it with Python 3.11 prints:
utterances 1200, speakers 180
corpus WER old 0.1124, new 0.0977
utterance bootstrap 95% CI [0.1058, 0.1195] width 0.0137
speaker bootstrap 95% CI [0.0995, 0.1258] width 0.0263
width ratio 1.91
paired delta new-old -0.0147, speaker bootstrap 95% CI [-0.0212, -0.0083]
same delta, resampled as if unpaired: [-0.0325, +0.0031]
Three things to read off it. First, the speaker-clustered interval is 1.91 times as wide as the utterance-level one, and with about 6.7 utterances per speaker that ratio is typical rather than extreme; the narrow interval was never real. Second, the new version's 1.47-point improvement has a paired interval of -2.12 to -0.83 points, wholly below zero, so it is a claim the team can make. Third, resample the same data as if the versions had been scored on different speakers and the interval widens to -3.25 to +0.31 points, 2.8 times as wide, and crosses zero. Same transcripts, same errors; only the pairing turned the evidence into a result.
Match the interval method to the metric. Wilson is for one binary proportion over independent units, such as the share of scenarios that pass 14. A difference between two proportions on shared scenarios takes McNemar or a paired bootstrap; on independent samples, Newcombe's method. WER and entity error rate take a bootstrap clustered by speaker or call, and a percentile such as P95 turn gap takes a bootstrap clustered by call. A Wilson interval on WER, on clustered utterances or on a latency percentile has the look of rigor and the wrong width. The general methods are on the site's confidence intervals and bootstrap and paired tests pages.
How do you gate on segments without drowning in false alarms?
Test each of the 12 segments with a paired candidate-minus-control delta, scenario as the unit, Holm-corrected across the 12, and block when the one-sided bound rules out staying within the allowed drop. Add a second Holm family that pools each flow group across languages, and report other slices without gating them.
Steelyard's 12 gated segments cross 3 language groups (English, Spanish-dominant, Spanglish) with 4 flow groups: promise (F01, F09); dispute and already paid (F02, F03); hardship and complaint (F04, F10); and other parties and machines (F05 to F08). DPD bucket, carrier, hour, state and code-switch density are monitored slices: every release reports them, and none of them can block. The split is a budget decision. A gated cell needs enough scenarios to mean something, and each added gated cell spends power through the correction. Language groups come from the account preference and the call, never from a surname.
Figure 2: The nightly 240-scenario run laid out as Steelyard's gate sees it. Every cell is a gated segment; every column is also pooled into a flow-group test. Power figures are the planner's simulations under its stated assumptions (old version at 95% per trial, seed 20261120), not Steelyard measurements.
Leave the 12 tests uncorrected at 0.05 each and a healthy release triggers at least one false block with probability 1 - 0.95^12 = 46%, if the tests were independent. A gate that blocks every other good release gets overridden, and the team learns to click past it. Holm's step-down procedure keeps the family-wise false-block rate at or under 5% whatever the dependence between tests: order the 12 p-values, test the smallest against 0.05/12, the next against 0.05/11, and stop at the first that does not clear 15. It blocks everything Bonferroni blocks and sometimes more. When some false blocks are acceptable in exchange for power, Benjamini-Hochberg and the other options are on the multiple comparisons page.
Mind the direction of the test. Steelyard's null hypothesis is "this segment dropped by no more than the allowed 5 points", and a block needs evidence against it. That keeps false blocks rare and puts the burden of proof on the alarm, which suits a nightly gate, but it also means a segment with thin evidence passes. Row 2 of the planner prices that: about 26% power against a 15-point drop in a single segment.
Decide before the first block what a rerun means. A team that reruns a blocked night and ships on the first green result has quietly changed the gate. Take a segment that really dropped from 95% to 80% and blocks on 26% of nights: one run lets it through 74% of the time, and "rerun once if blocked" lets it through 1 - 0.26^2 = 93% of the time. Steelyard's rule is that a rerun adds its trials to the blocked run's and the test runs once on the pooled data; anything else is an override, signed by name on the report.
Two additions buy back most of the lost power without adding scenarios:
- A flow-group family. Pool the three language segments of each flow group, 60 scenarios, and run the same paired test with Holm across the 4 flow groups. Against the planner's 95% -> 80% drop it fires about 89% of the time. A dispute regression that touches English, Spanish-dominant and Spanglish calls alike shows up here even when no individual segment blocks.
- Per-segment pass^4 on the release report. At n = k = 4 a scenario's estimate is 1 only if all four trials passed, so a segment's pass^4 is the share of its 20 scenarios that went 4 for 4. At 95% per trial the expected value is 0.81; at 80% it is 0.41. A floor of 0.60 would catch the 80% segment 93% of the time, with a 0.6% false-alarm rate per healthy segment at 95%, or about 7% across 12. Floors are absolute, though: a segment that properly runs at 90% per trial would trip a 0.60 floor about 22% of nights. Set each floor from that segment's own history, or report pass^4 and leave it ungated.
The recipe's seeded run (seed 20261120) shows both levels at work. It plants a regression in one dispute and already paid segment: the segment's paired mean moves from 0.912 to 0.613, a delta of -0.300 with a one-sided upper bound of -0.163 and a Holm-adjusted p of 0.006, and the gate blocks. The new pass^4 in that segment is 0.300, meaning 6 of its 20 scenarios passed all four trials. Over all 240 scenarios the pooled delta is only -0.022, which a suite-level gate would file as noise. Pooling dilutes a concentrated regression; segmenting without correction manufactures regressions that are not there. The gate needs both levels at once.
What does a release report look like?
Every number carries its estimate, interval and interval method, n, k, and the versions of each model, judge and dataset behind it, followed by per-segment paired deltas with Holm-adjusted p-values and the decision. A number without n, k or versions cannot be reproduced or compared with next week's.
| Field | Contents |
|---|
| Versions | agent (hazel-3.2.1), prompt (collections-v17), STT (vendorA-2026-10), turn detector (td-1.1), LLM (model-x-2026-09-30), TTS (vendorB-v4@8k), policy pack (us-regf-2026-11), taxonomy (gl-fail-v4); simulator version and replay or live mode |
| Dataset | gold set release (gold-v7), scenario count, how the nightly 240 were sampled from its 600 cards |
| Judges | each judge id and version with its validation status per language group; J1 failed Spanglish validation, is advisory and never gates |
| Run shape | n scenarios, k trials, temperature, kernel and serving settings that affect determinism |
| Task success | pass^1 with a scenario-clustered interval; pass^4 by the unbiased estimator |
| Segment table | for each gated segment: control, candidate, paired delta, one-sided bound, raw p, Holm p, pass^4, decision |
| Flow-group table | the same columns for the 4 pooled flow groups |
| Monitored slices | DPD bucket, carrier, hour, state, code-switch density: estimates and intervals, no decision |
| Entity and STT | amount and date entity error rate with speaker-clustered intervals; WER as a diagnostic only |
| Latency | P95 platform turn gap by turn type, with an interval from a bootstrap clustered by call |
| Decision | ship, block, or ship with conditions; who signed; which tripwires are armed |
The judge row points to the judges chapter, which owns judge agreement statistics; the report records only which judge version scored the run and whether it was allowed to gate. Bounds on rare violations seen in production belong to the monitoring chapter and are not part of a pre-release report.
What gate would have caught it?
Run 240 audio scenarios nightly at k = 4, 20 in each gated segment. Compare each segment with the current production version by paired delta, Holm-correct across the 12, and allow a 5-point drop. Run the flow-group family and report per-segment pass^4 alongside it, and never release on a single trial per scenario.
Now replay Hazel v3.2 through that gate. The dispute and already paid flow group holds 60 scenarios across English, Spanish-dominant and Spanglish instead of 4, and each one runs 4 times. A flow at 80% per trial passes all 240 trials with probability 0.8^240, about 5.5 x 10^-24, so the green run that shipped v3.2 cannot happen again. Each dispute segment's pass^4 lands near 0.41, where a healthy 95% flow would sit near 0.81. If the production version runs at 95% (the planner's assumption), each language segment blocks on its own about 26% of the time, at least one of the three blocks about 60% of the time if they are treated as independent, and the flow-group test blocks about 89% of the time.
That spread is the design. The per-segment gate is a tripwire for large regressions concentrated in one cell, the flow-group test catches regressions that spread across languages, and pass^k on the report tells a reader what a borrower who gets four calls lives through. The pass^k planner recipe runs the whole gate end to end on seeded data.
What goes in the decision memo?
The memo names one owner for the release gate and records the decision, the incident behind it, the options rejected, the evidence with its intervals, the cadence, the tripwires that reopen it and the facts that will date. Steelyard's version follows.
Decision. Every release needs the nightly audio run of 240 scenarios at k = 4 to pass. Each of the 12 gated segments (3 language groups x 4 flow groups) is tested by paired delta against production with Holm correction and blocks when its one-sided bound excludes a 5-point drop; the 4 flow groups form a second Holm family with the same test.
Context. Hazel v3.2 shipped after one green run of 60 scenarios, 4 of them in the dispute flow, which then failed 1 call in 5 (illustrative). Under C11, each failed conversation also costs that account a week of contact.
Options considered. (a) One trial per scenario: the gate that failed. (b) A pooled suite gate: misses concentrated regressions (recipe: pooled -0.022 against -0.300 in one segment). (c) Per-segment gates without correction: about 46% false blocks on a healthy release. (d) Holm-corrected segments plus a flow-group family: chosen.
Evidence. gold-v7 scenario cards; unbiased pass^4 per segment; planner power about 0.26 per segment and 0.89 per flow group against a 95% -> 80% drop; paired, speaker-clustered intervals for entity error rate and WER.
Owner and cadence. The eval platform staff engineer owns the gate and the report; it runs nightly and before every release.
Tripwires. A flow group blocks while no segment does; live-simulation variance exceeds replay variance; a gating judge changes version.
Dated facts. tau-bench (June 2024); tau-Voice (March 2026); Thinking Machines Lab (September 2025); Regulation F 1006.14 as read 2026-11-20.
What to do this week
Take last month's release runs and rewrite every headline number with its n, k and interval. Any segment whose interval is wider than the drop you say you would block was never actually gated.
- Save the per-scenario counts (c, n) for every run, not only the suite rate, and compute pass^k as C(c,k)/C(n,k).
- Require at least k trials per scenario at release, and print pass^1 next to pass^k.
- Derive k from your contact rules: count how many conversations one borrower can have with the agent before the account changes bucket.
- Run a sample of scenarios two ways, frozen-audio replay and live simulation, and compare the variance of the two.
- Swap any utterance-level or Wilson interval on WER or entity error rate for a speaker-clustered bootstrap, and score version deltas on paired replicates.
- List your gated segments and monitored slices, then compute the gate's power against the drop you fear most before you trust the next green run.
NOTE
What will date on this page. tau-bench figures (arXiv 2406.12045 v1, June 2024; tau2-bench and tau3 keep the same metric; checked 2026-11-20). The Codex paper's estimator results (arXiv 2107.03374 v2, July 2021). Anthropic's "Demystifying evals for AI agents" (published 9 January 2026, checked 2026-11-20). Atil et al. on temperature-0 variance (arXiv 2408.04667) and Thinking Machines Lab on batch invariance (September 2025); serving stacks change, so re-test determinism on your own. tau-Voice's paired permutation results (arXiv 2603.13686 v1, March 2026); Sierra's leaderboard has since moved to newer benchmark versions whose scores are not comparable. Regulation F 1006.14, the FDCPA definitions and CONC 7.9 were read 2026-11-20. Planner power figures are simulations under stated assumptions; recipe numbers are tied to seed 20261120 and the WER example to seed 20261121.