Hazel v3.2, Steelyard Credit's collections agent, shipped after one green run of 60 scenarios. The dispute segment had 4 of them. A month later the dispute flow failed 1 call in 5. Nothing in that release run was wrong. It was one trial per scenario, 4 scenarios where it mattered most, and a gate that read "all passed" as "safe".
This recipe builds the arithmetic that would have stopped that release. A collections agent does not get one try: the same flow runs thousands of times a day, so the question is how often every run succeeds, not whether one did. If each call independently succeeds 80% of the time, the chance that 8 calls in a row all succeed is 0.8^8, about 17%. tau-bench calls this reliability metric pass^k, the chance that all k trials of the same task succeed, and in that paper the gpt-4o function-calling agent scored about 61% pass^1 on tau-retail but fell below 25% at pass^8 1. The concepts live in the voice reliability statistics chapter; the general treatment is in pass^k and consistency.
The 30 minutes: five to set up and run, twenty to read the five output blocks against the decisions they gate, five to point it at your own trial results.
The script and synthetic data for this recipe are also in the ai-evals-course repository on GitHub, in the folder voice-ai/labs/voice-pass-k-planner/. That copy also carries the India edition's constants block, which this page leaves out.
What you build
One standard-library script, voice-pass-k-planner.py, with five parts:
- pass^k and pass@k for one task.
pass_hat_k(c, n, k) is the tau-bench estimator C(c,k)/C(n,k) for c successes in n trials. pass_at_k(c, n, k) is its mirror, 1 - C(n-c,k)/C(n,k), the chance that at least one of k trials succeeds.
- Suite estimators on 60 scenarios x 8 trials. Synthetic per-scenario success rates with a hard tail, and a dispute segment of 4 scenarios whose true rate averages 0.80. The unbiased suite pass^k sits beside two shortcuts people use: averaging (c/n)^k per task, and raising the suite's mean pass rate to the power k.
- Intervals for single rates. Wilson and exact Clopper-Pearson, applied to the release run and to the dispute segment, plus the number of all-pass scenarios it takes to clear a 0.90 bar.
- Run counts. Closed-form sample sizes for detecting a drop from 90% to 85% at 80% power, one-sided alpha 0.05: unpaired (two independent samples of scenarios) and paired (McNemar on the same scenarios run on both versions). A seeded power simulation checks both.
- A release gate over the 12 gated segments. Steelyard gates on 3 language groups (English, Spanish-dominant, Spanglish) x 4 flow groups (promise; dispute and already paid; hardship and complaint; other parties and machines). Old and new agent versions each run 240 scenarios, 20 per segment, at k=4, with one seeded regression. Each segment gets a paired bootstrap delta, and Holm corrects across the 12.
All data is synthetic. The dataset is named "steelyard-calls (synthetic, seed 20261121)", and the random draws use one statistics stream, seed 20261120, in both editions, so every computed number below matches the India edition and the chapter. The trial outcomes are coin flips at rates the script sets; they describe no real agent.
Step 1: set up a fresh environment
The script needs nothing outside the standard library, so the venv has no installs. Use a fresh one anyway, so nothing from your global site-packages leaks in:
python3.11 -m venv .venv-r9
.venv-r9/bin/python --version # Python 3.11.x
Step 2: save the script
Every recipe in this voice series starts with the same GLOBAL constants block, so the fictional lender's products, campaigns, calling windows and Customer Advocate stay identical across recipes. This one adds the nightly run shape (240 scenarios, k=4, 20 per gated segment) and the three language groups. The script also takes --edition global|india (default global). The listing below is the global edition: it leaves out the India edition's constants block, which sits between two marker comments in the full file in the course repository. Without that block, --edition india exits with a usage error. Save this as voice-pass-k-planner.py:
"""pass^k and a run-count planner for voice agent release gates (aievals.co recipe R9).
Dataset: steelyard-calls (synthetic, seed 20261121). Standard library only.
Python 3.11. No network. Seeded synthetic data.
--edition india switches to the India edition's fictional lender and segment labels
(its constants block sits between the INDIA EDITION markers below). The statistics
are the same in both editions, so every computed line prints the same numbers.
Five parts:
1. pass^k and pass@k for one task, exact and unbiased.
2. Suite estimators on 60 scenarios x 8 trials, against the plug-in shortcuts.
3. Wilson and Clopper-Pearson intervals for single rates (the dispute segment).
4. Run counts: closed-form unpaired and paired (McNemar) sizes, checked by a
seeded power simulation.
5. A release gate over the 12 gated segments: paired bootstrap deltas per
segment, Holm across segments.
Run:
python3.11 voice-pass-k-planner.py # global edition
python3.11 voice-pass-k-planner.py --edition india # India edition
"""
from __future__ import annotations
import argparse
import math
import random
from math import comb, sqrt
from statistics import NormalDist
# ---- GLOBAL constants (copied from the world bible, world-global-v1.1) ---------
GLOBAL = {
"lender": "Steelyard Credit, Inc. (US lender, head office Tucson AZ)",
"fiction_note": "Synthetic data.",
"dataset": "steelyard-calls (synthetic, seed 20261121)",
"seed": 20261121,
"products": { # code: (ticket $ range, payment $ range; SDL payments are biweekly)
"AUTO": ((8_000, 35_000), (245, 680)),
"PIL": ((1_000, 40_000), (95, 1_850)),
"POS": ((300, 6_000), (45, 520)),
"SBL": ((25_000, 250_000), (980, 7_200)),
"SDL": ((300, 3_000), (35, 140)),
},
"campaigns": {"AUTO-early-nov": "AUTO", "PIL-digital-nov": "PIL", "POS-short-nov": "POS",
"SBL-guar-nov": "SBL", "SDL-biwk-nov": "SDL"},
"windows": {"default": ("08:00", "21:00"), "SDL": ("09:00", "20:00")}, # borrower's local time
"languages": {"English": 64, "Spanish-dominant": 18, "Spanglish": 14, "other": 4},
"bot_languages": ("English", "Spanish-dominant", "Spanglish"), # other -> human
"customer_advocate": {"name": "Ms Dana Brooks", "email": "advocate@steelyard.example",
"phone": "(520) 555-0199",
"address": "1200 Example Avenue, Tucson, AZ 85701"},
"money": "integer cents in tools and CRM; dollars in speech",
"dates": "ISO in tools; words in speech; en-US month-first, es-US day-first",
"versions": {"agent": "hazel-3.2.1", "prompt": "collections-v17", "gold": "gold-v7"},
"nightly": {"scenarios": 240, "k": 4, "per_segment": 20, "segments": 12},
# WG15: 3 language groups x 4 flow groups = 12 gated segments
"lang_groups": ("English", "Spanish-dominant", "Spanglish"),
}
EDITIONS = {"global": GLOBAL}
# ---- INDIA EDITION START --------------------------------------------------------
# (left out of this listing: the India edition's constants block, which adds
# EDITIONS["india"]; it is in the full file in the course repository)
# ---- INDIA EDITION END ----------------------------------------------------------
ARGS = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ARGS.add_argument("--edition", choices=list(EDITIONS), default="global",
help="which fictional world's data to use (default: global)")
WORLD = EDITIONS[ARGS.parse_args().edition]
# One random stream family for the suite, the release run, the power simulations and
# the gate, the same in both editions, so every computed line prints the same numbers
# in both. The edition seed names the edition's dataset.
STATS_SEED = 20261120
# Language groups map by position across editions (lead language group, the second
# group, the third), so the seeded regression lands in the same cell in both.
LANG_GROUPS = WORLD["lang_groups"]
LANG_W = max(15, max(len(x) for x in LANG_GROUPS) + 1)
FLOW_GROUPS = ("promise", "dispute+paid", "hardship+complaint", "other parties")
Z = NormalDist()
# ---- Part 1: pass^k and pass@k ----------------------------------------------
def pass_hat_k(c: int, n: int, k: int) -> float:
"""Unbiased per-task pass^k from c successes in n trials (tau-bench): C(c,k)/C(n,k)."""
if k > n:
raise ValueError("pass^k needs at least k trials per task")
return comb(c, k) / comb(n, k)
def pass_at_k(c: int, n: int, k: int) -> float:
"""Unbiased per-task pass@k (Codex paper): 1 - C(n-c,k)/C(n,k)."""
if k > n:
raise ValueError("pass@k needs at least k trials per task")
return 1.0 - comb(n - c, k) / comb(n, k)
# ---- Part 2: suite estimators ------------------------------------------------
def suite_pass_hat_k(counts: list[int], n: int, k: int) -> float:
return sum(pass_hat_k(c, n, k) for c in counts) / len(counts)
def suite_pass_at_k(counts: list[int], n: int, k: int) -> float:
return sum(pass_at_k(c, n, k) for c in counts) / len(counts)
def plugin_per_task(counts: list[int], n: int, k: int) -> float:
"""Shortcut 1: average of (c/n)^k. Biased upward for every task with 0 < c < n."""
return sum((c / n) ** k for c in counts) / len(counts)
def plugin_suite(counts: list[int], n: int, k: int) -> float:
"""Shortcut 2: (mean pass rate)^k. Ignores that hard tasks drag pass^k down."""
return (sum(counts) / (n * len(counts))) ** k
# ---- Part 3: intervals for single rates --------------------------------------
def wilson(k: int, n: int, conf: float = 0.95) -> tuple[float, float]:
z = Z.inv_cdf(1 - (1 - conf) / 2)
p = k / n
d = 1 + z * z / n
centre = (p + z * z / (2 * n)) / d
half = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
return max(0.0, centre - half), min(1.0, centre + half)
def binom_cdf(k: int, n: int, p: float) -> float:
return sum(comb(n, i) * p ** i * (1 - p) ** (n - i) for i in range(k + 1))
def clopper_pearson(k: int, n: int, conf: float = 0.95) -> tuple[float, float]:
"""Exact two-sided interval by bisection on the binomial CDF."""
a = (1 - conf) / 2
def bisect(f, lo=0.0, hi=1.0):
for _ in range(100):
mid = (lo + hi) / 2
lo, hi = (mid, hi) if f(mid) else (lo, mid)
return (lo + hi) / 2
lower = 0.0 if k == 0 else bisect(lambda p: 1 - binom_cdf(k - 1, n, p) < a)
upper = 1.0 if k == n else bisect(lambda p: binom_cdf(k, n, p) > a)
return lower, upper
def all_pass_needed(bar: float, conf: float = 0.95) -> int:
"""Smallest n where n of n passing puts the Wilson lower bound at or above `bar`."""
n = 1
while wilson(n, n, conf)[0] < bar:
n += 1
return n
# ---- Part 4: run counts --------------------------------------------------------
def n_unpaired(p1: float, p2: float, alpha: float = 0.05, power: float = 0.80) -> float:
"""Per-arm n for a one-sided two-proportion z test (pooled variance under H0)."""
za, zb = Z.inv_cdf(1 - alpha), Z.inv_cdf(power)
pb = (p1 + p2) / 2
num = za * sqrt(2 * pb * (1 - pb)) + zb * sqrt(p1 * (1 - p1) + p2 * (1 - p2))
return num ** 2 / (p1 - p2) ** 2
def n_paired(p10: float, p01: float, alpha: float = 0.05, power: float = 0.80) -> float:
"""Pairs for a one-sided McNemar test (normal approximation).
p10: old passes, new fails. p01: old fails, new passes.
"""
za, zb = Z.inv_cdf(1 - alpha), Z.inv_cdf(power)
pd, d = p10 + p01, p10 - p01
return (za * sqrt(pd) + zb * sqrt(pd - d * d)) ** 2 / d ** 2
def sim_power_unpaired(n: int, p1: float, p2: float, reps: int, rng: random.Random,
alpha: float = 0.05) -> float:
za = Z.inv_cdf(1 - alpha)
hits = 0
for _ in range(reps):
x1 = sum(rng.random() < p1 for _ in range(n))
x2 = sum(rng.random() < p2 for _ in range(n))
pb = (x1 + x2) / (2 * n)
se = sqrt(2 * pb * (1 - pb) / n)
hits += se > 0 and (x1 - x2) / n / se > za
return hits / reps
def sim_power_paired(n: int, cells: dict[str, float], reps: int, rng: random.Random,
alpha: float = 0.05) -> float:
"""Each pair lands in one of four cells; McNemar z on the discordant counts."""
za = Z.inv_cdf(1 - alpha)
edges, acc = [], 0.0
for name in ("10", "01", "11", "00"):
acc += cells[name]
edges.append((acc, name))
hits = 0
for _ in range(reps):
b = c = 0
for _ in range(n):
u = rng.random()
cell = next(name for edge, name in edges if u < edge)
b += cell == "10"
c += cell == "01"
hits += b + c > 0 and (b - c) / sqrt(b + c) > za
return hits / reps
# ---- Part 5: segment gate --------------------------------------------------------
def holm(pvals: list[float]) -> list[float]:
order = sorted(range(len(pvals)), key=lambda i: pvals[i])
m, adj, running = len(pvals), [0.0] * len(pvals), 0.0
for rank, i in enumerate(order):
running = max(running, min(1.0, (m - rank) * pvals[i]))
adj[i] = running
return adj
def paired_bootstrap(deltas: list[float], margin: float, reps: int, rng: random.Random,
conf: float = 0.95) -> tuple[float, float]:
"""One-sided upper bound on the mean paired delta, and the bootstrap p-value for
H0: delta >= -margin (no regression beyond the allowed drop)."""
n = len(deltas)
means = sorted(sum(deltas[rng.randrange(n)] for _ in range(n)) / n for _ in range(reps))
upper = means[min(reps - 1, int(conf * reps))]
p = (sum(m >= -margin for m in means) + 1) / (reps + 1)
return upper, p
def main() -> None:
rng = random.Random(STATS_SEED)
print(WORLD["fiction_note"])
print(f"dataset: {WORLD['dataset']}")
if WORLD["seed"] != STATS_SEED:
print(f"statistics stream: seed {STATS_SEED}, shared by both editions so computed lines match")
# Part 1 ---------------------------------------------------------------
print("\n[1] pass^k for one task with a fixed per-trial rate p: p^k")
print("p k=1 k=4 k=8 k=16")
for p in (0.95, 0.90, 0.80):
print(f"{p:.2f} " + " ".join(f"{p ** k:.4f}" for k in (1, 4, 8, 16)))
assert round(0.8 ** 8, 4) == 0.1678
assert round(0.9 ** 8, 4) == 0.4305
assert [round(0.9 ** k * 100) for k in (4, 8, 16)] == [66, 43, 19]
u, s = pass_hat_k(6, 8, 4), (6 / 8) ** 4
print(f"c=6 of n=8, k=4: unbiased C(6,4)/C(8,4) = {comb(6, 4)}/{comb(8, 4)} = {u:.4f}; "
f"shortcut (6/8)^4 = {s:.4f}; pass@4 = {pass_at_k(6, 8, 4):.4f}")
assert round(u, 4) == 0.2143 and round(s, 4) == 0.3164
# Part 2 ---------------------------------------------------------------
n_trials = 8
scen = [] # (flow group, true per-trial rate)
for p in (0.85, 0.80, 0.75, 0.80): # dispute segment: 4 scenarios, mean 0.80
scen.append(("dispute+paid", p))
for _ in range(56):
hard = rng.random() < 0.2
p = rng.betavariate(6, 2) if hard else rng.betavariate(40, 1)
scen.append(("other", p))
trials = [[rng.random() < p for _ in range(n_trials)] for _, p in scen]
counts = [sum(t) for t in trials]
print("\n[2] Suite of 60 scenarios x 8 trials (synthetic; dispute segment = 4 scenarios)")
# The v3.2 release run: one trial per scenario, a separate seeded draw chosen to
# be green, as it was in the incident. Print how likely a green run was.
rel_rng = random.Random(STATS_SEED + 6)
release = [rel_rng.random() < p for _, p in scen]
disp_release = release[:4]
p_green = math.prod(p for _, p in scen)
p_disp = math.prod(p for _, p in scen[:4])
print(f"release run (1 trial per scenario): {sum(release)}/60 passed; "
f"dispute {sum(disp_release)}/4")
print(f"chance of that run being all green: {p_green:.3f}; dispute alone 4/4: {p_disp:.3f}")
print("k unbiased pass^k per-task (c/n)^k (mean p)^k pass@k")
rows = {}
for k in (1, 2, 4, 8):
rows[k] = (suite_pass_hat_k(counts, n_trials, k), plugin_per_task(counts, n_trials, k),
plugin_suite(counts, n_trials, k), suite_pass_at_k(counts, n_trials, k))
a, b, c, d = rows[k]
print(f"{k:<3} {a:>15.3f} {b:>16.3f} {c:>10.3f} {d:>6.3f}")
for k in (2, 4, 8):
assert rows[k][1] >= rows[k][0], "per-task plug-in must not fall below unbiased"
assert rows[8][3] > rows[1][0] > rows[8][0]
true8 = sum(p ** 8 for _, p in scen) / len(scen)
print(f"true suite pass^8 from the generating rates: {true8:.3f}")
# Frozen numbers the chapter and this page quote (identical in both editions)
assert f"{p_green:.3f}/{p_disp:.3f}/{true8:.3f}" == "0.037/0.408/0.751"
assert [f"{v:.3f}" for v in rows[8]] == ["0.800", "0.857", "0.776", "1.000"]
assert [f"{v:.3f}" for v in rows[4][:3]] == ["0.886", "0.904", "0.881"]
# Part 3 ---------------------------------------------------------------
print("\n[3] Intervals for single rates (95%)")
disp_counts = counts[:4]
k_d, n_d = sum(disp_counts), 4 * n_trials
lines = [("release run, all 60", sum(release), 60),
("release run, dispute", sum(disp_release), 4),
("8 trials, dispute (32 runs, 4 tasks)", k_d, n_d)]
print("rate k/n Wilson Clopper-Pearson")
for name, k, n in lines:
w, cp = wilson(k, n), clopper_pearson(k, n)
print(f"{name:<38} {k:>2}/{n:<4} [{w[0]:.3f}, {w[1]:.3f}] [{cp[0]:.3f}, {cp[1]:.3f}]")
lo4 = wilson(4, 4)[0]
need = all_pass_needed(0.90)
print(f"4 of 4 passing: Wilson lower bound {lo4:.3f}; to clear a 0.90 bar with zero "
f"failures you need {need} of {need}")
print("the 32 dispute runs come from 4 scenarios, so they are not 32 independent "
"trials: the scenario is the unit for any gate")
assert sum(disp_release) == 4, "the seeded release run must show a green dispute segment"
assert round(lo4, 3) == 0.510 and need == 35
# Part 4 ---------------------------------------------------------------
print("\n[4] Runs needed to detect 90% -> 85% (one-sided alpha 0.05, power 0.80)")
nu = n_unpaired(0.90, 0.85)
cells = {"10": 0.07, "01": 0.02, "11": 0.83, "00": 0.08}
npd = n_paired(cells["10"], cells["01"])
print(f"unpaired, closed form: {nu:.1f} -> {math.ceil(nu)} scenarios per arm "
f"({2 * math.ceil(nu):,} scenario runs)")
print(f"paired (McNemar), discordant cells 7% / 2%: {npd:.1f} -> {math.ceil(npd)} scenarios, "
f"each run on both versions")
assert math.ceil(nu) == 540 and round(npd, -1) == 220
sim_rng = random.Random(STATS_SEED + 4)
pw_u = sim_power_unpaired(math.ceil(nu), 0.90, 0.85, 2000, sim_rng)
pw_p = sim_power_paired(math.ceil(npd), cells, 2000, sim_rng)
pw_p60 = sim_power_paired(60, cells, 2000, sim_rng)
print(f"seeded simulation, 2,000 reps: unpaired power {pw_u:.3f}; paired power {pw_p:.3f}; "
f"paired at 60 scenarios {pw_p60:.3f}")
assert 0.76 < pw_u < 0.84 and 0.76 < pw_p < 0.84 and pw_p60 < 0.45
assert f"{pw_u:.3f}/{pw_p:.3f}/{pw_p60:.3f}" == "0.791/0.804/0.358"
# Part 5 ---------------------------------------------------------------
print("\n[5] Release gate: 240 scenarios, 20 per gated segment, k=4, old vs new")
k_n, per_seg, margin = WORLD["nightly"]["k"], WORLD["nightly"]["per_segment"], 0.05
regress = (LANG_GROUPS[1], "dispute+paid") # seeded regression, second language group
seg_rows = []
gate_rng = random.Random(STATS_SEED + 5)
for lang in LANG_GROUPS:
for flow in FLOW_GROUPS:
deltas, new_counts, old_sum, new_sum = [], [], 0, 0
for _ in range(per_seg):
p_old = gate_rng.betavariate(18, 2)
p_new = p_old - 0.30 if (lang, flow) == regress else p_old
old = sum(gate_rng.random() < p_old for _ in range(k_n))
new = sum(gate_rng.random() < max(0.0, p_new) for _ in range(k_n))
deltas.append((new - old) / k_n)
new_counts.append(new)
old_sum, new_sum = old_sum + old, new_sum + new
upper, p = paired_bootstrap(deltas, margin, 2000, gate_rng)
seg_rows.append([lang, flow, old_sum / (per_seg * k_n), new_sum / (per_seg * k_n),
sum(deltas) / per_seg, upper, p,
suite_pass_hat_k(new_counts, k_n, k_n)])
adj = holm([r[6] for r in seg_rows])
print(f"allowed drop {margin:.2f}; block when the Holm-adjusted p for "
f"'delta below -{margin:.2f}' is under 0.05")
print(f"{'language':<{LANG_W + 1}}flow old new delta upper95 p "
"p_holm new pass^4 gate")
blocked = []
for r, pa in zip(seg_rows, adj):
verdict = "BLOCK" if pa < 0.05 else "pass"
if verdict == "BLOCK":
blocked.append((r[0], r[1]))
print(f"{r[0]:<{LANG_W}} {r[1]:<18} {r[2]:.3f} {r[3]:.3f} {r[4]:+.3f} {r[5]:+.3f} "
f"{r[6]:.4f} {pa:.4f} {r[7]:.3f} {verdict}")
print(f"blocked segments: {', '.join(' x '.join(b) for b in blocked)}")
assert blocked == [regress], blocked
pooled = sum(r[4] for r in seg_rows) / len(seg_rows)
hit = seg_rows[LANG_GROUPS.index(regress[0]) * len(FLOW_GROUPS) + 1]
assert [f"{v:.3f}" for v in hit[2:6]] == ["0.912", "0.613", "-0.300", "-0.163"]
assert f"{hit[6]:.4f}/{adj[seg_rows.index(hit)]:.4f}/{hit[7]:.3f}" == "0.0005/0.0060/0.300"
print(f"pooled delta over all 240 scenarios: {pooled:+.3f} (the regression is diluted "
f"12 ways; a pooled gate alone would read it as a small dip)")
assert f"{pooled:+.3f}" == "-0.022"
print("\nall assertions passed")
if __name__ == "__main__":
main()
Step 3: run it
.venv-r9/bin/python voice-pass-k-planner.py
Running it with Python 3.11 prints:
Synthetic data.
dataset: steelyard-calls (synthetic, seed 20261121)
statistics stream: seed 20261120, shared by both editions so computed lines match
[1] pass^k for one task with a fixed per-trial rate p: p^k
p k=1 k=4 k=8 k=16
0.95 0.9500 0.8145 0.6634 0.4401
0.90 0.9000 0.6561 0.4305 0.1853
0.80 0.8000 0.4096 0.1678 0.0281
c=6 of n=8, k=4: unbiased C(6,4)/C(8,4) = 15/70 = 0.2143; shortcut (6/8)^4 = 0.3164; pass@4 = 1.0000
[2] Suite of 60 scenarios x 8 trials (synthetic; dispute segment = 4 scenarios)
release run (1 trial per scenario): 60/60 passed; dispute 4/4
chance of that run being all green: 0.037; dispute alone 4/4: 0.408
k unbiased pass^k per-task (c/n)^k (mean p)^k pass@k
1 0.969 0.969 0.969 0.969
2 0.939 0.943 0.938 0.998
4 0.886 0.904 0.881 1.000
8 0.800 0.857 0.776 1.000
true suite pass^8 from the generating rates: 0.751
[3] Intervals for single rates (95%)
rate k/n Wilson Clopper-Pearson
release run, all 60 60/60 [0.940, 1.000] [0.940, 1.000]
release run, dispute 4/4 [0.510, 1.000] [0.398, 1.000]
8 trials, dispute (32 runs, 4 tasks) 28/32 [0.719, 0.950] [0.710, 0.965]
4 of 4 passing: Wilson lower bound 0.510; to clear a 0.90 bar with zero failures you need 35 of 35
the 32 dispute runs come from 4 scenarios, so they are not 32 independent trials: the scenario is the unit for any gate
[4] Runs needed to detect 90% -> 85% (one-sided alpha 0.05, power 0.80)
unpaired, closed form: 539.9 -> 540 scenarios per arm (1,080 scenario runs)
paired (McNemar), discordant cells 7% / 2%: 220.5 -> 221 scenarios, each run on both versions
seeded simulation, 2,000 reps: unpaired power 0.791; paired power 0.804; paired at 60 scenarios 0.358
[5] Release gate: 240 scenarios, 20 per gated segment, k=4, old vs new
allowed drop 0.05; block when the Holm-adjusted p for 'delta below -0.05' is under 0.05
language flow old new delta upper95 p p_holm new pass^4 gate
English promise 0.925 0.938 +0.013 +0.075 0.9675 1.0000 0.750 pass
English dispute+paid 0.925 0.863 -0.062 +0.013 0.4488 1.0000 0.550 pass
English hardship+complaint 0.963 0.963 +0.000 +0.050 0.9830 1.0000 0.850 pass
English other parties 0.887 0.925 +0.037 +0.125 0.9580 1.0000 0.750 pass
Spanish-dominant promise 0.925 0.925 +0.000 +0.087 0.8521 1.0000 0.800 pass
Spanish-dominant dispute+paid 0.912 0.613 -0.300 -0.163 0.0005 0.0060 0.300 BLOCK
Spanish-dominant hardship+complaint 0.800 0.950 +0.150 +0.237 1.0000 1.0000 0.800 pass
Spanish-dominant other parties 0.950 0.925 -0.025 +0.037 0.8081 1.0000 0.700 pass
Spanglish promise 0.912 0.900 -0.013 +0.062 0.8586 1.0000 0.600 pass
Spanglish dispute+paid 0.925 0.863 -0.062 +0.013 0.4433 1.0000 0.550 pass
Spanglish hardship+complaint 0.900 0.938 +0.037 +0.100 0.9940 1.0000 0.750 pass
Spanglish other parties 0.863 0.825 -0.037 +0.062 0.6057 1.0000 0.450 pass
blocked segments: Spanish-dominant x dispute+paid
pooled delta over all 240 scenarios: -0.022 (the regression is diluted 12 ways; a pooled gate alone would read it as a small dip)
all assertions passed
The run takes about a second; the two power simulations are most of it. The script asserts on every fixed number before it prints "all assertions passed": 0.1678 and 0.4305, the 0.2143 vs 0.3164 pair, the suite estimates, 540 per arm, about 220 pairs, the power figures, the blocked segment's row and the pooled delta. A changed estimator that moves any of them fails loudly.
Step 4: read the pass^k table
Block [1] is the reason one green run proves little. At 90% per trial, pass^k is 0.6561 at k=4, 0.4305 at k=8 and 0.1853 at k=16, the same 66/43/19% as the pass^k chapter. At 80%, eight in a row happen 16.78% of the time.
Choose k from the deployment, not from the benchmark. Steelyard collects its own debts in its own name, so the FDCPA and Regulation F largely do not bind it 2 3; it adopts their conduct rules as company policy, and state law, the TCPA and the CFPB's ban on unfair, deceptive or abusive practices still apply. One adopted rule sets k: under Regulation F, a telephone conversation with a person about a debt starts seven consecutive days with no further calls to that person about it 4 (Steelyard's check C11). A borrower who stays in the 1-30 days-past-due bucket can therefore talk to Hazel about four times (30 / 7 = 4.3; five at most, on days 1, 8, 15, 22 and 29), and at 80% per call meets at least one failed call 1 - 0.8^4 = 59.0% of the time. That is the number a flow owner should see, not 80%, and it is why the nightly run uses k=4.
The second line is the estimator. With 6 successes in 8 trials, the unbiased pass^4 is C(6,4)/C(8,4) = 15/70 = 0.2143. The shortcut (6/8)^4 = 0.3164 overstates it by half. The unbiased form counts how many 4-trial subsets of the 8 are all successes; the shortcut pretends the observed 0.75 is the true rate. pass@4 is 1.0000 here because only 2 trials failed, so every 4-subset contains a success. Report pass@k for exploration tasks where one success is enough. A collections call is not one of them.
Step 5: read the suite estimators
Block [2] runs the same comparison over 60 scenarios. At k=8 the unbiased suite pass^8 is 0.800, the per-task shortcut gives 0.857, and (mean p)^8 gives 0.776. The generating rates put the true value at 0.751.
The two shortcuts fail in opposite directions, for different reasons:
- Per-task (c/n)^k is biased upward. For k >= 2 and any task with 0 < c < n, (c/n)^k exceeds C(c,k)/C(n,k), so the average does too. The Codex paper showed the mirror problem for pass@k: the plug-in 1 - (1 - p_hat)^k gives a consistent underestimate, and the paper uses an unbiased per-task estimator instead 5. That pass^k is biased the other way is reasoning from the same algebra, not the paper's finding; the 0.857 vs 0.800 row shows it on this suite.
- (mean p)^k ignores the hard tail. pass^k of a suite is the average of each task's p^k, which is at least (mean p)^k. The generating rates average 0.950, and 0.950^8 = 0.664 against a true 0.751. The 0.776 lands close only by accident: these trials came out luckier than their rates (an observed pass rate of 0.969), which lifts the shortcut and the unbiased estimate alike. Do not count on that.
Two more things in block [2]. pass@8 is 1.000: every scenario succeeded at least once in 8 trials, which says nothing about reliability. And the release run is 60 of 60 because the script picks a release seed that comes out green, as the real one did. Under these rates a fully green run had a 3.7% chance, but the dispute segment alone passes 4 of 4 with probability 0.85 x 0.80 x 0.75 x 0.80 = 0.408. A segment that small passes a clean run 2 times in 5 while failing 1 call in 5.
Unbiased pass^k needs at least k trials of every scenario; the functions raise an error otherwise. Trials vary even when nothing changes. Atil et al. found min-to-max accuracy gaps of up to 15% across 10 runs of five models at temperature 0, top-p 1 and a fixed seed 6. Thinking Machines Lab got 80 distinct completions from 1000 temperature-0 runs of one prompt on Qwen3-235B in vLLM, and attributes this mainly to inference kernels that are not batch-invariant, so results depend on server load 7. A voice stack adds its own variance on top: STT on the simulated caller's audio, VAD timing, TTS and the caller simulator.
Step 6: read the intervals
Block [3] puts intervals on single rates 8. The dispute segment's release result, 4 of 4, has a Wilson interval of [0.510, 1.000] and an exact Clopper-Pearson interval of [0.398, 1.000]. Wilson admits a segment that fails nearly half its calls, and Clopper-Pearson one that fails 6 in 10. That is too wide to gate on: every bar a compliance owner would accept sits above 0.510.
To clear a 0.90 bar on the Wilson lower bound with zero failures, the script finds you need 35 of 35. With a failure or two you need more. A segment with 4 scenarios cannot produce that evidence however many times you rerun it, which leads to the third line.
Eight trials of the 4 dispute scenarios give 28 of 32 and a Wilson interval of [0.719, 0.950]. That interval overstates what you know, and the script says why: the 32 runs come from 4 scenarios, and trials of the same scenario share its difficulty. Wilson assumes 32 independent draws. The scenario is the unit, so more trials narrow pass^k for those 4 scenarios but do not tell you about the dispute calls they do not cover. Use Wilson and Clopper-Pearson for one binary rate over independent units, never for clustered trials, WER, entity error rate or a difference of two rates. The confidence intervals chapter covers the alternatives.
Step 7: read the run counts
Block [4] answers "how many scenarios?" once you fix the minimum drop you must detect. For 90% -> 85% at 80% power and one-sided alpha 0.05 (z values 1.645 and 0.842):
- Unpaired: n = [1.645 x sqrt(2 x 0.875 x 0.125) + 0.842 x sqrt(0.90 x 0.10 + 0.85 x 0.15)]^2 / 0.05^2 = 539.9, so 540 scenarios per arm, 1,080 scenario runs.
- Paired (McNemar): run the same scenarios on both versions and only the discordant pairs carry signal 9. With old-passes-new-fails on 7% of scenarios and the reverse on 2% (both pass 83%, both fail 8%, so the old rate is 0.90 and the new 0.85), n = [1.645 x sqrt(0.09) + 0.842 x sqrt(0.09 - 0.05^2)]^2 / 0.05^2 = 220.5, so about 220 scenarios, 221 when rounded up.
The seeded simulation agrees: power 0.791 unpaired at 540 per arm and 0.804 paired at 221. At 60 scenarios, the size of the v3.2 release run, paired power is 0.358. That run would have missed a 5-point drop nearly two times in three even if it had compared versions.
The paired size depends on the discordant cells, and those are an assumption until you measure them. Run your current suite twice on the same version and count how often a scenario flips; that rate sets the floor of your discordant cells. If flips are common, the paired advantage shrinks. The statistical power chapter has the tables.
Step 8: read the segment gate
Block [5] is the gate Steelyard runs nightly. Each segment has 20 scenarios run 4 times on the old version and 4 times on the new. The paired delta per scenario is (new successes - old successes) / 4, the bootstrap resamples scenarios within the segment, and the test is one-sided against an allowed drop of 0.05: the p-value is the share of bootstrap means at or above -0.05. Holm then adjusts the 12 p-values 10. tau-Voice used the same family of tools: two-sided paired permutation tests rather than trial-level confidence intervals, pairing by task ID, with Holm-Bonferroni correction within each provider 11.
The seeded regression, Spanish-dominant x dispute and already paid, blocks: delta -0.300, one-sided upper bound -0.163, raw p 0.0005, Holm-adjusted 12 x 0.0005 = 0.0060. The other 11 pass. New pass^4 in that segment is 0.300: only 6 of its 20 scenarios passed all 4 trials.
Now read the two rows that pass at delta -0.062: English and Spanglish, dispute and already paid, both with an upper bound of +0.013. Each could be noise or a real 6-point drop; 20 scenarios cannot tell them apart. This gate blocks large regressions in one segment and lets small ones through. That is a deliberate choice, because a gate that blocks whenever it cannot rule out a drop would block nearly every night at this size. Cover the gap two ways: track the pooled delta (here -0.022 over 240 scenarios) against the 220-pair size from Step 7, and watch any segment that dips two nights in a row.
The pooled number also shows why the gate is per segment. A 30-point regression in one segment shows up as a 2-point dip overall, which a single suite-level gate would wave through. DPD bucket, carrier, hour, state and code-switch density stay out of the gate: they are monitored slices, reported on every release. Gating on them too would multiply the comparisons Holm has to pay for.
Run the full file with --edition india for the India edition's lender and segment labels; the India edition of the statistics chapter explains that output. Every computed line comes out the same in both editions; only the opening lines (the lender, the dataset name and the seed note, which the India edition omits because its dataset seed is the statistics seed) and the language-group labels differ.
What to do next
- Replace the synthetic suite with your own results: a list of per-scenario success counts out of n trials, with n at least the k you report. Feed it to
suite_pass_hat_k and report pass^1 beside pass^k, never pass^k alone.
- Measure your discordant cells by running the current release twice on the same version, then rerun
n_paired with them before you set the nightly scenario count.
- Put the gate's inputs in the release report: scenario count per segment, k, allowed drop, the gold set and judge versions, the per-segment delta with its bound, and the Holm-adjusted p. The bootstrap and paired tests chapter and the multiple comparisons chapter cover the choices behind each column.
- Score each scenario's outcome against the CRM end state with the task-success recipe, so a trial "pass" means the database diff matched, not that the call ended politely.