On 18 November Derek Wallace, an angry small-business borrower in Houston, told Hazel, Steelyard's collections agent, that he was filing a complaint with the CFPB, and Hazel answered that Steelyard might "have to send someone out to your house" (T09). A prompt patch had been live for three days, and about 5 calls a day carried a threat like that one. The judge read a uniform 2% of calls, so on average about 0.11 of those calls reached it each day. This recipe computes the chance that sample saw anything (about 28%), shows the stratified plan that replaced it, and gives you the four formulas you need to size your own.
The script and synthetic data for this recipe are also in the ai-evals-course repository on GitHub, in the folder voice-ai/labs/rare-violation-sampling/. That copy also carries the India edition's block, which this page leaves out.
What you build
One standard-library script that answers four sizing questions for a violation that is rare overall but concentrated on a few call paths. It copies the shared Steelyard constants block, the world rates for T09 and the judge prices from the world's invented cost model, and generates one seeded synthetic week to test the estimators against a known truth.
| Function | What it does |
|---|
p_detect(n, rate), calls_needed(rate, target) | The chance of seeing at least one violation in n calls, and the smallest n that reaches a target chance |
rule_of_three(n), zero_upper_one_sided(n), cp_upper(k, n, tail) | Upper bounds on the rate: the 3/n shortcut, the exact one-sided bound for zero events, and Clopper-Pearson for any count |
strata_per_day, threat_reach_prob, mc_reach | The T09 replay: stratum sizes from the world rates, the chance the judge sees a threat under each policy, and a Monte Carlo check |
allocate(N, S, n, method), stratified_se | Proportional, equal and Neyman allocation of a fixed audit budget, and the standard error each gives for the overall rate |
simulate_week | One synthetic week under the stratified plan, to show the naive rate over judged calls against the stratum-weighted rate |
Step 1: Create a fresh environment
python3.11 -m venv .venv
.venv/bin/python --version
There is nothing to install. The script uses argparse, math and random only and makes no network calls.
Step 2: Save the script
Save it as rare-violation-sampling.py. The world rates near the top are the T09 rates from the incident log: the angry path is 1 in 600 connected calls, and while collections-v17-p1 was live, 1 in 40 angry calls produced a threat. The three per-stratum rates in ASSUMED_RATE drive steps 7 and 8 only (sections 4 and 5 of the output). They are planning assumptions written into the script so the estimators have a truth to hit, not Steelyard measurements.
The script also takes --edition global|india (default global). The edition supplies the lender, the seed, the digit grouping and the judge prices; every rate, volume and formula is shared, so the sizing numbers come out the same in both. The listing below is the global edition: it leaves out the India edition's constants block, which sits between two marker comments in the full file. Without that block, --edition india exits with a usage error.
"""Size production samples for rare policy violations, and stratify toward them.
Standard library only. Python 3.11. No network. Seeded synthetic data.
Dataset: steelyard-calls (synthetic, seed 20261121).
--edition india switches to the India edition's fictional lender and data set
(its constants block sits between the INDIA EDITION markers below).
Run:
python3.11 rare-violation-sampling.py # global edition
python3.11 rare-violation-sampling.py --edition india # India edition
"""
from __future__ import annotations
import argparse
import math
import random
def western(n: float, decimals: int = 0) -> str:
"""Western digit grouping: 130000 -> 130,000."""
return f"{n:,.{decimals}f}"
def usd(x: float) -> str:
return f"${x:,.2f}"
# ---- GLOBAL constants (copied from WORLD-GLOBAL.md WG1-WG7; do not edit here) -----
GLOBAL = {
"lender": "Steelyard Credit, Inc. (US lender, head office Tucson AZ)",
"lender_note": "Synthetic data.",
"dataset": "steelyard-calls (synthetic, seed 20261121)",
"seed": 20261121,
"products": { # code: (ticket $ range, payment $ range)
"AUTO": ((8_000, 35_000), (245, 680)),
"PIL": ((1_000, 40_000), (95, 1_850)),
"POS": ((300, 6_000), (45, 520)),
"SBL": ((25_000, 250_000), (980, 7_200)),
"SDL": ((300, 3_000), (35, 140)), # biweekly
},
"campaigns": {"AUTO-early-nov": "AUTO", "PIL-digital-nov": "PIL", "POS-short-nov": "POS",
"SBL-guar-nov": "SBL", "SDL-biwk-nov": "SDL"},
"windows": {"default": ("08:00", "21:00"), "SDL": ("09:00", "20:00")}, # borrower's local time
"languages": {"English": 64, "Spanish-dominant": 18, "Spanglish": 14, "other": 4},
"bot_languages": ("English", "Spanish-dominant", "Spanglish"), # other -> human
"customer_advocate": {"name": "Ms Dana Brooks", "email": "advocate@steelyard.example",
"phone": "(520) 555-0199",
"address": "1200 Example Avenue, Tucson, AZ 85701"},
"money": "integer cents in tools and CRM; dollars in speech",
"dates": "ISO in tools; words in speech; en-US month-first, es-US day-first",
"versions": {"stt": "vendorA-2026-10", "taxonomy": "gl-fail-v4", "gold": "gold-v7"},
"connects_per_day": 130_000, "calling_days_per_month": 26, "calling_days_per_week": 6,
}
# Edition settings for this recipe: judge prices (WG20, $ per judged call per dimension),
# number and money formats, and the frozen monthly judge line (WG20).
GLOBAL.update({
"price": {"J1": 0.0075, "J3": 0.0075, "J2": 0.02},
"fmt": western, "fmt_money": usd, "money_col": "J1 $/week",
"month_cost": (2, 7003.36), # (rounding digits, value): $7,003.36 a month
"plan": "WG15", # world-bible section of the stratified judge plan
})
EDITIONS = {"global": GLOBAL}
# (left out of this listing: the India edition's constants block, which adds
# EDITIONS["india"]; it is in the full file in the course repository)
WORLD: dict = GLOBAL # set from --edition in main()
# World rates (WG15, WG16, WG17, WG20; identical to W15-W17 in the India edition).
# All [W] or [C]; none is an industry rate. Set in main() from the edition block.
CONNECTS = DAYS_WEEK = DAYS_MONTH = 0
ANGRY_SHARE = 1 / 600 # angry path, share of connected calls (T09)
THREAT_IN_ANGRY = 1 / 40 # threat rate on the angry path while collections-v17-p1 was live
PATCH_DAYS = 3 # 2026-11-16 to 2026-11-18
OLD_UNIFORM = 0.02 # judge sample before T09: 2% of all calls, uniform
RISK_PER_DAY = 5_200 # risk strata judged 100% from 2026-11-19 (about 4% of connects)
REST_FRACTION = 0.02 # plus 2% of the rest
J1_RECALL = 15 / 18 # J1 v1 validation recall (advisory judge; same in both editions)
AUDITS_PER_WEEK = 400
# Planning assumptions for the allocation and reweighting steps. These are NOT
# measurements of the fictional lender; they exist so the estimators have a known truth.
ASSUMED_RATE = {"angry": THREAT_IN_ANGRY, "other_risk": 0.004, "rest": 0.0005}
# ---- step 1: detection probability and calls needed ---------------------------
def p_detect(n: float, rate: float) -> float:
"""P(at least one violation among n independent calls at this rate)."""
return 1.0 - (1.0 - rate) ** n
def calls_needed(rate: float, target: float = 0.95) -> int:
"""Smallest n with p_detect(n, rate) >= target (the log formula, then checked)."""
n = math.ceil(math.log(1.0 - target) / math.log(1.0 - rate))
while p_detect(n - 1, rate) >= target:
n -= 1
while p_detect(n, rate) < target:
n += 1
return n
# ---- step 2: zero-event and exact bounds ----------------------------------------
def rule_of_three(n: int) -> float:
return 3.0 / n
def zero_upper_one_sided(n: int, alpha: float = 0.05) -> float:
"""Exact one-sided upper bound when 0 of n: solve (1 - p)^n = alpha."""
return 1.0 - alpha ** (1.0 / n)
def binom_cdf(k: int, n: int, p: float) -> float:
if p <= 0.0:
return 1.0
if p >= 1.0:
return 0.0 if k < n else 1.0
lp, lq = math.log(p), math.log1p(-p)
return sum(math.exp(math.lgamma(n + 1) - math.lgamma(i + 1) - math.lgamma(n - i + 1)
+ i * lp + (n - i) * lq) for i in range(k + 1))
def cp_upper(k: int, n: int, tail: float) -> float:
"""Clopper-Pearson upper limit: the p where P(X <= k | p) = tail (bisection)."""
if k >= n:
return 1.0
lo, hi = 0.0, 1.0
for _ in range(200):
mid = (lo + hi) / 2
if binom_cdf(k, n, mid) > tail:
lo = mid
else:
hi = mid
return (lo + hi) / 2
# ---- step 3: sampling policies over strata ---------------------------------------
def strata_per_day() -> dict[str, float]:
angry = CONNECTS * ANGRY_SHARE
return {"angry": angry, "other_risk": RISK_PER_DAY - angry, "rest": CONNECTS - RISK_PER_DAY}
def judged_per_day(fractions: dict[str, float]) -> float:
return sum(strata_per_day()[h] * f for h, f in fractions.items())
def threat_reach_prob(fractions: dict[str, float], days: float, recall: float = 1.0) -> float:
"""P(the judge sees at least one T09 threat), threats living only on the angry path."""
n_angry_judged = strata_per_day()["angry"] * fractions["angry"] * days
return p_detect(n_angry_judged, THREAT_IN_ANGRY * recall)
def mc_reach(fraction: float, days: int, trials: int, rng: random.Random) -> float:
"""Monte Carlo check: Poisson threats per day, each judged with probability `fraction`."""
lam = CONNECTS * ANGRY_SHARE * THREAT_IN_ANGRY
hits = 0
for _ in range(trials):
seen = False
for _ in range(days):
# Poisson by inversion (stdlib has no Poisson sampler)
u, k, p = rng.random(), 0, math.exp(-lam)
cdf = p
while u > cdf:
k += 1
p *= lam / k
cdf += p
if any(rng.random() < fraction for _ in range(k)):
seen = True
hits += seen
return hits / trials
# ---- step 4: allocation of a fixed audit budget --------------------------------------
def allocate(N: dict[str, int], S: dict[str, float], n: int, method: str) -> dict[str, float]:
keys = list(N)
if method == "proportional":
w = {h: N[h] for h in keys}
elif method == "equal":
w = {h: 1.0 for h in keys}
elif method == "neyman":
w = {h: N[h] * S[h] for h in keys}
else:
raise ValueError(method)
alloc, free, budget = {}, set(keys), float(n)
while True: # cap at the stratum size and give the excess to the others
tot = sum(w[h] for h in free)
trial = {h: budget * w[h] / tot for h in free}
over = [h for h in free if trial[h] > N[h]]
if not over:
alloc.update(trial)
return alloc
for h in over:
alloc[h] = float(N[h])
budget -= N[h]
free.discard(h)
def stratified_se(N: dict[str, int], p: dict[str, float], n: dict[str, float]) -> float:
total = sum(N.values())
var = 0.0
for h in N:
if n[h] <= 0:
return float("inf")
fpc = 1.0 - n[h] / N[h]
var += (N[h] / total) ** 2 * p[h] * (1 - p[h]) / n[h] * fpc
return math.sqrt(var)
# ---- step 5: reweighting on one synthetic week ----------------------------------------
def simulate_week(rng: random.Random, fractions: dict[str, float]) -> dict[str, dict[str, int]]:
per_day = strata_per_day()
out = {}
for h in per_day:
N = round(per_day[h] * DAYS_WEEK)
n = round(N * fractions[h])
population = [rng.random() < ASSUMED_RATE[h] for _ in range(N)]
sample = rng.sample(range(N), n)
out[h] = {"N": N, "n": n, "v_pop": sum(population),
"v_sample": sum(population[i] for i in sample)}
return out
def main() -> None:
global WORLD, CONNECTS, DAYS_WEEK, DAYS_MONTH
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--edition", choices=list(EDITIONS), default="global",
help="which fictional world's data to use (default: global)")
WORLD = EDITIONS[ap.parse_args().edition]
CONNECTS = WORLD["connects_per_day"]
DAYS_WEEK = WORLD["calling_days_per_week"]
DAYS_MONTH = WORLD["calling_days_per_month"]
PRICE, fmt, money = WORLD["price"], WORLD["fmt"], WORLD["fmt_money"]
rng = random.Random(WORLD["seed"])
print(WORLD["lender_note"])
print(f"Dataset: {WORLD['dataset']}\n")
# Step 1 --------------------------------------------------------------------------
print("1. Detection probability: P(at least one) = 1 - (1 - rate)^n")
n95 = calls_needed(1 / 1000)
print(f" calls for 95% at 1 in 1,000 : {fmt(n95)} "
f"(p = {p_detect(n95, 1/1000):.5f}; {fmt(n95 - 1)} gives {p_detect(n95 - 1, 1/1000):.5f})")
p1000 = p_detect(1000, 1 / 2000)
print(f" 1,000 calls at 1 in 2,000 : {p1000:.3f}")
n_week = calls_needed(1 / 2000)
print(f" calls for 95% at 1 in 2,000 : {fmt(n_week)} "
f"({fmt(n_week - 1)} gives {p_detect(n_week - 1, 1/2000):.5f})")
rest_week = round((CONNECTS - RISK_PER_DAY) * REST_FRACTION * DAYS_WEEK)
print(f" 2% of non-risk calls a week : {fmt(rest_week)} "
f"(p at 1 in 2,000 = {p_detect(rest_week, 1/2000):.4f})")
print(" rate n for 50% n for 95% n for 99%")
for denom in (500, 1000, 2000, 10000, 24000):
r = 1 / denom
print(f" 1/{denom:<8}{fmt(calls_needed(r, 0.50)):>10}"
f"{fmt(calls_needed(r, 0.95)):>13}{fmt(calls_needed(r, 0.99)):>13}")
assert n95 == 2995 and p_detect(2994, 1 / 1000) < 0.95 <= p_detect(2995, 1 / 1000)
assert round(p1000, 3) == 0.394
assert n_week == 5990 and round(p_detect(5989, 1 / 2000), 5) == 0.94998
assert rest_week == 14976 and rest_week > n_week
# Step 2 --------------------------------------------------------------------------
print("\n2. Upper bounds on the rate (95%)")
print(" k / n rule of 3 exact 1-sided Clopper-Pearson 2-sided")
rows = [(0, 300), (0, 400), (0, 3000), (1, 300), (2, 300)]
for k, n in rows:
r3 = f"{rule_of_three(n):.4f}" if k == 0 else " n/a"
one = zero_upper_one_sided(n) if k == 0 else cp_upper(k, n, 0.05)
two = cp_upper(k, n, 0.025)
print(f" {k} / {fmt(n):<8}{r3:>10}{one:>16.4f}{two:>26.4f}")
assert round(rule_of_three(300), 4) == 0.0100
assert round(cp_upper(0, 300, 0.025), 4) == 0.0122
assert abs(cp_upper(0, 300, 0.025) - (1 - 0.025 ** (1 / 300))) < 1e-9
assert round(zero_upper_one_sided(400) * 100, 3) == 0.746
assert round(rule_of_three(3000), 4) == 0.0010
# Step 3 --------------------------------------------------------------------------
per_day = strata_per_day()
print("\n3. T09: threats live on the angry path")
print(f" angry calls a day : {per_day['angry']:.1f} "
f"(a week: {fmt(per_day['angry'] * DAYS_WEEK)})")
threats_day = per_day["angry"] * THREAT_IN_ANGRY
print(f" threat calls a day : {threats_day:.2f} "
f"(1 in {fmt(1 / (ANGRY_SHARE * THREAT_IN_ANGRY))} connects)")
uniform = {"angry": OLD_UNIFORM, "other_risk": OLD_UNIFORM, "rest": OLD_UNIFORM}
strat = {"angry": 1.0, "other_risk": 1.0, "rest": REST_FRACTION}
same_cost = judged_per_day(strat) / CONNECTS
uniform_eq = {h: same_cost for h in uniform}
p_t09 = threat_reach_prob(uniform, PATCH_DAYS)
print(f" reaching a 2% judge : {threats_day * OLD_UNIFORM:.2f} a day")
mc = mc_reach(OLD_UNIFORM, PATCH_DAYS, 20_000, rng)
print(f" P(judge sees one, 3 days): {p_t09:.3f} exact, {mc:.3f} Monte Carlo (20,000 runs)")
print(" policy judged/day 1 day 3 days 1 week" + f"{WORLD['money_col']:>13}")
for name, fr in (("uniform 2% (before T09)", uniform),
(f"uniform {same_cost:.2%} (same count)", uniform_eq),
(f"stratified {WORLD['plan']} plan", strat)):
jd = judged_per_day(fr)
print(f" {name:<27}{fmt(jd):>10}{threat_reach_prob(fr, 1):>8.3f}"
f"{threat_reach_prob(fr, 3):>9.3f}{threat_reach_prob(fr, DAYS_WEEK):>9.3f}"
f"{fmt(jd * DAYS_WEEK * PRICE['J1'], 2):>13}")
flag_day = threat_reach_prob(strat, 1, J1_RECALL)
print(f" stratified, J1 flags at least one in a day (recall {J1_RECALL:.3f}): {flag_day:.3f}")
assert round(p_t09, 2) == 0.28 and abs(mc - p_t09) < 0.02
assert round(threats_day * OLD_UNIFORM, 2) == 0.11
assert round(judged_per_day(strat)) == 7696
for d in (1, 3, DAYS_WEEK): # never worse than uniform at the same judged count
assert threat_reach_prob(strat, d) >= threat_reach_prob(uniform_eq, d)
angry_week = per_day["angry"] * DAYS_WEEK
print(f"\n angry path judged 200 times a week needs {200 / angry_week:.1%} of "
f"{fmt(angry_week)}; judging all of it costs {money(angry_week * PRICE['J1'])} a week for J1")
all_three = sum(PRICE.values())
week_cost = judged_per_day(strat) * DAYS_WEEK * all_three
month_cost = judged_per_day(strat) * DAYS_MONTH * all_three
digits, frozen_month = WORLD["month_cost"]
print(f" {WORLD['plan']} plan with J1, J3 and J2: {money(week_cost)} a week, "
f"{money(round(month_cost, digits))} a month")
assert round(200 / angry_week, 3) == 0.154
assert round(month_cost, digits) == frozen_month
# Step 4 --------------------------------------------------------------------------
N = {h: round(per_day[h] * DAYS_WEEK) for h in per_day}
S = {h: math.sqrt(ASSUMED_RATE[h] * (1 - ASSUMED_RATE[h])) for h in N}
true_rate = sum(N[h] * ASSUMED_RATE[h] for h in N) / sum(N.values())
print(f"\n4. {AUDITS_PER_WEEK} human audits a week across strata "
f"(assumed rates: angry 1/40, other risk 1/250, rest 1/2,000)")
print(f" stratum sizes a week: " + ", ".join(f"{h} {fmt(N[h])}" for h in N))
print(" allocation angry other_risk rest expected found SE of overall rate")
results = {}
for m in ("proportional", "equal", "neyman"):
a = allocate(N, S, AUDITS_PER_WEEK, m)
found = sum(a[h] * ASSUMED_RATE[h] for h in N)
se = stratified_se(N, ASSUMED_RATE, a)
results[m] = (found, se)
print(f" {m:<14}{a['angry']:>7.1f}{a['other_risk']:>12.1f}{a['rest']:>9.1f}"
f"{found:>17.2f}{se:>21.6f}")
print(f" true overall rate under these assumptions: {true_rate:.6f}")
assert all(abs(sum(allocate(N, S, AUDITS_PER_WEEK, m).values()) - AUDITS_PER_WEEK) < 1e-6
for m in results)
assert results["neyman"][1] <= min(results["proportional"][1], results["equal"][1])
assert results["equal"][0] > results["neyman"][0] > results["proportional"][0]
# Step 5 --------------------------------------------------------------------------
wk = simulate_week(rng, strat)
tot_N = sum(s["N"] for s in wk.values())
tot_n = sum(s["n"] for s in wk.values())
tot_v = sum(s["v_sample"] for s in wk.values())
pop_rate = sum(s["v_pop"] for s in wk.values()) / tot_N
naive = tot_v / tot_n
weighted = sum(s["N"] * s["v_sample"] / s["n"] for s in wk.values()) / tot_N
print(f"\n5. One synthetic week under the {WORLD['plan']} plan (judge treated as perfect here)")
print(" stratum calls judged violations seen")
for h, s in wk.items():
print(f" {h:<11}{fmt(s['N']):>9}{fmt(s['n']):>11}{s['v_sample']:>17}")
print(f" population rate (truth) : {pop_rate:.6f}")
print(f" naive rate over judged : {naive:.6f} ({naive / pop_rate:.1f}x too high)")
print(f" stratum-weighted rate : {weighted:.6f}")
assert tot_n == 46176
assert naive > 3 * pop_rate and abs(weighted - pop_rate) / pop_rate < 0.25
print("\nAll assertions passed.")
if __name__ == "__main__":
main()
Step 3: Run it
.venv/bin/python rare-violation-sampling.py
Running it with Python 3.11 prints:
Synthetic data.
Dataset: steelyard-calls (synthetic, seed 20261121)
1. Detection probability: P(at least one) = 1 - (1 - rate)^n
calls for 95% at 1 in 1,000 : 2,995 (p = 0.95004; 2,994 gives 0.94999)
1,000 calls at 1 in 2,000 : 0.394
calls for 95% at 1 in 2,000 : 5,990 (5,989 gives 0.94998)
2% of non-risk calls a week : 14,976 (p at 1 in 2,000 = 0.9994)
rate n for 50% n for 95% n for 99%
1/500 347 1,497 2,301
1/1000 693 2,995 4,603
1/2000 1,386 5,990 9,209
1/10000 6,932 29,956 46,050
1/24000 16,636 71,897 110,522
2. Upper bounds on the rate (95%)
k / n rule of 3 exact 1-sided Clopper-Pearson 2-sided
0 / 300 0.0100 0.0099 0.0122
0 / 400 0.0075 0.0075 0.0092
0 / 3,000 0.0010 0.0010 0.0012
1 / 300 n/a 0.0157 0.0184
2 / 300 n/a 0.0208 0.0239
3. T09: threats live on the angry path
angry calls a day : 216.7 (a week: 1,300)
threat calls a day : 5.42 (1 in 24,000 connects)
reaching a 2% judge : 0.11 a day
P(judge sees one, 3 days): 0.280 exact, 0.272 Monte Carlo (20,000 runs)
policy judged/day 1 day 3 days 1 week J1 $/week
uniform 2% (before T09) 2,600 0.104 0.280 0.482 117.00
uniform 5.92% (same count) 7,696 0.277 0.623 0.858 346.32
stratified WG15 plan 7,696 0.996 1.000 1.000 346.32
stratified, J1 flags at least one in a day (recall 0.833): 0.990
angry path judged 200 times a week needs 15.4% of 1,300; judging all of it costs $9.75 a week for J1
WG15 plan with J1, J3 and J2: $1,616.16 a week, $7,003.36 a month
4. 400 human audits a week across strata (assumed rates: angry 1/40, other risk 1/250, rest 1/2,000)
stratum sizes a week: angry 1,300, other_risk 29,900, rest 748,800
allocation angry other_risk rest expected found SE of overall rate
proportional 0.7 15.3 384.0 0.27 0.001297
equal 133.3 133.3 133.3 3.93 0.001870
neyman 4.3 40.1 355.6 0.45 0.001207
true overall rate under these assumptions: 0.000675
5. One synthetic week under the WG15 plan (judge treated as perfect here)
stratum calls judged violations seen
angry 1,300 1,300 35
other_risk 29,900 29,900 110
rest 748,800 14,976 9
population rate (truth) : 0.000621
naive rate over judged : 0.003335 (5.4x too high)
stratum-weighted rate : 0.000763
All assertions passed.
Every assertion in the script passed on this run. Each number the course relies on (2,995; 0.394; 0.0100 and 0.0122; 0.28; 5,990; 15.4%; 7,696 judged calls a day; $7,003.36 a month) is checked in code, so a change that breaks one fails loudly.
Step 4: Size for detection, not for estimation
The chance of seeing at least one violation in n independent calls is 1 - (1 - rate)^n. Set it to 95% and solve for n: at 1 in 1,000 you need 2,995 calls, and 2,994 falls just short at 0.94999. A sample of 1,000 calls at 1 in 2,000 sees a violation only 39.4% of the time, so a clean week on that sample says almost nothing.
Keep the table under the headline numbers next to the sampling plan. Read across a row: halving the rate roughly doubles every column, and going from a 50% to a 95% chance costs a little over four times the calls. A violation at 1 in 24,000 connects, the T09 rate, needs 71,897 judged calls for a 95% chance. At 2,600 judged calls a day that is about 28 calling days, which is why uniform sampling cannot be the plan for a failure that lives on one path.
Step 5: Bound the rate when you see nothing
If an audit finds zero violations in n calls, the honest statement is an upper bound, not a zero. The rule of three gives a one-sided 95% bound of about 3/n, the shortcut Hanley and Lippman-Hand discuss for zero numerators 1. It comes from solving (1 - p)^n = 0.05: -ln(0.05) is 2.996, so p is about 3/n.
Zero in 300 audited calls bounds the rate at 1.00% by the rule of three and 0.99% exactly, one-sided. If your report uses a two-sided 95% interval, the Clopper-Pearson upper limit is 1.22%, because only 2.5% sits in the upper tail. Say which one you report. Zero in 400 TTS amount lines bounds the wrong-amount rate at 0.75%, the number the TTS acceptance gate uses.
The last two rows show what happens once a violation appears. One in 300 gives an exact one-sided bound of 1.57%, and no 3/n shortcut produces it. Use cp_upper for any count above zero, and keep the rule of three for the zero case it was made for.
Step 6: Replay T09 under three policies
Under the old policy the judge read 2% of 130,000 connects, 2,600 calls a day. About 5.42 threat calls a day happened, so about 0.11 a day reached the judge, and over the three days the patch was live the chance it saw at least one was 0.280. The Monte Carlo run, which draws threat calls as Poisson arrivals, gives 0.272 over 20,000 simulated three-day windows, close enough to the exact figure for the check in the script (within 0.02). Its draws come from the edition's seed, so this one line differs slightly between editions.
The middle row is the comparison that matters. Spend the same judge budget as the stratified plan, 7,696 calls a day, but spread it uniformly at 5.92%, and the three-day chance rises only to 0.623. Put the same 7,696 calls on every risk-stratum call (angry, complaint, dispute, hardship, stop request and window-edge, 5,200 a day) plus 2% of the rest, and the one-day chance is 0.996. Same cost, J1 at $346.32 a week either way. Where the samples go decides detection; how many there are matters much less.
The J1 line is a caution, not a gate. J1 v1 caught 15 of 18 Spanglish violations in its validation (recall 0.833) and failed the 0.90 bar, so it is advisory and its flags go to human audit. Even at that recall, the stratified plan flags at least one threat on the first day with probability 0.990. The deterministic lexicon still runs on 100% of calls. It held "police", "jail" and "arrest" but not "send someone out", and adding that phrase and its Spanish counterpart "mandar a alguien" [send someone] is the cheapest fix of the lot.
Judging the angry path at least 200 times a week needs 15.4% of its 1,300 calls. At $0.0075 per judged call, all 1,300 cost $9.75 a week for J1, so Steelyard judges all of it. The whole stratified plan with J1, J3 and the J2 blueprint judge comes to $1,616.16 a week, or $7,003.36 over 26 calling days, the judge line in the monthly eval budget that the load and cost chapter prices. Steelyard's prices are invented for the fictional world, not any vendor's list price.
Step 7: Allocate the human audits by what they are for
Four hundred human audits a week will not measure the overall violation rate. Under the assumed rates the true overall rate is 0.000675. The best allocation, Neyman, gives a standard error of 0.001207 on it, almost twice the rate itself. No allocation of 400 audits fixes that: at 0.0675% overall, a sample that size holds too few violations to pin the rate down.
What the allocations do differently is where the violations land. Proportional allocation puts 384 audits on the rest stratum and under one on the angry path, and expects to see 0.27 violations a week. Equal allocation puts 133 on each stratum and expects 3.93, which is what you want if the audits exist to check the judge's flags and its misses. Neyman, which weights each stratum by its size times its standard deviation, minimizes the standard error and still finds only 0.45.
So pick the allocation by the job. The judge, cheap and run on every risk-stratum call, is the detector. The audits are the judge's calibration set: allocate them to get enough violations per stratum to measure recall, and estimate the reported rate with judge scores corrected by audit labels, as the judge validation recipe does with prediction-powered inference.
Step 8: Reweight before you report a rate
The synthetic week judges every call in the angry and other-risk strata and 14,976 of the 748,800 other calls. Pooling the 154 violations seen over 46,176 judged calls gives 0.003335, 5.4 times the true population rate of 0.000621. That number is what lands on a dashboard if you divide flags by judged calls.
The stratum-weighted estimate multiplies each stratum's sample rate by its share of the population and sums: 0.000763 on this draw. It has no bias from the oversampling, but it is noisy. The rest stratum holds 96% of the week's calls and showed only 9 violations in its sample, so each violation found there or missed moves the estimate by about 0.000064, roughly a tenth of the true rate. That is why this draw lands 23% above the truth, inside the 25% the script asserts. Report the weighted rate with its stratum counts and an interval, size the rest stratum's sample for the precision you need, and never compare an oversampled rate with last month's uniform one.
Step 9: Point it at your own numbers
Replace connects_per_day, the stratum sizes in strata_per_day and the judge prices with your own, and set ASSUMED_RATE from your last month of audits, rounded up. Keep the stratum that holds the failure you fear at 100% while it is cheap, and size the rest with calls_needed for the rate you cannot afford to miss within a week. Rerun the sizing whenever a prompt patch touches a rare path: T09's patch changed the in-stratum rate, not the traffic, and a uniform sample is slow to notice.
Run the full file with --edition india for the India edition's data (its own lender, seed, digit grouping and judge prices); the India edition of the monitoring chapter explains that output. The sizing tables, the bounds, the T09 probabilities, the allocation and the 7,696 judged calls a day come out the same in both editions; the Monte Carlo line, the synthetic week and the cost lines differ because the seed and the prices do.
What to do next
Read production monitoring for voice agents for the drift signals that sit next to this plan and for the loop that turns a call like T09 into a regression case that fails before the fix ships. The general version of the stratify-then-reweight pattern, for text agents, is in sampling strategies.