On 16 November Ryan Mitchell said "I can pay it... next month on the..." and stopped for 720 ms before the date. AUTO-early-nov ran a 300 ms silence budget, so Hazel took the floor at 300 ms and asked for a payment that day while he was still finding the date (T05). On 17 November the team moved the default budget to 400 ms; once Hazel has asked for an amount or a date, the budget is 600 ms. This recipe rebuilds that decision from labeled pauses: a curve per language group, an interval on each point, and a cost model that picks a budget per campaign and call state.
The script and synthetic data for this recipe are also in the ai-evals-course repository on GitHub, in the folder voice-ai/labs/endpointing-operating-point/.
What you build
One standard-library script that treats endpointing as a curve, not a number. It generates its own seeded pauses, sweeps the silence budget from 200 to 1,250 ms, and chooses the cheapest point under a cost model you can edit.
| Function | What it does |
|---|
gen_pauses(group, n, seed) | Whole synthetic calls of 4 turns: each turn has 0 to 3 hold pauses and one true end. Holds before amounts and dates are drawn 1.6 times longer |
sweep, point | False-cutoff rate on hold pauses and mean dead air on true ends, at every budget |
at_budget(ms), at_cutoff(rate) | eot-bench's four headline numbers: false cutoff at a 300 ms and a 600 ms latency budget, latency at a 5% and a 10% false-cutoff budget |
bootstrap, choice_interval | 95% intervals that resample whole calls, for a rate and for the chosen budget itself |
expected_cost(weights), choose(campaign, state) | Cost per 1,000 turns: content-weighted false cutoffs plus seconds of dead air, minimized over the budget grid |
write_csv | One row per pause with the labels eot and hold, ready to swap for your own |
The four headline metrics come from LiveKit's eot-bench, which labels every silence of at least 100 ms in a user turn as a true end of turn or a mid-turn hold, and counts latency as dead air after the true end, not compute time 1. LiveKit built the benchmark, the dataset and the top-ranked model.
CAUTION
Vendor-reported. Treat it as a hypothesis to test on your own calls.
In LiveKit's own open benchmark (eot-bench, English, June 2026), LiveKit Turn Detector v1 had a 9.9% false-cutoff rate at a 300 ms mean-latency budget, against 12.9% for Deepgram Flux and 27.7% for ultraVAD; on the harder v1.1 English set (November 2026) the figures are 13.4% for Turn Detector v1.1, 22.3% for Flux and 39.7% for ultraVAD 2. Both sets are English, so they say nothing about how a detector handles a Spanish-dominant caller or one who switches between English and Spanish mid-sentence, and none of these numbers describes a collections call. Your Spanish audio needs its own curve. This script shows the method on synthetic pauses.
Step 1: Create a fresh environment
python3.11 -m venv .venv
.venv/bin/python --version
There is nothing to install. The script uses random, math, csv, argparse and dataclasses, and nothing touches the network.
Step 2: Save the script
Save it as endpointing-operating-point.py. It opens with the GLOBAL constants block that every recipe in this group copies from the Steelyard world, and that block carries the operating points the team chose (the WG12 table). The script never reads the operating points to make its choice; it uses them only to check that its own answer agrees.
The script also takes --edition global|india (default global). The listing below is the global edition: it leaves out the India edition's constants block, which sits between two marker comments in the full file in the course repository. Without that block, --edition india exits with a usage error.
The detector is a simple two-part rule. It commits after the silence budget, unless its hold score for the pause is at or above the campaign threshold, in which case it waits another 500 ms. The silence-only baseline ignores the score. Every generator parameter (pause medians, the 1.6 stretch before amounts and dates, the hold-score distributions, the campaign pace multipliers and language mixes) was invented for this synthetic set. None of them is a Steelyard measurement or a vendor's number.
The cost model is the part you should argue about. One unit is one second of dead air on a finished turn. A false cutoff costs 1.0 on small talk, 1.5 during a dispute and 2.5 before an amount or a date, because a cutoff there loses the entity the call exists to capture.
"""Choose an endpointing operating point per campaign and call state.
Dataset: steelyard-calls (synthetic, seed 20261121). Standard library only. Python 3.11.
No network. --edition india switches to the India edition's fictional lender and
data set (its constants block sits between the INDIA EDITION markers below).
Run:
python3.11 endpointing-operating-point.py
"""
from __future__ import annotations
import argparse
import csv
import math
import random
from dataclasses import dataclass
# --- GLOBAL constants block (copied from WORLD-GLOBAL.md, world-global-v1.1) -----
GLOBAL = {
"lender": "Steelyard Credit, Inc.",
"banner": "Synthetic data.",
"seed": 20261121,
"products": ["AUTO", "PIL", "POS", "SBL", "SDL"],
"campaigns": {
"AUTO-early-nov": {"book": "AUTO", "dpd": (1, 30)},
"PIL-digital-nov": {"book": "PIL", "dpd": (1, 60)},
"POS-short-nov": {"book": "POS", "dpd": (1, 30)},
"SBL-guar-nov": {"book": "SBL", "dpd": (1, 90)},
"SDL-biwk-nov": {"book": "SDL", "dpd": (1, 30)},
},
"calling_window": {"default": ("08:00", "21:00"), "SDL": ("09:00", "20:00")},
"customer_advocate": ("Ms Dana Brooks", "advocate@steelyard.example",
"(520) 555-0199", "1200 Example Avenue, Tucson, AZ 85701"),
# WG12 endpointing operating points (ms): default, amount-or-date asked, threshold
"points": {
"AUTO-early-nov": (400, 600, 0.70),
"PIL-digital-nov": (400, 600, 0.70),
"POS-short-nov": (400, 600, 0.70),
"SBL-guar-nov": (450, 650, 0.72),
"SDL-biwk-nov": (600, 800, 0.78),
},
"points_label": "WG12",
"detector_allowance_ms": 150, # WG11: end-of-turn delay SLO = silence budget + 150 ms
}
# ---------------------------------------------------------------------------
# Synthetic generator settings per edition. Every value here was invented for this
# recipe; none is a lender measurement or a vendor's number. The pause model per
# language group: median hold pause (ms), lognormal sigma, and the beta parameters
# of the detector's hold score on hold pauses (how well it separates hold from end).
SYNTH = {
"global": {
"groups": ["English", "Spanish-dominant", "Spanglish"],
"group_code": {"English": "ENG", "Spanish-dominant": "SPD", "Spanglish": "SPG"},
"group_model": {
"English": {"hold_median": 215, "sigma": 0.50, "hold_beta": (5.0, 1.5)},
"Spanish-dominant": {"hold_median": 215, "sigma": 0.50, "hold_beta": (4.5, 1.8)},
"Spanglish": {"hold_median": 220, "sigma": 0.50, "hold_beta": (5.0, 1.5)},
},
# Campaign profiles: pace multiplier on pauses and language mix (WG2 languages).
"campaign_profile": {
"AUTO-early-nov": {"pace": 1.00, "mix": {"English": 0.7, "Spanish-dominant": 0.3}},
"PIL-digital-nov": {"pace": 1.00, "mix": {"Spanglish": 0.6, "English": 0.4}},
"POS-short-nov": {"pace": 1.00, "mix": {"Spanish-dominant": 0.6, "English": 0.4}},
"SBL-guar-nov": {"pace": 1.10, "mix": {"English": 0.6, "Spanglish": 0.4}},
"SDL-biwk-nov": {"pace": 1.30, "mix": {"English": 0.6, "Spanish-dominant": 0.4}},
},
"longest": "SDL-biwk-nov", # older borrowers, longest budget (WG12)
"t05": ("SY-T05", "English"), # WG16 T05: P01, English, 720 ms hold before a date
"csv_group": "English",
"csv": "steelyard_pauses_english.csv",
},
}
EDITIONS = {"global": GLOBAL}
# ---- INDIA EDITION START --------------------------------------------------------
# (left out of this listing: the India edition's constants block, which adds
# EDITIONS["india"] and SYNTH["india"]; it is in the full file in the course repository)
# ---- INDIA EDITION END ----------------------------------------------------------
ARGS = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ARGS.add_argument("--edition", choices=list(EDITIONS), default="global",
help="which fictional world's data to use (default: global)")
EDITION = ARGS.parse_args().edition
WORLD = EDITIONS[EDITION]
GEN = SYNTH[EDITION]
SEED = WORLD["seed"]
GROUPS = GEN["groups"]
GROUP_MODEL = GEN["group_model"]
CAMPAIGN_PROFILE = GEN["campaign_profile"]
COL = max(16, max(len(g) for g in GROUPS) + 2) # group column width
TAGS = ["small_talk", "dispute", "amount", "date"]
BUDGETS = list(range(200, 1251, 50)) # silence budgets swept, ms
HOLD_EXTENSION_MS = 500 # extra wait when the model says "hold"
T05_HOLD_MS = 720 # T05: the caller's pause before a date (both editions)
END_BETA = (2.0, 5.0) # detector hold score on true end-of-turn pauses
CONTENT_STRETCH = {"small_talk": 1.0, "dispute": 1.2, "amount": 1.6, "date": 1.6}
# Content mix of hold pauses by call state.
STATE_TAGS = {
"default": {"small_talk": 0.6, "dispute": 0.2, "amount": 0.1, "date": 0.1},
"amount_asked": {"small_talk": 0.15, "dispute": 0.05, "amount": 0.4, "date": 0.4},
}
# Cost units: 1.0 = one second of dead air on a finished turn.
WEIGHTS = {"small_talk": 1.0, "dispute": 1.5, "amount": 2.5, "date": 2.5}
DEAD_AIR_PER_S = 1.0
@dataclass
class Pause:
call_id: str
turn_id: int
group: str
label: str # "eot" (true end of turn) or "hold" (mid-turn pause)
tag: str # content that follows a hold, or the turn's content for eot
ms: int # silence length; for eot, the caller stays silent
score: float # detector hold score in [0, 1]
def gen_pauses(group: str, n: int, seed: int, state: str = "default",
pace: float = 1.0, prefix: str = "") -> list[Pause]:
"""At least n labeled pauses, whole calls of 4 turns; holds before amounts and dates run longer."""
rng = random.Random(f"{seed}:{group}:{state}:{pace}:{prefix}")
m = GROUP_MODEL[group]
tags, probs = zip(*STATE_TAGS[state].items())
out: list[Pause] = []
call = 0
while len(out) < n:
call += 1
call_id = f"{prefix}{GEN['group_code'][group]}-{call:04d}"
call_pace = pace * math.exp(rng.gauss(0, 0.25)) # speaker effect: clusters
for turn in range(4):
for _ in range(rng.choice([0, 1, 1, 2, 2, 3])):
tag = rng.choices(tags, probs)[0]
ms = m["hold_median"] * call_pace * CONTENT_STRETCH[tag] * math.exp(rng.gauss(0, m["sigma"]))
out.append(Pause(call_id, turn, group, "hold", tag, max(100, int(ms)),
rng.betavariate(*m["hold_beta"])))
out.append(Pause(call_id, turn, group, "eot", rng.choices(tags, probs)[0],
5000, rng.betavariate(*END_BETA)))
return out
def wait_ms(p: Pause, budget: int, thr: float | None) -> int:
"""Silence before the detector commits. thr=None is the silence-only baseline."""
if thr is not None and p.score >= thr:
return budget + HOLD_EXTENSION_MS
return budget
def point(pauses: list[Pause], budget: int, thr: float | None) -> tuple[float, float]:
"""(false-cutoff rate on holds, mean dead air in ms on true ends)."""
holds = [p for p in pauses if p.label == "hold"]
ends = [p for p in pauses if p.label == "eot"]
cut = sum(p.ms >= wait_ms(p, budget, thr) for p in holds) / len(holds)
lat = sum(wait_ms(p, budget, thr) for p in ends) / len(ends)
return cut, lat
def sweep(pauses, thr, budgets=BUDGETS):
return [(b, *point(pauses, b, thr)) for b in budgets]
def at_budget(curve, ms: int) -> float:
"""False-cutoff rate at a mean-latency budget: best point with latency <= ms."""
ok = [c for b, c, lat in curve if lat <= ms]
return min(ok) if ok else float("nan")
def at_cutoff(curve, rate: float) -> float:
"""Mean latency at a false-cutoff budget: lowest latency with cutoff <= rate."""
ok = [lat for b, c, lat in curve if c <= rate]
return min(ok) if ok else float("nan")
def by_call(pauses):
calls: dict[str, list[Pause]] = {}
for p in pauses:
calls.setdefault(p.call_id, []).append(p)
return list(calls.values())
def bootstrap(pauses, stat, reps=1000, seed=SEED):
"""95% percentile interval, resampling whole calls (pauses in a call are correlated)."""
rng = random.Random(seed)
calls = by_call(pauses)
vals = sorted(stat([p for c in rng.choices(calls, k=len(calls)) for p in c]) for _ in range(reps))
return vals[int(0.025 * reps)], vals[int(0.975 * reps) - 1]
def expected_cost(pauses, budget, thr, weights=WEIGHTS):
"""Cost per 1,000 turns: weighted false cutoffs plus seconds of dead air."""
turns = sum(p.label == "eot" for p in pauses)
cost = 0.0
for p in pauses:
w = wait_ms(p, budget, thr)
if p.label == "hold" and p.ms >= w:
cost += weights[p.tag]
elif p.label == "eot":
cost += DEAD_AIR_PER_S * w / 1000
return 1000 * cost / turns
def campaign_pauses(campaign, state, n=600):
prof = CAMPAIGN_PROFILE[campaign]
out = []
for g, share in prof["mix"].items():
out += gen_pauses(g, round(n * share), SEED, state, prof["pace"], prefix=f"{campaign[:3]}-")
return out
def best_budget(pauses, thr, weights=WEIGHTS):
return min((expected_cost(pauses, b, thr, weights), b) for b in BUDGETS)
def choose(campaign, state, weights=WEIGHTS):
thr = WORLD["points"][campaign][2]
pauses = campaign_pauses(campaign, state)
best_cost, best_b = best_budget(pauses, thr, weights)
return best_b, best_cost, pauses, thr
def choice_interval(pauses, thr, reps=200, seed=SEED):
"""Range of chosen budgets over call-clustered resamples (central 95%)."""
rng = random.Random(seed)
calls = by_call(pauses)
picks = sorted(best_budget([p for c in rng.choices(calls, k=len(calls)) for p in c], thr)[1]
for _ in range(reps))
return picks[int(0.025 * reps)], picks[int(0.975 * reps) - 1]
def write_csv(path, pauses):
"""One row per pause, with eot-bench's label names (eot, hold)."""
with open(path, "w", newline="") as f:
w = csv.writer(f)
w.writerow(["call_id", "turn_id", "language", "label", "content_tag", "pause_ms", "hold_score"])
for p in pauses:
w.writerow([p.call_id, p.turn_id, p.group, p.label, p.tag,
p.ms if p.label == "hold" else "", f"{p.score:.3f}"])
def main():
# Frozen numbers shared by both editions: the operating points and the T05 replay.
assert list(WORLD["points"].values()) == [(400, 600, 0.70), (400, 600, 0.70), (400, 600, 0.70),
(450, 650, 0.72), (600, 800, 0.78)]
assert WORLD["detector_allowance_ms"] == 150 and T05_HOLD_MS == 720
label = WORLD["points_label"]
print(WORLD["banner"])
print(f"seed {SEED}; budgets {BUDGETS[0]}-{BUDGETS[-1]} ms in 50 ms steps\n")
# Step 1-2: curves per language group, silence-only vs model detector.
sets = {g: gen_pauses(g, 600, SEED) for g in GROUPS}
print("Curves per language group (default state; pauses = whole calls, at least 600)")
print(f"{'group':<{COL}}{'detector':<14}{'pauses':>7}{'holds':>6}{'cut@300':>9}{'cut@600':>9}{'lat@5%':>8}{'lat@10%':>9}")
for g, ps in sets.items():
for name, thr in (("silence-only", None), ("model t=0.70", 0.70)):
curve = sweep(ps, thr)
cuts = [c for _, c, _ in curve]
lats = [lat for _, _, lat in curve]
assert all(a >= b for a, b in zip(cuts, cuts[1:])), "cutoff must not rise with budget"
assert all(a <= b for a, b in zip(lats, lats[1:])), "latency must not fall with budget"
holds = sum(p.label == "hold" for p in ps)
print(f"{g:<{COL}}{name:<14}{len(ps):>7}{holds:>6}{at_budget(curve, 300):>9.1%}{at_budget(curve, 600):>9.1%}"
f"{at_cutoff(curve, 0.05):>8.0f}{at_cutoff(curve, 0.10):>9.0f}")
assert at_budget(sweep(ps, 0.70), 600) < at_budget(sweep(ps, None), 600)
print()
# Step 3: call-clustered intervals at the lead campaign's budgets before and after T05.
print("False cutoff at a fixed silence budget, model t=0.70, 95% CI clustered by call")
for g, ps in sets.items():
calls = len(by_call(ps))
row = f"{g:<{COL}}{calls:>4} calls"
for b in (300, 400):
est = point(ps, b, 0.70)[0]
lo, hi = bootstrap(ps, lambda x, b=b: point(x, b, 0.70)[0])
row += f" {b} ms {est:5.1%} [{lo:.1%}, {hi:.1%}]"
print(row)
print()
# Step 4-5: cost-weighted choice per campaign and call state, with its uncertainty.
print("Chosen silence budget per campaign and state (cost per 1,000 turns;")
print("range = central 95% of choices over 200 call-clustered resamples)")
print(f"{'campaign':<16}{'state':<14}{'thr':>5}{'chosen':>8}{'range':>12}{'cost':>6}{'cutoff':>8}{label:>6}")
chosen = {}
for camp, (w_def, w_amt, thr) in WORLD["points"].items():
picks = []
for state, ref in (("default", w_def), ("amount_asked", w_amt)):
b, cost, ps, _ = choose(camp, state)
lo, hi = choice_interval(ps, thr)
picks.append(b)
print(f"{camp:<16}{state:<14}{thr:>5.2f}{b:>8}{f'{lo}-{hi}':>12}{cost:>6.0f}"
f"{point(ps, b, thr)[0]:>8.1%}{ref:>6}")
assert lo <= ref <= hi, f"{camp} {state}: {label} point outside the resampled range"
chosen[camp] = tuple(picks)
assert picks[1] >= picks[0], f"{camp}: amount-or-date state must not be shorter"
longest = chosen[GEN["longest"]]
assert all(longest[0] >= d and longest[1] >= a for d, a in chosen.values()), \
f"{GEN['longest']} must be longest"
print()
# Step 6: sensitivity. Doubling the amount and date weights never shortens a choice.
heavy = dict(WEIGHTS, amount=2 * WEIGHTS["amount"], date=2 * WEIGHTS["date"])
print(f"Sensitivity: amount and date weights {WEIGHTS['amount']} -> {heavy['amount']}")
for camp in WORLD["points"]:
d2 = choose(camp, "default", heavy)[0]
a2 = choose(camp, "amount_asked", heavy)[0]
assert d2 >= chosen[camp][0] and a2 >= chosen[camp][1], "heavier weights must not shorten"
print(f"{camp:<16}default {chosen[camp][0]} -> {d2} ms amount/date {chosen[camp][1]} -> {a2} ms")
print()
# Step 7: T05's 720 ms hold before a date, scored by the wait rule (same in both editions).
t05_id, t05_group = GEN["t05"]
replay = []
for b in (300, 400, 600):
for score, note in ((0.40, "no hold signal"), (0.90, "hold signal")):
t05 = Pause(t05_id, 0, t05_group, "hold", "date", T05_HOLD_MS, score)
w = wait_ms(t05, b, 0.70)
replay.append((b, note, w, t05.ms >= w))
print(f"T05 at {b} ms, {note:<14}: waits {w:>4} ms -> {'cut off' if t05.ms >= w else 'held'}")
assert [(w, cut) for _, _, w, cut in replay] == [(300, True), (800, False), (400, True),
(900, False), (600, True), (1100, False)]
rows = sets[GEN["csv_group"]]
write_csv(GEN["csv"], rows)
print(f"\nwrote {GEN['csv']} ({len(rows)} rows)")
print("All assertions passed. The numbers are synthetic; the method transfers to your labeled pauses.")
if __name__ == "__main__":
main()
Step 3: Run it
.venv/bin/python endpointing-operating-point.py
It takes a few seconds. Running it with Python 3.11 prints:
Synthetic data.
seed 20261121; budgets 200-1250 ms in 50 ms steps
Curves per language group (default state; pauses = whole calls, at least 600)
group detector pauses holds cut@300 cut@600 lat@5% lat@10%
English silence-only 605 353 38.0% 6.5% 700 500
English model t=0.70 605 353 18.4% 2.3% 504 354
Spanish-dominant silence-only 602 354 34.7% 6.5% 700 550
Spanish-dominant model t=0.70 602 354 20.3% 3.4% 502 402
Spanglish silence-only 602 358 35.2% 8.7% 750 550
Spanglish model t=0.70 602 358 19.3% 4.7% 552 452
False cutoff at a fixed silence budget, model t=0.70, 95% CI clustered by call
English 63 calls 300 ms 14.7% [10.8%, 18.9%] 400 ms 7.4% [4.5%, 10.6%]
Spanish-dominant 62 calls 300 ms 15.5% [11.6%, 20.4%] 400 ms 8.2% [5.4%, 11.3%]
Spanglish 61 calls 300 ms 15.6% [11.6%, 20.1%] 400 ms 11.5% [7.8%, 15.7%]
Chosen silence budget per campaign and state (cost per 1,000 turns;
range = central 95% of choices over 200 call-clustered resamples)
campaign state thr chosen range cost cutoff WG12
AUTO-early-nov default 0.70 450 350-450 579 4.3% 400
AUTO-early-nov amount_asked 0.70 550 500-750 806 6.9% 600
PIL-digital-nov default 0.70 350 200-450 594 9.1% 400
PIL-digital-nov amount_asked 0.70 500 450-650 706 5.9% 600
POS-short-nov default 0.70 450 250-550 613 6.5% 400
POS-short-nov amount_asked 0.70 650 500-700 771 2.9% 600
SBL-guar-nov default 0.72 450 300-550 614 5.4% 450
SBL-guar-nov amount_asked 0.72 650 600-650 878 6.1% 650
SDL-biwk-nov default 0.78 550 500-650 849 12.7% 600
SDL-biwk-nov amount_asked 0.78 700 600-900 997 7.9% 800
Sensitivity: amount and date weights 2.5 -> 5.0
AUTO-early-nov default 450 -> 450 ms amount/date 550 -> 750 ms
PIL-digital-nov default 350 -> 450 ms amount/date 500 -> 650 ms
POS-short-nov default 450 -> 550 ms amount/date 650 -> 700 ms
SBL-guar-nov default 450 -> 450 ms amount/date 650 -> 650 ms
SDL-biwk-nov default 550 -> 650 ms amount/date 700 -> 900 ms
T05 at 300 ms, no hold signal: waits 300 ms -> cut off
T05 at 300 ms, hold signal : waits 800 ms -> held
T05 at 400 ms, no hold signal: waits 400 ms -> cut off
T05 at 400 ms, hold signal : waits 900 ms -> held
T05 at 600 ms, no hold signal: waits 600 ms -> cut off
T05 at 600 ms, hold signal : waits 1100 ms -> held
wrote steelyard_pauses_english.csv (605 rows)
All assertions passed. The numbers are synthetic; the method transfers to your labeled pauses.
It also writes steelyard_pauses_english.csv. Every assertion in the script passed on this run: both curves are monotone for every group, the model beats silence-only at the 600 ms budget, every WG12 point sits inside its resampled range, the amount-or-date state is never shorter than the default, SDL-biwk-nov is never shorter than any other campaign, doubling the amount and date weights never shortens a choice, and the T05 replay gives the waits the rule predicts.
Step 4: Read the curves, not a single number
Each row is one point on two curves. At a 300 ms latency budget the silence-only rule cuts off 38.0% of English hold pauses; the model with a hold signal cuts off 18.4%. At 600 ms the gap narrows to 6.5% against 2.3%. The size of the gap depends on the budget you read it at, which is why eot-bench reports two budgets in each direction 1.
Read the language rows apart, and read them with intervals. On this set the Spanglish model needs 552 ms of mean dead air to get under 5% false cutoffs, against 504 ms for English and 502 ms for Spanish-dominant. Yet Spanglish pauses were generated with the same score model as English and a median hold only 5 ms longer, while the Spanish-dominant hold scores were generated to separate less cleanly. At about 60 calls per group, a 50 ms gap on this metric is inside the noise. Bootstrap it by call, as Step 5 does for the cutoff rate, before you report a gap between languages. A pooled curve hides real gaps between languages; a per-language curve without intervals invents them. Note the difference between the two tables: the first one indexes by mean latency (the eot-bench convention, so it includes the 500 ms extension on true ends the model wrongly holds), and the interval table indexes by the silence budget you actually configure.
Step 5: Put an interval on every point
The pauses in one call share a speaker, so they are not independent. The script resamples whole calls 1,000 times. At a 400 ms budget the English false-cutoff rate is 7.4% with a 95% interval of 4.5% to 10.6% from 63 calls, and the Spanglish rate is 11.5% with an interval of 7.8% to 15.7% from 61 calls. The intervals overlap, and two overlapping intervals neither prove nor rule out a difference: bootstrap the difference itself by call before you call it a language effect. The synthetic speaker effect here is mild, so resampling single pauses would give about the same width on this set. On real calls, where one hesitant speaker contributes many holds, a pause-level interval comes out narrower than the truth. Resample calls either way.
When you report a false-cutoff rate per language, report the call count and this interval with it. Steelyard's tripwire for this metric fires when the Spanish-dominant false-cutoff rate stays above 4% at the chosen operating point for two weeks; read it against the interval, because a gate on the point estimate alone will flap from week to week.
Step 6: Choose a point per campaign and call state
choose minimizes cost per 1,000 turns over the 50 ms grid, using each campaign's WG12 threshold. Two patterns hold, and the script asserts both: the amount-or-date state never gets a shorter budget than the default state, and SDL-biwk-nov, whose synthetic profile stretches every pause by 1.3 for its older borrowers, gets the longest budgets of all.
The point estimates do not all equal the WG12 values. AUTO-early-nov's default state picks 450 ms where the team runs 400 ms, and PIL-digital-nov's amount-or-date state picks 500 ms where the team runs 600 ms. The range column says why that is not a contradiction: over 200 call-clustered resamples the chosen budget for AUTO's default state moves between 350 and 450 ms, and PIL's amount-or-date choice between 450 and 650 ms. On 600 pauses, the cost curve is flat enough near its minimum that 50 ms steps are inside the noise. The script asserts that every WG12 point falls inside its range, which is the right test: the team's choice is consistent with the data, and the data cannot pin it closer. If you need a tighter choice, label more pauses. Do not tune the cost weights until the point lands where you hoped.
The cutoff column is the price of each choice. SDL-biwk-nov's default state still cuts off 12.7% of holds at 550 ms, because its pauses run 1.3 times longer and its stricter 0.78 threshold gives fewer holds the 500 ms extension. That is the number to put in the operating-point memo, next to the dead air it buys.
Step 7: Test the weights before you trust the choice
The sensitivity block doubles the amount and date weights from 2.5 to 5.0. SBL-guar-nov does not move. AUTO-early-nov's amount-or-date state moves from 550 to 750 ms, and SDL-biwk-nov's from 700 to 900 ms. When a plausible change in the weights moves a campaign by 200 ms, the weights are the decision, so write them down, name an owner, and revisit them when the dispute or complaint rate changes.
Step 8: Replay the incident
The last block replays T05's 720 ms hold before a date. With no hold signal, the detector cuts it off at 300, 400 and even 600 ms of silence. With a hold signal it waits 800 ms or more and the caller keeps the floor. Under this rule, the move from 300 to 400 ms does not save this pause on its own, and neither does the 600 ms state budget. What saves it is the content: a detector that hears an unfinished date, plus a call state that knows Hazel just asked for one. That is why the gold set needs hesitations before amounts and dates oversampled, not a random sample of pauses. These six lines depend only on the wait rule and the 720 ms pause, so they are the same in both editions.
Step 9: Point it at your own pauses
Export one row per silence of at least 100 ms from your labeled calls, in the CSV layout the script writes: call_id, turn_id, language, label (eot or hold), content_tag, pause_ms and your detector's hold_score. Load it into Pause objects in place of gen_pauses, keep the call id so the bootstrap clusters correctly, and set the weights with the people who own the campaign. Run the sweep per language group and per call state. Rerun it whenever the turn detector, the STT model or the SDK version changes, because the defaults move under you.
Run the full file from the course repository with --edition india to reproduce the India edition's run, with its own seed, campaigns and language groups; the India edition of the turn-taking chapter explains that output.
What to do next
Read turn-taking, barge-in and recovery for the three turn-taking failures this recipe does not separate (talk-over, early jump-in and missed barge-in) and for recovery after an interruption. The end-of-turn delay you choose here is the first stage of the turn gap, which the latency waterfall recipe budgets against the platform SLO.