On 2026-11-13 at 17:52, at peak load, Steelyard's collections agent Hazel left a borrower 3,990 ms of silence on a tool turn, and when he said "hello? hello?" she answered twice (trace T04). The failure only shows up above about 1,800 concurrent calls, when the CRM's P95 doubles and the LLM's first-token tail widens at the same moment. A load test at 50 calls never gets there. This recipe plans the load test that would have.
The method comes from two practitioner guides. LiveKit recommends ramping concurrency up gradually and benchmarking the sustained phase, because a zero-to-full burst tests "cold-start behavior under artificial conditions, not steady-state performance" 1. AssemblyAI recommends replaying your hardest real calls at projected peak concurrency, measuring entity accuracy, P95 latency, rejected connections and cost per completed task, and running a separate spike test up to 3x your busiest hour, because real bursts happen 2. "Projected peak" on this page means today's 2,400 concurrent calls, not the 6,000 of the capstone.
The script and synthetic data for this recipe are also in the ai-evals-course repository on GitHub, in the folder voice-ai/labs/voice-load-ramp-plan/.
What you build
One standard-library script, voice-load-ramp-plan.py, that generates its own data (dataset "steelyard-calls (synthetic, seed 20261121)") and asserts every number this page relies on:
- The plan, derived from Steelyard's volumes: 15 ramp steps of 240 calls from 240 to 3,600 (1.5x of 2,400), a 10-minute hold at 2,400, and a 3-minute spike to 7,200 (3x).
- A discrete-event model of Hazel's cascaded stack on one
heapq clock: trunk and STT stream limits that reject connections, an LLM provider limit with 429s, backoff and retries, TTFT that inflates as the provider fills, CRM latency that rises with concurrent calls, and a caller who says "hello?" after 3 s of dead air.
- A per-step table: achieved concurrency, P95 tool-turn gap with a call-clustered bootstrap interval, CRM and TTFT P95, rejected connections, an amount-accuracy proxy, and LLM cost per completed task.
- The knee, the stop conditions a real run would apply, and a load gate at 1.0x and 1.5x of peak.
- The spike's rejected connections with a Wilson interval, and a cold burst to 2,400 set against the sustained phase at 2,400.
Step 1: create a fresh environment
python3.11 -m venv .venv
There is nothing to install. The script uses argparse, heapq, math and random from the standard library and never touches the network.
Step 2: save the script
The script opens with the GLOBAL constants block that every recipe in this group copies from the Steelyard world, followed by Steelyard's volumes and prices: 250,000 dials a day at a 52% connect rate, 150 s connected calls, and the invented LLM prices of $1.00 per million input tokens, $0.25 per million cached and $4.00 per million output. The tool-turn SLO is the one Steelyard set for AUTO-early-nov: the 400 ms silence budget plus 2,200 ms, so 2,600 ms on the platform turn gap.
Everything in GEN is a generator parameter invented for this synthetic stack. The limits, medians and inflation curves are not Steelyard measurements and not any vendor's limits. They were picked so the model reproduces the shape T04 describes. Swap in numbers from your own providers and your own traces, and keep the method. The stack model is the same in both editions of the course and runs on one seed, so the latency, rejection and amount-safe lines are the same whichever edition you run. Only the labels, the dialing day and the money change.
Four modeling choices matter more than the parameters:
- Arrivals are open-loop. Calls connect at the rate that would hold the target concurrency if every call lasted 150 s (Little's law). When calls get longer under load, achieved concurrency drifts above the target, as it does on a real dialer.
- Admission happens once per call. A call takes a trunk channel and an STT stream for its whole life. When either pool is full, the connection is rejected and counted.
- The LLM limit is a hard ceiling on in-flight requests. A request over the ceiling gets a 429 and retries after 250, 500 and 1,000 ms, each with jitter. After the third retry the call drops.
- Tool calls run in sequence, as they did in T04, and each one reads the CRM latency at the current concurrency. That is the shared-load case: CRM and LLM tails rise together, so the stage tails line up instead of averaging out.
The amount-accuracy column is a proxy. An amount turn counts as safe when the LLM acted on the final STT transcript, not on a partial it moved on with after a 250 ms wait, and when no "hello?" duplicate reply landed on the amount turn or the tool turn after it. A completed task is a call that finished, was amount-safe and had no duplicate reply anywhere. The cost column counts only the agent's LLM tokens at Steelyard's prices: $0.01575, or 1.575 cents, per connected call when 60% of input tokens hit the prefix cache. The table prints it in cents per completed task so the small numbers stay readable.
The script also takes --edition global|india (default global). The listing below is the global edition: it leaves out the India edition's constants block, which sits between two marker comments in the full file in the course repository. Without that block, --edition india exits with a usage error.
"""Plan a voice load test (aievals.co cookbook recipe R10).
Dataset: steelyard-calls (synthetic, seed 20261121). Python 3.11, standard library only, no network.
--edition india switches to the India edition's fictional lender and prices (its constants
block sits between the INDIA EDITION markers below). The stack model is the same in both
editions and runs on one seed, so every latency, rejection and amount-safe line matches.
A discrete-event model (heapq) of the agent's cascaded stack under load:
- admission limits: trunk channels and STT streaming concurrency (rejected connections)
- an LLM provider concurrency limit with 429s, exponential backoff and at most 3 retries
- LLM TTFT that inflates, and whose tail widens, as the provider fills up
- CRM latency that rises with concurrent calls (the T04 mechanism)
- STT final delay that rises with stream utilization (the amount-accuracy proxy)
- a caller who says "hello?" after 3 s of dead air, which makes a duplicate reply (T04)
It runs the plan: ramp in 10% steps of today's 2,400 projected peak up to 1.5x, a sustained
hold at 1.0x, a 3x spike, and a cold burst for contrast. It reports, per step, P95 tool-turn
gap with a call-clustered bootstrap interval, rejected connections, an amount-accuracy proxy
and LLM cost per completed task, then applies stop conditions and the load gate.
Run:
python3.11 voice-load-ramp-plan.py # global edition
python3.11 voice-load-ramp-plan.py --edition india # India edition
"""
from __future__ import annotations
import argparse
import heapq
import math
import random
# ---- GLOBAL constants (copied from the world bible, world-global-v1.1) ---------
GLOBAL = {
"lender": "Steelyard Credit, Inc. (US lender)",
"dataset": "steelyard-calls (synthetic, seed 20261121)",
"seed": 20261121,
"products": {"AUTO": "Auto loan", "PIL": "Personal installment, digital",
"POS": "Retail point-of-sale", "SBL": "Small business (guarantor)",
"SDL": "Credit-builder and small-dollar"},
"campaigns": { # campaign: (book, default silence budget ms, amount/date-asked budget ms)
"AUTO-early-nov": ("AUTO", 400, 600), # 300 ms before T05's fix on 2026-11-17
"PIL-digital-nov": ("PIL", 400, 600),
"POS-short-nov": ("POS", 400, 600),
"SBL-guar-nov": ("SBL", 450, 650),
"SDL-biwk-nov": ("SDL", 600, 800),
},
"calling_window_local": ("08:00", "21:00"),
"calling_window_sdl": ("09:00", "20:00"),
"customer_advocate": {"name": "Ms Dana Brooks", "email": "advocate@steelyard.example",
"phone": "(520) 555-0199",
"address": "1200 Example Avenue, Tucson, AZ 85701"},
"money_unit": "integer cents",
"usd_1500_forms": ["fifteen hundred", "one thousand five hundred", "1,500", "1500",
"$1,500.00", "mil quinientos", "mil quinientos dolares",
"1.500 [es-ES]"],
"turn_types": ["greeting", "verification", "amount-quote", "readback", "tool",
"objection", "closing"],
"hop_ms": 10, # orchestration hop, WG11
"carrier_offset_ms": 212, # WG13 example call
}
# WG1 volumes and WG20 prices [W]; everything derived from them below is [C]
GLOBAL.update({
"lender_short": "Steelyard",
"note": "Synthetic data.",
"stack_note": "The stack model is synthetic and edition-neutral (seed 20261120 in both editions).",
"volumes": {"dials_per_day": 250000, "connect_rate": 0.52, "connected_s": 150,
"attempt_s": 30, "day_hours": 16, "peak_today": 2400, "peak_capstone": 6000},
"day_label": "the 16-hour dialing day (08:00 Eastern to 21:00 Pacific)",
"mean_conc": 401, # WG1: about 400 [C]
"prices": {"input_tokens": 25000, "cached_share": 0.60, "output_tokens": 500,
"per_m_input": 1.00, "per_m_cached": 0.25, "per_m_output": 4.00},
"per_call": 0.01575, # WG20 prices, 60% cached [C]
"money": ("$", 1, 5), # (symbol, display scale, decimals) for per-call cost
"task_col": ("cents/task", 100, 3), # (header, display scale, decimals) for the table
"sms_tool": "send_disclosure_sms",
"price_ref": "WG20",
})
EDITIONS = {"global": GLOBAL}
# ---- INDIA EDITION START --------------------------------------------------------
# (left out of this listing: the India edition's constants block, which adds
# EDITIONS["india"]; it is in the full file in the course repository)
# ---- INDIA EDITION END ----------------------------------------------------------
ARGS = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ARGS.add_argument("--edition", choices=list(EDITIONS), default="global",
help="which fictional world's data to use (default: global)")
WORLD = EDITIONS[ARGS.parse_args().edition]
VOL = WORLD["volumes"]
PRICES = WORLD["prices"]
LEAD = list(WORLD["campaigns"])[0] # the first campaign in the block: AUTO-early-nov
_, BUDGET_MS, BUDGET_AMOUNT_MS = WORLD["campaigns"][LEAD] # 400 and 600 ms in both editions
SLO_TOOL_GAP_MS = BUDGET_MS + 2200 # WG11/W11: lead campaign budget 400 ms + 2,200 ms on tool turns
SLO_NONTOOL_GAP_MS = BUDGET_MS + 1500
HELLO_AFTER_MS = 3000 # T04: "hello? hello?" came after 2,990 ms of dead air
# ---- The stack model. Invented generator parameters for this synthetic test, ----
# ---- the same in both editions; not measurements of either fictional lender ----
# ---- and not any vendor's limits. Replace with yours. ----
GEN = {
"trunk_channels": 5000, # SIP channels
"stt_streams": 4500, # STT streaming concurrency limit (binds before the trunk)
"llm_concurrency": 450, # in-flight requests the LLM provider accepts, then 429
"llm_retries": 3, "llm_backoff_ms": 250, # 250, 500, 1000 ms, each x U(0.5, 1.5)
"llm_gen_hold_ms": 500, # slot held this long after the first token
"ttft_median_ms": {"tool": 520, "nontool": 430}, "ttft_sigma": 0.33,
"ttft_inflate": 2.0, "ttft_sigma_widen": 0.60, # median x (1 + 2 u^2), sigma + 0.6 u^2
"cold_prefix_ms": 700, "cache_warmup_s": 30, # cold prefix cache after traffic starts
"crm_median_ms": 200, "crm_sigma": 0.45, "crm_knee_calls": 1950, "crm_power": 3,
"stt_final_median_ms": 160, "stt_sigma": 0.40, "stt_inflate": 4.0, "stt_power": 6,
"stt_wait_cap_ms": 250, # orchestrator waits this long for the final, then uses the partial
"detector_median_ms": 60, "detector_sigma": 0.5,
"tts_ttfb_median_ms": 170, "tts_sigma": 0.30, "playout_ms": 50,
"caller_speech_median_s": 5.2, "caller_sigma": 0.45,
"agent_play_median_s": 12.0, "agent_sigma": 0.35,
"amd_ms": 2380, # WG13/W13 example call
"hello_prob": 0.6, # caller says "hello?" once dead air passes 3 s
}
# Agent turns of one F01-style call: (turn type, CRM tool calls in sequence, state before it)
CALL_PLAN = [
("greeting", 0, "open"), # scripted opener: no LLM request
("verification", 1, "default"), # verify_identity
("amount-quote", 2, "default"), # get_dues -> check_payment_status, sequential as in T04
("objection", 0, "default"),
("readback", 0, "amount-asked"), # acts on the caller's amount: the amount turn
("tool", 2, "default"), # record_promise_to_pay -> send_payment_link
("tool", 1, "default"), # the disclosure SMS (WORLD["sms_tool"])
("closing", 0, "default"),
]
LLM_TURNS = sum(1 for t in CALL_PLAN if t[2] != "open")
AMOUNT_TURN = 4
# The plan. Stop conditions and gate thresholds are this recipe's choices; set your own.
PLAN = {
"peak": VOL["peak_today"], "step_frac": 0.10, "max_mult": 1.5, "spike_mult": 3.0,
"step_s": 300, "measure_last_s": 150, "hold_s": 600, "spike_s": 180, "settle_s": 300,
"stop_p95_mult": 2.0, # stop if P95 tool-turn gap > 2x SLO
"stop_reject": 0.01, # stop if more than 1% of connections are rejected (not in the spike)
"stop_turn_fail": 0.005, # stop if more than 0.5% of calls drop on exhausted retries
"gate_amount_safe": 0.99, # gate: amount-safe share at least 99%
"gate_reject": 0.001, # gate: rejected connections at most 0.1% at 1.0x and 1.5x
}
# The stack model and its seed are edition-neutral: both editions run the same synthetic
# stack, so the ramp, knee, gate, spike and burst lines match and only labels and money differ.
SEED = 20261120
HOP = WORLD["hop_ms"]
def lognorm(rng: random.Random, median: float, sigma: float) -> float:
return median * math.exp(sigma * rng.gauss(0.0, 1.0))
def pctl(values: list[float], q: float) -> float:
"""Linear-interpolated percentile (numpy's default method)."""
s = sorted(values)
if not s:
return float("nan")
pos = (len(s) - 1) * q
lo = int(pos)
hi = min(lo + 1, len(s) - 1)
return s[lo] + (s[hi] - s[lo]) * (pos - lo)
def wilson(k: int, n: int, z: float = 1.959963984540054) -> tuple[float, float]:
if n == 0:
return (0.0, 1.0)
p = k / n
d = 1 + z * z / n
c = (p + z * z / (2 * n)) / d
h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
return (max(0.0, c - h), min(1.0, c + h))
def cluster_boot_p95(by_call: list[list[float]], rng: random.Random, b: int = 200):
"""95% interval for P95, resampling whole calls (turns in one call share its load)."""
n = len(by_call)
stats = []
for _ in range(b):
pool: list[float] = []
for i in rng.choices(range(n), k=n):
pool.extend(by_call[i])
stats.append(pctl(pool, 0.95))
return pctl(stats, 0.025), pctl(stats, 0.975)
def llm_request_cost(cached: bool) -> float:
"""Edition prices, one LLM request: 25,000 input tokens and 500 output tokens per call."""
inp = PRICES["input_tokens"] / LLM_TURNS
out = PRICES["output_tokens"] / LLM_TURNS
if cached:
c = inp * (1 - PRICES["cached_share"]) * PRICES["per_m_input"] + \
inp * PRICES["cached_share"] * PRICES["per_m_cached"]
else:
c = inp * PRICES["per_m_input"]
return (c + out * PRICES["per_m_output"]) / 1e6
# ---- Discrete-event simulation ---------------------------------------------------
class Sim:
"""Open-loop arrivals (dials connect at a scheduled rate), one event heap, one clock in ms."""
def __init__(self, phases: list[tuple[str, float, float]], seed: int, warm_at_ms: float = 0.0):
self.rng = random.Random(seed)
self.phases = phases # (label, duration s, target concurrency)
self.heap: list = []
self.seq = 0
self.active = 0 # admitted calls holding a trunk channel and an STT stream
self.llm_inflight = 0
self.warm_at = warm_at_ms # traffic start for the prefix-cache warm-up
self.calls: dict[int, dict] = {}
self.rejected: list[float] = []
self.attempts: list[float] = []
self.samples: list[tuple[float, int]] = []
def push(self, t: float, kind: str, data) -> None:
self.seq += 1
heapq.heappush(self.heap, (t, self.seq, kind, data))
def schedule_arrivals(self) -> None:
t = 0.0
for _label, dur_s, conc in self.phases:
end = t + dur_s * 1000
rate = conc / VOL["connected_s"] / 1000 # Little's law: arrivals per ms
if rate > 0:
nxt = t + self.rng.expovariate(rate)
while nxt < end:
self.push(nxt, "arrive", None)
nxt += self.rng.expovariate(rate)
t = end
self.end_ms = t
for s in range(0, int(t // 1000) + 1):
self.push(s * 1000.0, "sample", None)
def cache_hit(self, t: float) -> bool:
warm = min(1.0, max(0.0, (t - self.warm_at) / (GEN["cache_warmup_s"] * 1000)))
return self.rng.random() < warm
def run(self) -> "Sim":
self.schedule_arrivals()
while self.heap:
t, _, kind, data = heapq.heappop(self.heap)
getattr(self, "on_" + kind)(t, data)
return self
# events
def on_sample(self, t, _):
self.samples.append((t, self.active))
def on_arrive(self, t, _):
self.attempts.append(t)
if self.active >= min(GEN["trunk_channels"], GEN["stt_streams"]):
self.rejected.append(t)
return
self.active += 1
cid = len(self.calls)
self.calls[cid] = {"start": t, "turns": [], "dropped": None, "cost": 0.0,
"partial_amount": False, "dup": set(), "end": None}
# scripted opener after AMD, then the caller answers
play = GEN["amd_ms"] + lognorm(self.rng, GEN["tts_ttfb_median_ms"], GEN["tts_sigma"])
play_end = t + play + 1000 * lognorm(self.rng, 6.0, 0.2)
self.push(play_end + 1000 * lognorm(self.rng, GEN["caller_speech_median_s"],
GEN["caller_sigma"]), "caller_end", (cid, 1))
def on_caller_end(self, t, data):
cid, ti = data
call = self.calls[cid]
_ttype, _ntools, state = CALL_PLAN[ti]
budget = BUDGET_AMOUNT_MS if state == "amount-asked" else BUDGET_MS
eot = budget + lognorm(self.rng, GEN["detector_median_ms"], GEN["detector_sigma"])
u_stt = self.active / GEN["stt_streams"]
stt = lognorm(self.rng, GEN["stt_final_median_ms"] *
(1 + GEN["stt_inflate"] * u_stt ** GEN["stt_power"]), GEN["stt_sigma"])
wait = min(max(0.0, stt - eot), GEN["stt_wait_cap_ms"])
if ti == AMOUNT_TURN and stt > eot + GEN["stt_wait_cap_ms"]:
call["partial_amount"] = True # the LLM acted on a partial transcript
turn = {"ti": ti, "t0": t, "eot": eot, "retries": 0}
self.push(t + eot + wait + HOP, "llm_request", (cid, turn))
def on_llm_request(self, t, data):
cid, turn = data
call = self.calls[cid]
if self.llm_inflight >= GEN["llm_concurrency"]:
if turn["retries"] >= GEN["llm_retries"]:
call["dropped"] = t
self.end_call(t, cid)
return
back = GEN["llm_backoff_ms"] * 2 ** turn["retries"] * self.rng.uniform(0.5, 1.5)
turn["retries"] += 1
self.push(t + back, "llm_request", (cid, turn))
return
self.llm_inflight += 1
u = self.llm_inflight / GEN["llm_concurrency"]
ttype, ntools, _ = CALL_PLAN[turn["ti"]]
tool = ntools > 0
med = GEN["ttft_median_ms"]["tool" if tool else "nontool"] * (1 + GEN["ttft_inflate"] * u ** 2)
cached = self.cache_hit(t)
ttft = lognorm(self.rng, med, GEN["ttft_sigma"] + GEN["ttft_sigma_widen"] * u ** 2)
if not cached:
ttft += GEN["cold_prefix_ms"]
call["cost"] += llm_request_cost(cached)
first_token = t + ttft
self.push(first_token + GEN["llm_gen_hold_ms"], "llm_release", None)
# tools run in sequence; CRM latency rises with concurrent calls
crm_med = GEN["crm_median_ms"] * (1 + (self.active / GEN["crm_knee_calls"]) ** GEN["crm_power"])
tt = first_token
crm = []
for _ in range(ntools):
d = lognorm(self.rng, crm_med, GEN["crm_sigma"])
crm.append(d)
tt += HOP + d
tts = lognorm(self.rng, GEN["tts_ttfb_median_ms"], GEN["tts_sigma"])
play_start = tt + HOP + tts + GEN["playout_ms"]
gap = play_start - turn["t0"]
turn.update(gap=gap, ttft=ttft, crm=crm, tool=tool, t_play=play_start)
call["turns"].append(turn)
if gap > HELLO_AFTER_MS and self.rng.random() < GEN["hello_prob"]:
call["dup"].add(turn["ti"]) # "hello?" -> a second LLM request and reply
call["cost"] += llm_request_cost(cached)
play_end = play_start + 1000 * lognorm(self.rng, GEN["agent_play_median_s"], GEN["agent_sigma"])
if turn["ti"] + 1 < len(CALL_PLAN):
nxt = play_end + 1000 * lognorm(self.rng, GEN["caller_speech_median_s"], GEN["caller_sigma"])
self.push(nxt, "caller_end", (cid, turn["ti"] + 1))
else:
self.push(play_end, "hangup", cid)
def on_llm_release(self, t, _):
self.llm_inflight -= 1
def on_hangup(self, t, cid):
self.end_call(t, cid)
def end_call(self, t, cid):
self.calls[cid]["end"] = t
self.active -= 1
# ---- Measuring a window ----------------------------------------------------------
def anchor(call: dict) -> float:
"""Call-level metrics belong to the moment of the amount turn (or the drop)."""
for tr in call["turns"]:
if tr["ti"] == AMOUNT_TURN:
return tr["t0"]
return call["dropped"] if call["dropped"] is not None else call["start"]
def amount_safe(call: dict) -> bool:
return (call["dropped"] is None and not call["partial_amount"]
and not ({AMOUNT_TURN, AMOUNT_TURN + 1} & call["dup"]))
def completed(call: dict) -> bool:
return call["dropped"] is None and amount_safe(call) and not call["dup"] and call["end"] is not None
def measure(sim: Sim, t0: float, t1: float, rng: random.Random, boot: int = 200) -> dict:
tool_by_call, nontool, crm, ttft_tool = [], [], [], []
for call in sim.calls.values():
g = [tr["gap"] for tr in call["turns"] if tr["tool"] and t0 <= tr["t0"] < t1]
if g:
tool_by_call.append(g)
for tr in call["turns"]:
if t0 <= tr["t0"] < t1:
if tr["tool"]:
crm.extend(tr["crm"])
ttft_tool.append(tr["ttft"])
else:
nontool.append(tr["gap"])
tool = [x for g in tool_by_call for x in g]
anchored = [c for c in sim.calls.values() if t0 <= anchor(c) < t1]
att = sum(1 for a in sim.attempts if t0 <= a < t1)
rej = sum(1 for a in sim.rejected if t0 <= a < t1)
done = [c for c in anchored if completed(c)]
conc = [a for (t, a) in sim.samples if t0 <= t < t1]
lo, hi = cluster_boot_p95(tool_by_call, rng, boot)
return {
"conc": sum(conc) / len(conc), "p95_tool": pctl(tool, 0.95), "ci": (lo, hi),
"p95_nontool": pctl(nontool, 0.95), "crm_p95": pctl(crm, 0.95),
"ttft_p95": pctl(ttft_tool, 0.95), "attempts": att, "rejected": rej,
"reject_rate": rej / att if att else 0.0,
"drop_rate": sum(1 for c in anchored if c["dropped"] is not None) / max(1, len(anchored)),
"amount_safe": sum(1 for c in anchored if amount_safe(c)) / max(1, len(anchored)),
"dup_rate": sum(1 for c in anchored if c["dup"]) / max(1, len(anchored)),
"partial_amount": sum(1 for c in anchored if c["partial_amount"]) / max(1, len(anchored)),
"dup_amount": sum(1 for c in anchored if {AMOUNT_TURN, AMOUNT_TURN + 1} & c["dup"])
/ max(1, len(anchored)),
"cost_per_task": sum(c["cost"] for c in anchored) / max(1, len(done)),
"n_calls": len(anchored), "n_tool_turns": len(tool),
}
# ---- Step 1: the plan from the world constants -----------------------------------
def step1_plan() -> list[tuple[str, float, float]]:
print(f"Part 1. The plan, from {WORLD['lender_short']}'s volumes and prices")
w = VOL
connects = w["dials_per_day"] * w["connect_rate"]
call_s = connects * w["connected_s"] + (w["dials_per_day"] - connects) * w["attempt_s"]
mean_conc = call_s / (w["day_hours"] * 3600)
assert connects == 130000 and round(mean_conc) == WORLD["mean_conc"] # [C]
print(f" mean concurrency over {WORLD['day_label']}: {mean_conc:.0f} calls; peak today {w['peak_today']:,}"
f" ({w['peak_today'] / mean_conc:.1f}x the mean)")
per_call = LLM_TURNS * llm_request_cost(True)
assert abs(per_call - WORLD["per_call"]) < 1e-9 # WG20/W20 prices [C]
sym, scale, dec = WORLD["money"]
print(f" agent LLM cost per connected call at {WORLD['price_ref']} prices (60% cached): {sym}{per_call * scale:.{dec}f}")
peak = PLAN["peak"]
steps = [round(peak * PLAN["step_frac"] * k) for k in range(1, round(PLAN["max_mult"] / PLAN["step_frac"]) + 1)]
assert steps[0] == 240 and steps[9] == 2400 and steps[-1] == 3600 and len(steps) == 15
spike = round(peak * PLAN["spike_mult"])
assert spike == 7200
phases = []
for c in steps[:10]:
phases.append((f"ramp {c / peak:.1f}x", PLAN["step_s"], c))
phases.append(("hold 1.0x", PLAN["hold_s"], peak))
for c in steps[10:]:
phases.append((f"ramp {c / peak:.1f}x", PLAN["step_s"], c))
phases.append(("settle 1.0x", PLAN["settle_s"], peak))
phases.append(("spike 3.0x", PLAN["spike_s"], spike))
phases.append(("recover 1.0x", PLAN["settle_s"], peak))
total_s = sum(p[1] for p in phases)
offered = sum(p[1] * p[2] / VOL["connected_s"] for p in phases)
print(f" ramp steps of {int(peak * PLAN['step_frac'])} calls: {steps[0]:,} .. {steps[-1]:,} "
f"(1.5x of {peak:,}); hold {PLAN['hold_s'] // 60} min at {peak:,}; spike to {spike:,} (3x)")
print(f" {len(phases)} phases, {total_s / 60:.0f} minutes, about {offered:,.0f} call attempts")
return phases
# ---- Step 2 and 3: run the ramp, read the table ---------------------------------
def step2_run(phases) -> tuple[Sim, list[dict]]:
sim = Sim(phases, SEED).run()
rng = random.Random(SEED + 1)
rows, t = [], 0.0
for label, dur, target in phases:
t1 = t + dur * 1000
if not label.startswith(("settle", "recover")):
t0 = t if label.startswith("spike") else t1 - PLAN["measure_last_s"] * 1000
if label.startswith("hold"):
t0 = t + 150_000
m = measure(sim, t0, t1, rng)
m.update(label=label, target=target)
rows.append(m)
t = t1
return sim, rows
def verdict(m: dict) -> str:
if m["ci"][1] <= SLO_TOOL_GAP_MS:
return "pass"
if m["ci"][0] > SLO_TOOL_GAP_MS:
return "BREACH"
return "unclear"
def step3_table(rows: list[dict]) -> dict:
print()
print("Part 2. Ramp, hold and spike (P95 tool-turn gap SLO 2,600 ms; interval = call-clustered bootstrap)")
head, tscale, tdec = WORLD["task_col"]
print(f" phase target conc P95 tool gap [95% CI] CRM P95 TTFT P95 reject amount-safe {head} verdict")
for m in rows:
print(f" {m['label']:<12} {m['target']:>6,} {m['conc']:>6,.0f} {m['p95_tool']:>6,.0f} "
f"[{m['ci'][0]:>6,.0f}, {m['ci'][1]:>6,.0f}] {m['crm_p95']:>6,.0f} {m['ttft_p95']:>8,.0f}"
f" {100 * m['reject_rate']:>5.1f}% {100 * m['amount_safe']:>10.1f}% {m['cost_per_task'] * tscale:>{len(head)}.{tdec}f}"
f" {verdict(m)}")
ramp = [m for m in rows if m["label"].startswith("ramp")]
base = ramp[0]
# knee: the first step whose whole P95 interval sits above the tool-turn SLO
knee = next(m for m in ramp if verdict(m) == "BREACH")
prev = ramp[ramp.index(knee) - 1]
crm_x = {m["target"]: m["crm_p95"] / base["crm_p95"] for m in ramp}
ttft_x = {m["target"]: m["ttft_p95"] / base["ttft_p95"] for m in ramp}
print()
print(f" knee: P95 tool-turn gap is flat to about 1,200 calls ({base['p95_tool']:,.0f} -> "
f"{ramp[4]['p95_tool']:,.0f} ms), then leaves the SLO between {prev['target']:,} "
f"({verdict(prev)}, {prev['p95_tool']:,.0f}) and {knee['target']:,} ({verdict(knee)}, "
f"{knee['p95_tool']:,.0f})")
print(f" CRM P95 vs the {base['target']} step: x{crm_x[1440]:.2f} at 1,440, x{crm_x[1680]:.2f} at 1,680, "
f"x{crm_x[1920]:.2f} at 1,920, x{crm_x[2400]:.2f} at 2,400")
print(f" TTFT P95 vs the {base['target']} step: x{ttft_x[1440]:.2f} at 1,440, x{ttft_x[1680]:.2f} at 1,680, "
f"x{ttft_x[1920]:.2f} at 1,920, x{ttft_x[2400]:.2f} at 2,400")
for lbl in ("hold 1.0x", "ramp 1.2x", "ramp 1.5x"):
m = next(r for r in rows if r["label"] == lbl)
print(f" why amounts go unsafe at {lbl:<9}: duplicate reply on the amount turns "
f"{100 * m['dup_amount']:4.1f}% of calls, LLM acted on an STT partial {100 * m['partial_amount']:4.1f}%")
hold = next(r for r in rows if r["label"] == "hold 1.0x")
assert hold["partial_amount"] < 0.005 and hold["dup_amount"] > 0.10 # at peak: duplicates, not STT
assert next(r for r in rows if r["label"] == "ramp 1.5x")["partial_amount"] > 0.01
assert prev["target"] == 1680 and verdict(prev) == "pass" # knee near 1,800 [W19]
assert knee["target"] == 1920
assert crm_x[1680] < 2.0 <= crm_x[1920] # CRM P95 doubles near 1,800 [W]
assert ttft_x[2400] > ttft_x[1440] > 1.0 # the LLM tail widens with it
assert base["p95_tool"] < SLO_TOOL_GAP_MS and verdict(base) == "pass"
# frozen across editions (same stack, same seed): the knee and the hold
assert (round(prev["p95_tool"]), round(knee["p95_tool"]), round(hold["p95_tool"])) == (2549, 2787, 3451)
# W19: a load test at 50 concurrent calls never sees T04
small = Sim([("50 calls", 600, 50)], SEED + 3).run()
s = measure(small, 300_000, 600_000, random.Random(SEED + 4))
print(f" a 50-call test for 10 minutes: P95 tool gap {s['p95_tool']:,.0f} ms "
f"[{s['ci'][0]:,.0f}, {s['ci'][1]:,.0f}] on {s['n_tool_turns']} tool turns: {verdict(s)}")
assert verdict(s) == "pass"
return {"knee": knee, "prev": prev}
def step4_stop_and_gate(rows: list[dict]) -> None:
print()
print("Part 3. Stop conditions and the load gate")
stop = None
for m in rows:
if m["label"].startswith("spike"):
continue
reasons = []
if m["p95_tool"] > PLAN["stop_p95_mult"] * SLO_TOOL_GAP_MS:
reasons.append(f"P95 tool gap {m['p95_tool']:,.0f} > {PLAN['stop_p95_mult'] * SLO_TOOL_GAP_MS:,.0f}")
if m["reject_rate"] > PLAN["stop_reject"]:
reasons.append(f"rejections {100 * m['reject_rate']:.1f}% > 1%")
if m["drop_rate"] > PLAN["stop_turn_fail"]:
reasons.append(f"calls dropped on exhausted LLM retries {100 * m['drop_rate']:.2f}% > 0.5%")
if reasons and stop is None:
stop = (m, reasons)
if stop:
print(f" a real run stops at '{stop[0]['label']}' ({stop[0]['target']:,} calls): " + "; ".join(stop[1]))
else:
print(" no stop condition fired on the ramp")
for lbl in ("hold 1.0x", "ramp 1.5x"):
m = next(r for r in rows if r["label"] == lbl)
checks = {
"P95 tool gap <= 2,600 (upper CI)": m["ci"][1] <= SLO_TOOL_GAP_MS,
"amount-safe >= 99%": m["amount_safe"] >= PLAN["gate_amount_safe"],
"rejections <= 0.1%": m["reject_rate"] <= PLAN["gate_reject"],
}
res = "PASS" if all(checks.values()) else "FAIL"
bad = [k for k, v in checks.items() if not v]
print(f" gate at {lbl:<9} ({m['target']:,}): {res}" + (f" failed: {', '.join(bad)}" if bad else ""))
hold = next(r for r in rows if r["label"] == "hold 1.0x")
assert verdict(hold) == "BREACH" # T04 is a peak-load failure: today's stack fails at 1.0x
spike = next(r for r in rows if r["label"].startswith("spike"))
lo, hi = wilson(spike["rejected"], spike["attempts"])
limit = min(GEN["trunk_channels"], GEN["stt_streams"])
print(f" spike to 7,200 for {PLAN['spike_s']} s: {spike['rejected']:,} of {spike['attempts']:,} connections "
f"rejected ({100 * spike['reject_rate']:.1f}%, Wilson {100 * lo:.1f}-{100 * hi:.1f}%); "
f"admitted calls capped at the {limit:,} STT streams")
ramp15 = next(r for r in rows if r["label"] == "ramp 1.5x")
assert spike["rejected"] > 0 and ramp15["rejected"] == 0
assert (spike["rejected"], spike["attempts"]) == (3389, 8583) # frozen across editions
# ---- Step 4: cold burst vs the sustained phase ------------------------------------
def step5_burst(rows: list[dict]) -> None:
print()
print("Part 4. A cold burst to 2,400 vs the sustained phase at 2,400")
# 2,400 calls connect in 10 s from idle, with a cold prefix cache; measure the first 150 s
burst_rate_conc = 2400 / 10 * VOL["connected_s"] # concurrency-equivalent arrival rate
sim = Sim([("burst", 10, burst_rate_conc), ("after", 290, 0)], SEED + 7).run()
m = measure(sim, 0, 150_000, random.Random(SEED + 8))
retries = sum(tr["retries"] for c in sim.calls.values() for tr in c["turns"])
llm_req = sum(len(c["turns"]) for c in sim.calls.values())
hold = next(r for r in rows if r["label"] == "hold 1.0x")
print(f" cold burst : {len(sim.calls):,} calls, P95 tool gap {m['p95_tool']:,.0f} ms "
f"[{m['ci'][0]:,.0f}, {m['ci'][1]:,.0f}], TTFT P95 {m['ttft_p95']:,.0f}, "
f"{100 * retries / max(1, llm_req):.1f}% of LLM requests retried after a 429, "
f"amount-safe {100 * m['amount_safe']:.1f}%")
print(f" sustained : P95 tool gap {hold['p95_tool']:,.0f} ms [{hold['ci'][0]:,.0f}, {hold['ci'][1]:,.0f}], "
f"TTFT P95 {hold['ttft_p95']:,.0f}, amount-safe {100 * hold['amount_safe']:.1f}%")
print(f" the burst reports {m['p95_tool'] - hold['p95_tool']:+,.0f} ms on P95: cold caches and "
f"synchronised first turns, not the steady state a peak hour runs in")
assert m["p95_tool"] > hold["p95_tool"]
def main() -> None:
print(f"{WORLD['lender']}: load-test plan, {WORLD['dataset']}")
print(f"{WORLD['note']} {WORLD['stack_note']}")
print()
phases = step1_plan()
_sim, rows = step2_run(phases)
step3_table(rows)
step4_stop_and_gate(rows)
step5_burst(rows)
print()
print("All assertions passed.")
if __name__ == "__main__":
main()
Step 3: run it
.venv/bin/python voice-load-ramp-plan.py
The run takes about 15 seconds on a laptop. It simulates about 85,000 call attempts over 98 minutes of model time, and every run prints the same numbers.
Running it with Python 3.11 prints:
Steelyard Credit, Inc. (US lender): load-test plan, steelyard-calls (synthetic, seed 20261121)
Synthetic data. The stack model is synthetic and edition-neutral (seed 20261120 in both editions).
Part 1. The plan, from Steelyard's volumes and prices
mean concurrency over the 16-hour dialing day (08:00 Eastern to 21:00 Pacific): 401 calls; peak today 2,400 (6.0x the mean)
agent LLM cost per connected call at WG20 prices (60% cached): $0.01575
ramp steps of 240 calls: 240 .. 3,600 (1.5x of 2,400); hold 10 min at 2,400; spike to 7,200 (3x)
19 phases, 98 minutes, about 85,440 call attempts
Part 2. Ramp, hold and spike (P95 tool-turn gap SLO 2,600 ms; interval = call-clustered bootstrap)
phase target conc P95 tool gap [95% CI] CRM P95 TTFT P95 reject amount-safe cents/task verdict
ramp 0.1x 240 234 2,088 [ 2,050, 2,124] 424 890 0.0% 100.0% 1.575 pass
ramp 0.2x 480 490 2,093 [ 2,066, 2,127] 431 908 0.0% 100.0% 1.579 pass
ramp 0.3x 720 698 2,119 [ 2,097, 2,153] 440 918 0.0% 100.0% 1.575 pass
ramp 0.4x 960 973 2,151 [ 2,126, 2,181] 471 917 0.0% 99.9% 1.577 pass
ramp 0.5x 1,200 1,191 2,228 [ 2,213, 2,252] 507 953 0.0% 100.0% 1.577 pass
ramp 0.6x 1,440 1,408 2,345 [ 2,323, 2,361] 571 969 0.0% 99.9% 1.580 pass
ramp 0.7x 1,680 1,718 2,549 [ 2,523, 2,568] 718 988 0.0% 98.9% 1.614 pass
ramp 0.8x 1,920 1,978 2,787 [ 2,753, 2,815] 862 1,033 0.0% 97.3% 1.670 BREACH
ramp 0.9x 2,160 2,100 2,921 [ 2,888, 2,950] 935 1,057 0.0% 95.1% 1.762 BREACH
ramp 1.0x 2,400 2,426 3,383 [ 3,353, 3,417] 1,244 1,115 0.0% 85.7% 2.264 BREACH
hold 1.0x 2,400 2,474 3,451 [ 3,437, 3,470] 1,281 1,128 0.0% 83.2% 2.434 BREACH
ramp 1.1x 2,640 2,685 3,850 [ 3,807, 3,888] 1,525 1,179 0.0% 73.6% 3.283 BREACH
ramp 1.2x 2,880 2,993 4,525 [ 4,490, 4,575] 1,967 1,261 0.0% 54.6% 6.839 BREACH
ramp 1.3x 3,120 3,261 5,221 [ 5,172, 5,260] 2,392 1,349 0.0% 46.3% 11.828 BREACH
ramp 1.4x 3,360 3,442 5,775 [ 5,729, 5,843] 2,731 1,413 0.0% 41.4% 17.881 BREACH
ramp 1.5x 3,600 3,891 7,446 [ 7,368, 7,523] 3,700 1,714 0.0% 38.1% 36.011 BREACH
spike 3.0x 7,200 4,126 10,386 [10,295, 10,466] 5,179 2,416 39.5% 25.5% 41.579 BREACH
knee: P95 tool-turn gap is flat to about 1,200 calls (2,088 -> 2,228 ms), then leaves the SLO between 1,680 (pass, 2,549) and 1,920 (BREACH, 2,787)
CRM P95 vs the 240 step: x1.35 at 1,440, x1.69 at 1,680, x2.03 at 1,920, x2.93 at 2,400
TTFT P95 vs the 240 step: x1.09 at 1,440, x1.11 at 1,680, x1.16 at 1,920, x1.25 at 2,400
why amounts go unsafe at hold 1.0x: duplicate reply on the amount turns 16.8% of calls, LLM acted on an STT partial 0.0%
why amounts go unsafe at ramp 1.2x: duplicate reply on the amount turns 45.3% of calls, LLM acted on an STT partial 0.0%
why amounts go unsafe at ramp 1.5x: duplicate reply on the amount turns 60.5% of calls, LLM acted on an STT partial 3.1%
a 50-call test for 10 minutes: P95 tool gap 2,063 ms [2,011, 2,188] on 396 tool turns: pass
Part 3. Stop conditions and the load gate
a real run stops at 'ramp 1.3x' (3,120 calls): P95 tool gap 5,221 > 5,200
gate at hold 1.0x (2,400): FAIL failed: P95 tool gap <= 2,600 (upper CI), amount-safe >= 99%
gate at ramp 1.5x (3,600): FAIL failed: P95 tool gap <= 2,600 (upper CI), amount-safe >= 99%
spike to 7,200 for 180 s: 3,389 of 8,583 connections rejected (39.5%, Wilson 38.5-40.5%); admitted calls capped at the 4,500 STT streams
Part 4. A cold burst to 2,400 vs the sustained phase at 2,400
cold burst : 2,450 calls, P95 tool gap 4,255 ms [4,155, 4,352], TTFT P95 2,472, 3.6% of LLM requests retried after a 429, amount-safe 86.2%
sustained : P95 tool gap 3,451 ms [3,437, 3,470], TTFT P95 1,128, amount-safe 83.2%
the burst reports +805 ms on P95: cold caches and synchronised first turns, not the steady state a peak hour runs in
All assertions passed.
Part 1 also shows why the peak, not the mean, sizes the test. Steelyard's dialing day runs 16 hours, from 08:00 Eastern to 21:00 Pacific, so its mean concurrency is about 400 calls (401 in the output), and the 2,400-call peak is 6.0x that. A test sized on the mean would sit well below the knee.
Step 4: find the knee
P95 tool-turn gap is flat from 240 to 1,200 calls (2,088 to 2,228 ms), bends, and leaves its 2,600 ms SLO between the 1,680 step (2,549 ms, interval below the SLO) and the 1,920 step (2,787 ms, interval above it). That is the knee near 1,800 calls that T04 exposed. The dependency columns say why: CRM P95 is 2.03x its light-load value at 1,920 calls, and TTFT P95 has widened by 16% at the same step and 25% at 2,400. Both tails land on the same turns, a slow first token followed by two sequential CRM calls, so they add instead of averaging out. In this model the CRM does most of the damage. Hold the TTFT curve flat (set ttft_inflate and ttft_sigma_widen to 0) and a steady 1,920-call step still breaches; hold the CRM flat (set crm_knee_calls very high) and it passes. That ranks the fixes before anyone opens a ticket.
Two lines in the output make the case for doing this at all. The 50-call test passes with a P95 of 2,063 ms, which is what a test sized for convenience reports. And the verdict column uses the bootstrap interval, resampled by whole call, because the tool turns in one call share that call's load and are not independent draws 3. A step counts as a breach only when the whole interval sits above the SLO.
Read the other columns past the knee:
- Amount-safe falls from 100.0% at light load to 83.2% in the hold at 2,400, and all of the loss is duplicate replies after "hello?" on the amount turns (16.8% of calls), the second half of T04. The LLM acting on an STT partial barely registers until the 1.5x step (3.1% of calls), as the STT streams fill.
- Cost per completed task rises from 1.575 cents to 2.434 cents in the hold, about 55% more, though the tokens per call barely change. Calls that ended with a duplicate reply or a wrong amount still burned their tokens and no longer count as completed tasks. At 1.5x it is 36.011 cents.
- Past the knee, achieved concurrency mostly runs above the target (3,891 against 3,600 at 1.5x), because calls with slower turns stay on the line longer.
Step 5: stop the run and apply the gate
A real run would stop at the 1.3x step, where P95 tool-turn gap reaches 5,221 ms, past the stop condition of twice the SLO. The model keeps going so you can see the shape. On a real stack the remaining steps only spend trunk minutes and provider quota on a system you already know is past its limit. Write the stop conditions down before the run and let the load generator apply them. Here they are P95 tool-turn gap above 2x the SLO, more than 1% rejected connections outside the spike, and more than 0.5% of calls dropped after exhausted LLM retries.
The gate runs at 1.0x and 1.5x of projected peak, as in Steelyard's load gate: the upper bound of the P95 tool-turn interval at or below 2,600 ms, amount-safe at or above 99%, and rejected connections at or below 0.1%. The amount-safe and rejection thresholds are this recipe's choices, not Steelyard numbers, so set your own. Today's stack fails at 1.0x on latency and on amounts. That is the finding T04 should have produced before it reached a borrower: the system is under-provisioned for the peak it already carries, and the fix is per dependency: parallel tool calls, a CRM read replica or cache, a higher LLM limit.
Step 6: read the spike and the cold burst
The spike to 7,200 calls for 180 s rejected 3,389 of 8,583 connections (39.5%, Wilson 38.5% to 40.5%) once admitted calls hit the 4,500 STT streams in the model. The 1.5x ramp step rejected none, so the spike is what found the admission ceiling. A rejection rate is a single binary proportion, so a Wilson interval fits here. Turn gaps during the spike are not a capacity number, because the stack is past its knee before the spike starts.
Find out what each provider does above its limit before you run the spike, because that decides whether the spike shows rejections or a bill. Retell's pricing page, for example, lists an optional Concurrency Burst add-on, at $0.10 a minute extra, under which calls above the concurrency limit connect instead of being rejected (vendor-reported, as of November 2026) 4.
The cold burst puts 2,400 calls on the stack within 10 s, from idle, with a cold prefix cache. It reports a P95 tool-turn gap of 4,255 ms, about 800 ms worse than the sustained hold at the same concurrency, a TTFT P95 of 2,472 ms against 1,128 ms, and 3.6% of LLM requests retried after a 429. Those come from cold caches and from every call reaching its first tool turn at the same moment, which a real peak hour does not do. Its amount-safe figure (86.2%) is not comparable either, because the burst's concurrency starts falling as soon as its calls end. Size the fix on the sustained phase, and keep the burst as a separate test of cold starts after a deploy or a failover.
Step 7: run it against your own stack
The model is the plan. The real test replaces each part with the real thing:
- Load generator: replay your hardest real calls, not the happy path 2. For a collections agent that means tool turns with two CRM calls, amount and date turns, English, Spanish and Spanglish callers and noisy lines, played into the real phone path from a simulated caller.
- Phases: keep the 10% steps, the hold at 1.0x and the separate spike. Measure the second half of every step so the system has settled at the new level.
- Columns: take P95 tool-turn gap from turn spans (the latency waterfall recipe shows the parser), per-dependency P95 from the same spans, rejected connections from the trunk's and the STT provider's own counters, and amount entity error rate from the replayed calls' known amounts in place of the proxy (the entity error rate recipe scores them).
- Cost: add STT, TTS and trunk minutes to the LLM tokens, and divide by completed tasks, never by calls. Steelyard budgets $2,500 for each pre-release load test run (an invented price), which is the number to check the plan's duration against.
Run the full file with --edition india for the India edition's lender, dialing day and prices; the India edition of the load and cost chapter explains that output. The ramp, knee, gate, spike and burst lines come out the same in both editions; only the labels, the mean concurrency and the cost column differ.
What to do next
- Run the plan twice a release: once before, as the gate at 1.0x and 1.5x of projected peak, and once after any change to a dependency's limits. Report P95 tool-turn gap with its call-clustered interval, never a mean.
- Re-plan for 6,000 concurrent calls, the capstone's peak after next quarter's expansion, by changing
PLAN["peak"]. Every admission limit in GEN then sits inside the ramp.
- Read the load and cost chapter for the eval budget that pays for these runs, and cost and latency evaluation for the general trade-off.
- For percentile intervals on clustered data, see bootstrap and paired tests.