At 17:52 on 13 November 2026, inside the evening peak, a borrower on Steelyard's auto book asked Hazel for his total, waited through 3,990 ms of silence on a tool turn, said "hello? hello?", and then heard Hazel answer twice, the second reply on top of the first (trace T04). The fault shows up only past about 1,800 concurrent calls, so the 50-call load test the team ran before release could not have seen it. This chapter first covers what a voice load test should measure and how to drive one, then what the evaluation program costs each month and how to spend a capped budget where the violations are.
What happened at peak?
The borrower wanted one number. To give it, Hazel made two CRM reads in a row, at the moment every other call on the platform was hitting the same CRM, and she started from a cold prompt prefix; the turn gap reached 3,990 ms, and his "hello? hello?" kicked off a second reply that played over the first.
Excerpt from T04:
t_ms ch event / text
40210 caller vad_end REF: "how much do I owe in total?"
40590 agent eot_commit (end-of-turn delay 380 ms)
40600 agent llm request (orchestration hop 10 ms); first token 41820 (TTFT 1,220 ms: cold prefix, account block placed before the policy text)
41830 agent tool get_dues (hop 10 ms), end 42690 (860 ms; CRM P95 under load)
42700 agent tool check_payment_status (hop 10 ms), end 43950 (1,250 ms; sequential, not parallel)
43200 caller REF: "hello? hello?" (2,990 ms of dead air so far)
43580 agent eot_commit on "hello hello" -> a second LLM request starts
43960 agent tts request (hop 10 ms)
44150 agent TTS first byte (TTFB 190 ms) -> play_start 44200 (playout 50 ms; turn gap 3,990 ms)
44900 agent the second reply starts over the first: "Yes, I'm here. Your total is..."
Two things change together once concurrency passes about 1,800: the CRM's P95 doubles, and the LLM's first-token tail gets wider. Steelyard's projected peak today is 2,400 concurrent calls (weekdays 12:00 to 14:00 and 17:00 to 19:00 Central), comfortably past that line. The pre-release test ran at 50 calls, about 2% of peak, where neither dependency had started to bend.
Diagnosis: a shared-load failure. Two dependencies slowed at once, their tails landed on the same turn, and nothing had been tested at a load where either one slows.
What should a voice load test measure?
Push your hardest real calls through at projected peak concurrency and read four numbers: entity accuracy, P95 latency, rejected connections, and cost per completed task 1. On a voice agent: amount entity error rate, P95 platform turn gap per turn type, refused calls, and spend per correctly completed task.
Replay the hard calls, not the easy ones. On Steelyard's book the risky turns are tool turns that need two CRM reads, the turns where an amount or a date is said, Spanish-dominant and Spanglish callers, and noisy mobile lines. Send those through the real phone path with a simulated caller, so STT and the turn detector receive the audio production gives them; the simulation chapter covers how such callers are built and validated. Load-test calls go to Steelyard's own test numbers, never to borrowers.
Score entities, because load changes what the agent hears. When STT streams queue up, the final transcript lands later, and an orchestrator that waits only so long acts on the partial instead. On an amount turn that is how "two fifty" can be written down as "two fifteen", or how a duplicate reply lands on the read-back. Every replayed call carries a known amount and date, so the run can be scored with entity error rate at each load step, the same way it is scored at rest.
Report P95 per turn type, never a mean. Tool turns get their own line, separate from greetings and read-backs. The terms (end-of-turn delay, LLM TTFT, tool time, TTS TTFB, playout) are the ones the latency chapter defines, and this page uses them unchanged. The interval on that percentile comes from a bootstrap that resamples whole calls, because turns inside one call share that call's load and are not independent draws 2.
Count rejected connections where they happen. A call the system refused never produces a trace, so the trace store will undercount it. Pull refusals from the SIP trunk, the dialer and each provider's own counters.
Price per completed task, not per call. At Steelyard's invented prices ($1.00 per million input tokens, $0.25 per million cached input tokens, $4.00 per million output tokens) Hazel's LLM tokens cost about $0.01575 per connected call: 10,000 uncached input tokens ($0.01), 15,000 cached ($0.00375) and 500 output ($0.002), since 60% of the 25,000 input tokens are cached. Load hardly moves that figure. It moves the denominator instead: if one call in six under load ends with a wrong amount or a doubled reply, the cost per completed task goes up by a fifth (1 / (5/6) = 1.2) while every unit price stays the same.
Drive arrivals at a rate, not a fixed concurrency. Little's law says concurrency equals arrival rate times mean call length 3. Steelyard's connected calls average 150 s, so 2,400 concurrent calls means 16 new calls a second. A generator that starts calls at that rate (open loop) lets concurrency rise past the target when calls slow down, which is what a real dialer does. A generator that keeps exactly 2,400 calls up (closed loop) starts fewer calls whenever calls get longer, and so hides the very overload the test exists to find.
Why ramp instead of burst?
A zero-to-full jump measures startup, not capacity; LiveKit calls it "cold-start behavior under artificial conditions, not steady-state performance" 4. Climb in steps, let each settle, and measure its second half. Run a separate spike test, because real bursts happen; AssemblyAI suggests up to 3x the busiest hour 1.
Steelyard's plan follows from its volumes. The ramp has 15 steps of 240 calls each (10% of the 2,400 peak), climbing to 3,600 (1.5x), with five minutes per step. A 10-minute hold at 2,400 follows. Last comes a 3-minute spike to 7,200 (3x), which needs 48 new calls a second. The knee is where P95 stops being flat; in practice, the highest step whose upper interval bound still sits under the SLO. On AUTO-early-nov the tool-turn SLO is the 400 ms silence budget plus 2,200 ms, which is 2,600 ms; on the slower SDL-biwk-nov campaign it is 2,800 ms.
Figure: Synthetic Steelyard data from the stack model in the load ramp recipe, AUTO-early-nov tool turns; only the plotted steps are drawn. Top: the test plan against today's 2,400-call peak. Bottom: P95 tool-turn gap per step. It is flat up to 1,200 calls, crosses the 2,600 ms SLO between the 1,680 and 1,920 steps, and passes the stop rule (twice the SLO) by 1.3x. The spike exists to measure admission and refusals; its latency is not a capacity figure.
Write the stop conditions before the run. Every step past the knee burns trunk minutes and provider quota on a system already shown to be over its limit. The load ramp recipe stops when P95 tool-turn gap passes twice the SLO, when more than 1% of connections are refused outside the spike, or when more than 0.5% of calls drop after the LLM retries run out. The load generator enforces those rules itself; nobody watches a dashboard waiting to press a button.
Use the spike to learn admission. Three questions: which pool fills first, what happens to the calls above it, and how long P95 takes to come back down. Before the run, ask each provider what it does when you exceed its limit, because the answer may come back as refusals or as an invoice.
CAUTION
Vendor-reported. Treat it as a hypothesis to test on your own calls.
One example of the invoice case: Retell's pricing page lists an optional Concurrency Burst add-on at $0.10 a minute extra, under which calls above the concurrency limit connect instead of being refused (vendor-reported, as of November 2026) 5. A spike test against such a plan passes on refusals and fails on the bill.
Run the cold burst too, on its own. After a deploy or a failover, traffic arrives on idle workers with empty prefix caches. That case deserves its own test. It is never the number you size capacity from.
Which dependencies fail first?
Usually the ones whose tails grow with concurrency and whose limits someone else sets: the CRM, STT streaming concurrency, the TTS rate limit, the LLM's in-flight limit and first-token tail, and the SIP trunk. Record a P95 for each from spans on one clock at every step; whichever bends first sets the knee.
| Dependency | Behavior as load rises | Record at each step | Role in T04 |
|---|
| CRM | Requests queue; tool time climbs with concurrent calls | Tool time P95 per tool name | get_dues 860 ms, then check_payment_status 1,250 ms, one after the other |
| LLM provider | 429s past its in-flight limit, backoff retries, a wider TTFT tail, more prefix-cache misses | TTFT P95, 429 and retry rate, share of input tokens served from cache | TTFT 1,220 ms on a cold prefix |
| STT provider | Refuses streams past its concurrency limit; the final transcript arrives later | STT final delay P95, refused streams, share of turns acted on a partial | Not the cause here; it is where the entity risk sits |
| TTS provider | Rate limits, queued requests, a TTFB tail | TTS TTFB P95, 429s | 190 ms, inside the 250 ms budget |
| SIP trunk and carriers | Channels run out, call-setup rate caps bite | Failed call setups, refused connections | Not involved |
Three ways of reading this table go wrong again and again.
Adding up the stage P95s. If stage delays were independent, the sum of their P95s would usually overstate the P95 of the turn. Under shared load the CRM and LLM tails rise together and line up, so the sum gets close to the real total, as it did in T04. Either way, measure the total from spans and use the stage numbers only to decide who fixes what; the latency chapter owns that rule.
Fixing the loudest dependency instead of the binding one. In T04 the CRM tail and the LLM tail landed on the same turn, one that made two CRM reads in sequence. The cheapest fix may not touch either provider: issue the two reads in parallel, and have Hazel start the filler on time ("One second, I'm checking that." or, on a Spanish call, "Un momento, lo estoy revisando." [One moment, I'm checking it.]) within 1,200 ms of the caller stopping.
Leaving the eval pipeline out of the test. STT re-transcription and the judges draw on the same provider quotas as the agent, and they want them at the same hours. Score off the call path, and put the eval workers inside the load test so their draw is counted; the judges chapter covers asynchronous scoring.
What does an eval cost per call?
On Steelyard's invented price list a deterministic check costs $0.0001 a call, a text judge about $0.0075 per dimension, the blueprint judge $0.02, batch STT re-transcription $0.075 and a human audit about $3.125. The 31,250-to-1 spread from cheapest to dearest, not any single price, decides what runs on every call.
The prices belong to the fictional world. They are not any vendor's list price.
| Line | Unit cost | Where it comes from |
|---|
| Deterministic checks and lexicon | $0.0001 per call | Code over the trace |
| Text judge (J1, J3) | about $0.0075 per judged call per dimension | about 6,000 input and 300 output tokens at $1.00 and $4.00 per million |
| Blueprint judge (J2) | $0.02 per judged call | Feature extraction, then a text judge |
| Raw-audio judge (for comparison only) | $0.125 per judged call | Not in the plan |
| STT re-transcription, both channels | $0.03 per call-minute, $0.075 per 150 s call | Batch |
| Simulated call | $0.30 per audio call; $0.025 per text scenario run | Caller model, TTS, STT, trunk minutes |
| Human audit | $1.25 per call-minute, about $3.125 per call | Bilingual English-Spanish reviewer at about $30 an hour, working at 2.5x real time |
| Load test | $2,500 per pre-release run | Trunk minutes, provider quota, people |
Two comparisons keep these numbers in proportion. First, the three judges together cost $0.035 per judged call, about 2.2 times the $0.01575 Hazel's own LLM tokens cost on that call; judging every call would cost more than running the agent. Second, Hazel's LLM bill comes to about $2,047.50 a day, $53,235 over 26 calling days, so the $50,000 monthly eval cap is roughly the size of the agent's token bill.
Here is what the cap buys today, line by line (all lines computed from the prices above and Steelyard's volumes: 26 calling days, 30 nightly runs, 22 working days of commits):
| Line | Monthly cost |
|---|
| Deterministic checks on 100% of connects (130,000 a day) | $338.00 |
| Judges: J1 and J3 (text) plus J2 (blueprint) on 7,696 judged calls a day | $7,003.36 |
| STT re-transcription on 5% of connects | $12,675.00 |
| Human audit, 400 calls a week | $5,416.67 |
| Nightly audio simulation, 240 scenarios at k=4 | $8,640.00 |
| Per-commit text runs, 600 at k=2, about 10 commits a day | $6,600.00 |
| Two pre-release load tests | $5,000.00 |
| Total | about $45,673.03, inside the $50,000 cap |
Three lines grow with call volume (checks, judges, STT, together $20,016.36); the other four do not. That split decides how much headroom the plan really has, and it comes back in the memo.
Vendor platforms bill their testing in the same units, per message and per minute.
CAUTION
Vendor-reported. Treat it as a hypothesis to test on your own calls.
Retell's docs (vendor-reported, as of November 2026) say tests bill at production rates: text tests (Playground, simulation, batch) cost roughly $0.001 to $0.05 per message depending on the model, a simulated run bills both the agent's model and the simulated user's model, and each batch case adds one post-call-analysis unit for grading; web call tests cost roughly $0.07 to $0.31 per minute; phone tests add about $0.015 per minute of telephony on a Retell number 6. AI QA on production calls is free for the first 100 minutes of analyzed call time per workspace and $0.10 per analyzed minute after that 7. Of those modes, only simulation testing returns a verdict on a run; web and phone tests give you real audio and no pass or fail.
When you compare quotes, then, price the verdict and the audio as two separate things.
How do you budget judges when the agent burns billions of tokens a day?
Measure judge spend against what the judges find, not against the agent's token bill. Hazel reads about 3.25 billion input tokens a day; Steelyard's judges read about 92 million. Price each sampling policy, count the violations it puts in front of a judge, and compare cost per detected violation.
A 35-to-1 token ratio makes judge spend look like a rounding error. It says nothing about whether judge tokens fall on the calls where violations happen, and per judged call the judges already cost more than the agent does.
Worked example: three policies, priced per 1,000 connected calls. The violation is T09's threat on the angry-borrower path ("we may have to send someone out to your house", banned by Steelyard policy). In Steelyard's world rates, 1 connected call in 600 takes the angry path, and while the bad prompt patch was live about 1 in 24,000 connects carried a threat, every one of them on that path. So 1,000 connects hold about 0.042 threat calls, roughly 5.4 a day at 130,000 connects. Deterministic checks and the lexicon run on 100% of calls under every policy ($0.10 per 1,000), so the table prices only the judges, at $0.035 per judged call for J1, J2 and J3 together.
| Policy | Calls judged per 1,000 | Judge cost per 1,000 | Threat calls reaching a judge per 1,000 | Per day | Cost per threat reaching a judge | Judge cost per month |
|---|
| A. Uniform 2% (the policy before T09) | 20 | $0.70 | 0.00083 | 0.11 | $840 | $2,366 |
| B. Stratified: every risk-stratum call (4%) plus 2% of the rest | 59.2 | $2.07 | 0.042 | 5.4 | about $49.73 | $7,003.36 |
| C. Every call | 1,000 | $35.00 | 0.042 | 5.4 | $840 | $118,300 |
What the table tells a staff engineer:
- A uniform sample pays a fixed price per detection. Policies A and C differ 50-fold in spend and cost exactly the same per threat found, because a uniform sample finds violations in proportion to what it spends. Only stratifying moves the ratio, here by about 17-fold.
- Policy C is ruled out by price before anything else. Its judge line alone is about 2.4 times the whole monthly cap.
- Judge recall scales every column by the same factor. J1 v1 had recall 0.833 in its Spanglish validation and is advisory, with its flags routed to human audit; that shrinks detections under all three policies alike, so the order does not change.
- Credit only marginal detections. A line earns credit only for violations nothing cheaper already caught. Once "send someone out" is in the lexicon next to "police", "jail" and "arrest", the judge has to earn its cost on the paraphrases the lexicon misses.
Cost per detected violation decides how money moves between lines. It never decides whether a compliance line exists at all: a check that finds nothing still buys evidence that the rate is low. How large a sample that evidence needs, and how to reweight a stratified sample back to unbiased rates, belong to the monitoring chapter; the general method is in sampling strategies.
The biggest line in the plan, STT re-transcription on 5% of connects, buys entity error rate, not violations. Price it by the interval width it buys on each language group and segment, not by detections.
What gate would have caught it?
A load gate before every release: P95 platform turn gap on tool turns and amount entity error rate, measured at 1.0x and 1.5x of today's projected peak (2,400 and 3,600 concurrent calls), plus the 3x spike for refused connections. T04 fails the first part at 1.0x, since its knee sits near 1,800.
The four parts:
- Tool-turn latency. At 2,400 and at 3,600 concurrent calls, the upper bound of the call-clustered 95% bootstrap interval on P95 tool-turn gap is at or under each campaign's SLO (2,600 ms for AUTO-early-nov, 2,800 ms for SDL-biwk-nov).
- Amount entity error rate. Compare the same replayed calls at 240 against 2,400 and 3,600 with a call-clustered paired bootstrap. Pass only when the one-sided 95% upper bound on the increase is at or below the allowed increase the memo wrote down before the run. Report it per language group, because Spanish-dominant and Spanglish amount turns are where a late final transcript costs most.
- Duplicate replies. T04's duplicate-response check runs on every replayed call and reports a rate per step.
- The spike to 7,200. Report refused connections as a single binary proportion with a Wilson interval 8. Pass when every refused call goes back to the dialer queue as unconnected, never as a half-open call, and P95 returns to its 1.0x value once the spike ends.
The gate also runs again after any change to a dependency's limits: a provider that lowers a rate limit moves the knee without a single change to Hazel. The capstone plans it again at 6,000 concurrent calls, the projected peak after next quarter's expansion.
What goes in the decision memo?
An eval budget memo: run the baseline plan at about $45,673.03 a month under the $50,000 cap, add the load gate, and write down in advance the order in which lines get cut, with compliance coverage kept out of that order.
Decision. Run the baseline plan at about $45,673.03 a month against the $50,000 cap, plus the load gate at 1.0x and 1.5x of 2,400. Compliance coverage is never cut to save money: deterministic checks and the lexicon stay on 100% of calls, and every risk-stratum call stays judged.
Context. T04 (a shared-load tail, knee near 1,800, invisible at 50 calls) and T09 (a threat path that a 2% uniform judge sample had little chance of seeing).
Options considered. (a) Uniform 2% judging: $2,366 a month, sees 2% of threat calls. (b) The stratified plan: chosen. (c) Judge every call: $118,300, 2.4x the cap. (d) Swap J2 for a raw-audio judge: $21,010.08 more, $66,683.11 in total, over the cap.
Evidence. Cost per threat reaching a judge: about $49.73 stratified against $840 uniform, at any sample rate. Steelyard's invented prices. Load gate on gold-v7 replayed calls, P95 with call-clustered intervals.
Owner and cadence. Staff engineer, voice platform; reviewed monthly with finance; load gate twice a month ($5,000).
Tripwires. Connects above about 158,000 a day (the volume lines push the plan past the cap at about 1.22x today's volume). Any unit-price change. A judge for a new language. The knee falling below 1.5x of peak. If the plan has to shrink, cut in this order: the STT re-transcription share (it buys interval width, not violations found), then per-commit text runs, then simulation k.
Dated facts. LiveKit and AssemblyAI load-test guidance and Retell prices, fetched 2026-11-20.
What to do this week
Pull last month's hardest call types from your traces (tool turns, amount turns, each language group) and build the ramp, hold and spike plan around today's projected peak, with the stop conditions written down first. Then price every eval line with your own unit costs.
- Add per-dependency P95 spans on one clock (CRM per tool, TTFT, STT final delay, TTS TTFB), and log 429s and retries.
- Move the load generator to open-loop arrivals at the rate your peak implies.
- Ask every provider, in writing, what happens above its concurrency limit: refuse, queue, degrade, or bill.
- Write the cut order into the budget memo before the month you need it, and mark the compliance lines as not cuttable.
- Run the load ramp recipe, then read cost and latency evaluation and online versus offline evaluation.
NOTE
What will date on this page. LiveKit's ramp-and-sustain advice and AssemblyAI's four measurements and 3x spike (practitioner blogs, fetched 2026-11-20). Retell's testing prices, AI QA allowance and Concurrency Burst add-on (vendor docs and pricing page, undated, fetched 2026-11-20; vendor prices change often).