On 2026-11-13 at 17:52, in the evening peak, Asha left a borrower 3,990 ms of silence on a tool turn, then, after his "hello? hello?", replied twice, the second reply over the first (trace T04). The failure appears only above about 1,800 concurrent calls, so a load test at 50 concurrent calls never sees it. This chapter covers what a voice load test measures and how to run one, then what the evaluation program costs and how to spend a fixed budget on it.
What happened at peak?
The borrower asked for his total. Asha ran two sequential CRM reads while every call on the floor hit the same CRM, after a cold-prefix first token, so the turn gap reached 3,990 ms; his "hello? hello?" then started a second reply that talked over the first.
Excerpt from T04:
t_ms ch event / text
40210 caller vad_end REF: "kitna baaki hai total?"
40590 agent eot_commit (end-of-turn delay 380 ms)
40600 agent llm request (orchestration hop 10 ms); first token 41820 (TTFT 1,220 ms: cold prefix, account block placed before the policy text)
41830 agent tool get_dues (hop 10 ms), end 42690 (860 ms; CRM P95 under load)
42700 agent tool check_payment_status (hop 10 ms), end 43950 (1,250 ms; sequential, not parallel)
43200 caller REF: "hello? hello?" (2,990 ms of dead air so far)
43580 agent eot_commit on "hello hello" -> a second LLM request starts
43960 agent tts request (hop 10 ms)
44150 agent TTS first byte (TTFB 190 ms) -> play_start 44200 (playout 50 ms; turn gap 3,990 ms)
44900 agent the second reply starts over the first: "Ji, main yahin hoon. Aapka total..."
Above about 1,800 concurrent calls the CRM's P95 doubles and the LLM's first-token tail widens at the same time. Today's projected peak is 2,400, well past that point; 50 calls is about 2% of it.
Diagnosis: a shared-load failure. Two dependencies degraded together, their tails lined up on one turn, and no test ran at a load where either degrades.
What should a voice load test measure?
Replay your hardest real calls at projected peak concurrency and measure four things: entity accuracy, P95 latency, rejected connections and cost per completed task 1. For a voice agent: amount entity error rate, P95 platform turn gap by turn type, refused calls, and total cost over correctly completed tasks.
Hardest calls, not the happy path. Asha's dangerous turns are tool turns with two sequential CRM calls, amount and date turns, code-switched callers and noisy lines. Replay those through the real phone path with a simulated caller, so STT and the turn detector see production audio; the simulation chapter covers building such callers.
Entity accuracy, because load changes what the agent hears. As STT streams fill, the final transcript arrives later and an orchestrator with a wait cap acts on the partial; a duplicate reply can land on an amount turn. Replayed calls carry known amounts, so score them with entity error rate.
P95 by turn type. Report the platform turn gap on tool turns separately from greetings and readbacks, never a mean. The vocabulary (end-of-turn delay, LLM TTFT, tool time, TTS TTFB, playout) is defined in the latency chapter. Put the interval on the percentile with a bootstrap that resamples whole calls: turns in one call share that call's load and are not independent draws 2.
Rejected connections, from the counters that see them. A refused call never writes a trace, so count rejections from the trunk, the dialer and each provider.
Cost per completed task, not per call. Asha's LLM tokens cost Rs 0.315 per connected call at Tarazu's invented prices (Rs 20 per million input tokens, Rs 5 per million cached input tokens, Rs 80 per million output tokens): 10,000 uncached input tokens (Rs 0.20), 15,000 cached (Rs 0.075) and 500 output (Rs 0.04), since 60% of the 25,000 input tokens are cached. Under load that cost barely moves, but the denominator does. If one call in six ends with a wrong amount or a duplicate reply, cost per completed task rises by a fifth (1 / (5/6) = 1.2) with no price change.
Arrivals by rate, not by fixed concurrency. By Little's law, concurrency equals arrival rate times mean call length 3. At 150 s a call, 2,400 concurrent calls need 16 call starts a second. Schedule arrivals at that rate (open loop), so concurrency climbs above target when calls slow down, as on a real dialer. A generator that holds exactly 2,400 calls (closed loop) starts fewer calls as they slow, and hides the overload it should measure.
Why ramp instead of burst?
A zero-to-full burst tests "cold-start behavior under artificial conditions, not steady-state performance" (LiveKit) 4. So ramp in steps, hold each one until it settles, and measure the second half. Real bursts still happen, so run a separate spike test, up to 3x the busiest hour 1.
Tarazu's plan, from its volumes: 15 steps of 240 calls (10% of the 2,400 peak) up to 3,600 (1.5x), five minutes each; a 10-minute hold at 2,400; then a 3-minute spike to 7,200 (3x), which needs 48 call starts a second. The knee is the concurrency where P95 stops being flat; operationally, the last step whose upper interval bound stays under the SLO. For TW-early-nov the tool-turn SLO is the 400 ms silence budget plus 2,200 ms, so 2,600 ms.
Figure: Synthetic Tarazu data from the stack model in the load ramp recipe; only the plotted steps are shown. Top: the test plan. Bottom: P95 tool-turn gap per step. Flat to 1,200 calls, it crosses the 2,600 ms SLO between the 1,680 and 1,920 steps and passes the stop condition (twice the SLO) by 1.3x. The spike measures admission and rejections; its latency is not a capacity number.
Stop conditions, written before the run. Past the knee, each step spends trunk minutes and quota on a system already known to be over its limit. The load ramp recipe stops on P95 tool-turn gap above twice the SLO, more than 1% of connections rejected outside the spike, or more than 0.5% of calls dropped after exhausted LLM retries. The load generator applies them, not a person at a dashboard.
What the spike is for. Admission: which pool fills first, what happens to calls above it, and how long P95 takes to recover. Ask each provider what it does above its limit first, because the answer may arrive as rejections or as a bill.
CAUTION
Vendor-reported. Treat it as a hypothesis to test on your own calls.
Retell's pricing page, for example, lists an optional Concurrency Burst add-on, at $0.10 a minute extra, under which calls above the concurrency limit connect instead of being rejected (vendor-reported, as of November 2026) 5.
Keep the cold burst, separately. A burst from idle with cold prefix caches happens after a deploy or a failover. Test it on its own; never size capacity from it.
Which dependencies fail first?
Usually the ones whose tails rise with concurrency and whose limits you do not control: the CRM, STT streaming concurrency, the TTS rate limit, the LLM's in-flight limit and first-token tail, and the SIP trunk. Record each one's P95 from spans on one clock at every step; the first to bend sets the knee.
| Dependency | What it does under load | Record per step | In T04 |
|---|
| CRM | Queues; tool time rises with concurrent calls | Tool time P95 per tool | get_dues 860 ms, check_payment_status 1,250 ms, sequential |
| LLM provider | 429s above its in-flight limit, retries with backoff, a wider TTFT tail, prefix-cache misses | TTFT P95, 429 and retry rate, cached-token share | TTFT 1,220 ms on a cold prefix |
| STT provider | Refuses streams above its concurrency limit; final delay grows | STT final delay P95, stream refusals, share of turns acted on a partial | Not the cause; it is the entity-accuracy risk |
| TTS provider | Rate limits, queued requests, a TTFB tail | TTS TTFB P95, 429s | 190 ms, inside budget |
| SIP trunk and carrier | Channel exhaustion, call-setup rate caps | Failed call setups, rejected connections | Not implicated |
Three misreadings recur.
Summing stage P95s. When stage delays are independent, adding up the stage P95s overstates the turn's P95. Under shared load, as in T04, the CRM and LLM tails rise together, line up, and the sum comes close to the total. Measure the total from spans and use the per-stage numbers to assign the fix. The latency chapter owns that rule.
Fixing the loudest dependency, not the binding one. Neither the CRM nor the LLM tail alone broke the SLO; together, on a turn with two sequential CRM calls, they did. The cheapest fix may touch neither provider: run the two reads in parallel and start the filler on time.
Forgetting the eval pipeline. STT re-transcription and judges draw on the same provider quotas at peak. Score off the call path and include the eval workers in the test; the judges chapter covers async scoring.
What does an eval cost per call?
In Tarazu's invented price list, a deterministic check costs Rs 0.002 a call, a text judge Rs 0.15 per dimension, the blueprint judge Rs 0.40, batch STT re-transcription Rs 1.50 and a human audit about Rs 62.50. The 31,250x spread, not the absolute prices, decides what can run on every call.
These are prices for the fictional world, not any vendor's list price.
| Item | Unit cost | Basis |
|---|
| Deterministic checks and lexicon | Rs 0.002 per call | Code on the trace |
| Text judge (J1, J3) | about Rs 0.15 per judged call per dimension | about 6,000 input and 300 output tokens at Rs 20 and Rs 80 per million |
| Blueprint judge (J2) | Rs 0.40 per judged call | Feature extraction plus a text judge |
| Raw-audio judge (comparison only) | Rs 2.50 per judged call | Not in the plan |
| STT re-transcription, both channels | Rs 0.60 per call-minute, Rs 1.50 per 150 s call | Batch |
| Simulated call | Rs 6 per audio call; Rs 0.50 per text scenario run | Caller model, TTS, STT, trunk minutes |
| Human audit | Rs 25 per call-minute, about Rs 62.50 per call | Bilingual reviewer at about Rs 600 an hour, 2.5x real time |
| Load test | Rs 50,000 per pre-release run | Trunk minutes, provider quota, people |
Two comparisons keep the numbers honest. The three judges together cost Rs 0.70 per judged call, about 2.2 times the Rs 0.315 of Asha's own LLM tokens on that call, so judging every call costs more than running the agent. And Asha's LLM bill is about Rs 40,950 a day, Rs 10,64,700 over 26 calling days: the Rs 10 lakh eval cap is about the size of the agent's token bill.
Vendor platforms bill testing the same way, per message and per minute.
Retell's docs (vendor-reported, US dollars) say testing bills at the same rates as production: text tests (Playground, simulation, batch) cost roughly $0.001 to $0.05 per message depending on the model, with a simulated run billing both the agent's model and the simulated user's model and each batch case adding one post-call-analysis unit for grading; web call tests cost roughly $0.07 to $0.31 per minute; and phone tests add about $0.015 per minute of telephony for a Retell number 6. AI QA on production calls is free for the first 100 minutes of analyzed call time per workspace, then $0.10 per analyzed minute 7. Only simulation testing grades a run; web and phone calls give real audio but no pass or fail.
So when comparing quotes, price the verdict separately from the audio.
How do you budget judges when the agent burns billions of tokens a day?
Compare judge spend with what it finds, not with the agent's token bill. Asha reads about 3.25 billion input tokens a day; Tarazu's judges read about 92 million. Price each sampling policy, count the violations each one puts in front of a judge, and divide: cost per detected violation is the efficiency metric.
The 35-to-1 token ratio makes judge spend look trivial. It says nothing about whether judge tokens land where violations live, and per judged call the judges cost more than the agent.
Worked example: three policies per 1,000 connected calls. The violation is T09's threat on the angry path. In Tarazu's world rates the angry path is 1 in 600 connected calls and the threat rate is about 1 in 24,000 connects, all of them on the angry path. So 1,000 connects carry 0.042 threat calls, about 5.4 a day at 1,30,000 connects. Deterministic checks run on 100% under every policy (Rs 2 per 1,000), so the table prices the judges, at Rs 0.70 per judged call for J1, J2 and J3.
| Policy | Judged per 1,000 | Judge cost per 1,000 | Threat calls reaching a judge per 1,000 | Per day | Cost per threat reaching a judge | Judges per month |
|---|
| A. Uniform 2% (before T09) | 20 | Rs 14.00 | 0.00083 | 0.11 | Rs 16,800 | Rs 47,320 |
| B. Stratified: all risk strata (4%) plus 2% of the rest | 59.2 | Rs 41.44 | 0.042 | 5.4 | about Rs 995 | Rs 1,40,067 |
| C. Every call | 1,000 | Rs 700.00 | 0.042 | 5.4 | Rs 16,800 | Rs 23,66,000 |
What a staff engineer reads from it:
- Uniform sampling has a fixed cost per detection. A and C differ 50-fold in spend and match per threat found, because a uniform sample finds violations in proportion to what it pays. Only stratifying changes the ratio, here about 17-fold.
- Policy C fails on price alone. Its judge line is 2.4 times the whole monthly cap.
- Recall multiplies every column equally. J1 v1 (recall 0.833 in its Hinglish validation, advisory, flags routed to human audit) shrinks detections under all three policies alike; the ranking holds.
- Count marginal detections. Credit a line only with violations no cheaper line caught. Once the lexicon has "ghar aa", the judge earns its cost on the paraphrases the lexicon misses.
Cost per detected violation allocates money between lines; it never decides whether a compliance line exists, because a check that finds nothing still buys evidence that the rate is low. Sizing the sample for that evidence, and reweighting a stratified sample into unbiased rates, belongs to the monitoring chapter. For the general method, see sampling strategies.
The largest line, STT re-transcription on 5% of connects, buys entity error rate, not violations: price it by the interval width it buys per segment.
What gate would have caught it?
A load gate before each release: P95 platform turn gap on tool turns and amount entity error rate at 1.0x and 1.5x of today's projected peak (2,400 and 3,600 concurrent calls), plus the 3x spike for rejected connections. T04 fails the first part at 1.0x, because its knee sits near 1,800.
The parts:
- Tool-turn latency. At 2,400 and 3,600, the upper bound of the call-clustered 95% bootstrap interval on P95 tool-turn gap is at or below each campaign's SLO (2,600 ms for TW-early-nov, 2,800 ms for MFI-fort-nov).
- Amount entity error rate. Compare the same replayed calls at 240 against 2,400 and 3,600 with a call-clustered paired bootstrap; pass only when the one-sided 95% upper bound on the increase is at or below the allowed increase written in the memo before the run.
- Duplicate replies. The duplicate-response check from T04 runs on every replayed call and reports a rate per step.
- The spike at 7,200. Report rejected connections as a single binary proportion with a Wilson interval 8. Pass when every refused call goes back to the dialer queue as unconnected, never as a half-open call, and P95 returns to its 1.0x value after the spike ends.
The gate also re-runs after any change to a dependency's limits: a provider that lowers a rate limit moves the knee with no change to Asha. The capstone re-plans it at 6,000 concurrent calls, the projected peak after next quarter's expansion.
What goes in the decision memo?
An eval budget memo: run the baseline plan at about Rs 9,13,460 a month under the Rs 10 lakh cap, add the load gate, and write the cut order in advance, with compliance coverage outside it.
Decision. Run the baseline plan at about Rs 9,13,460 a month against the Rs 10 lakh cap, plus the load gate at 1.0x and 1.5x of 2,400. Compliance coverage is never cut to save money: deterministic checks and the lexicon stay on 100%, and every risk-stratum call stays judged.
Context. T04 (shared-load tail, knee near 1,800, invisible at 50 calls) and T09 (a threat path a 2% uniform judge sample was unlikely to see).
Options considered. (a) Uniform 2% judging: Rs 47,320 a month, sees 2% of threat calls. (b) Stratified plan: chosen. (c) Judge every call: Rs 23,66,000, 2.4x the cap. (d) Replace J2 with a raw-audio judge: Rs 4,20,202 more, total Rs 13,33,662, over the cap.
Evidence. Cost per threat reaching a judge: about Rs 995 stratified against Rs 16,800 uniform at any rate. Invented W20 prices. Load gate on gold-v7 replayed calls, P95 with call-clustered intervals.
Owner and cadence. Staff engineer, voice platform; reviewed monthly with finance; load gate twice a month (Rs 1,00,000).
Tripwires. Connects above about 1,58,000 a day (the plan breaches the cap at 1.22x volume). Any unit-price change. A new language judge. The knee moving below 1.5x of peak. If the plan must shrink, cut in this order: the STT re-transcription share (it buys interval width, not violations found), per-commit text runs, simulation k.
Dated facts. LiveKit and AssemblyAI load-test guidance and Retell prices, fetched 2026-11-20.
What to do this week
Pull your hardest call types from last month's traces and build the ramp, hold and spike plan around today's projected peak, with stop conditions written down. Then price your eval lines with your own unit costs.
- Add per-dependency P95 spans (CRM per tool, TTFT, STT final delay, TTS TTFB) on one clock, and record 429s and retries.
- Switch the load generator to open-loop arrivals at the rate your peak implies.
- Ask every provider, in writing, what happens above its concurrency limit: reject, queue, degrade or bill.
- Write the cut order into the budget memo before the month you need it, with compliance lines marked as not cuttable.
- Run the load ramp recipe, then read cost and latency evaluation and online versus offline evaluation.
NOTE
What will date on this page. LiveKit's ramp-and-sustain advice and AssemblyAI's four measurements and 3x spike (practitioner blogs, fetched 2026-11-20). Retell's testing prices, AI QA allowance and Concurrency Burst add-on (vendor docs and pricing page, undated, fetched 2026-11-20; vendor prices change often).