At 17:52 on 13 November 2026, under peak load, a borrower on Steelyard's auto book asked Hazel what he owed in total and got 3,990 ms of silence back; he said "hello? hello?", and Hazel answered twice, her second reply starting on top of the first (trace T04). This chapter fixes the one latency vocabulary the section uses, from end-of-turn delay through LLM time to first token (TTFT) and TTS time to first byte (TTFB) to the turn gap, and shows how to trace it, budget it as P95 service-level objectives (SLOs) and block a release on it.
What happened on the call?
One tool turn combined a cold-prefix first token, two CRM lookups run one after the other and no filler line, and the result was 3,990 ms of silence; the caller's "hello?" was then committed as a fresh turn and produced a second reply over the first.
t_ms ch event / text
40210 caller vad_end REF: "how much do I owe in total?"
40590 agent eot_commit (end-of-turn delay 380 ms)
40600 agent llm request (orchestration hop 10 ms); first token 41820 (TTFT 1,220 ms: cold prefix, account block placed before the policy text)
41830 agent tool get_dues (hop 10 ms), end 42690 (860 ms; CRM P95 under load)
42700 agent tool check_payment_status (hop 10 ms), end 43950 (1,250 ms; sequential, not parallel)
43200 caller REF: "hello? hello?" (2,990 ms of dead air so far)
43580 agent eot_commit on "hello hello" -> a second LLM request starts
43960 agent tts request (hop 10 ms)
44150 agent TTS first byte (TTFB 190 ms) -> play_start 44200 (playout 50 ms; turn gap 3,990 ms)
44900 agent the second reply starts over the first: "Yes, I'm here. Your total is..."
Diagnosis: the stages sum to 380 + 1,220 + 860 + 1,250 + 190 + 50 = 3,950 ms, and four 10 ms orchestration hops make it 3,990 ms. AUTO-early-nov was still on its 300 ms silence budget that week, which put the tool-turn SLO at 2,500 ms and the filler deadline at 1,100 ms. The turn breaks only above about 1,800 concurrent calls, where the CRM P95 doubles while the LLM first-token tail widens on the same turns. A reviewer reading the transcript sees the doubled reply; the gap itself is visible only on the timeline.
Figure 1: Hazel's T04 tool turn as a waterfall. Synthetic Steelyard data. The two CRM lookups (2,110 ms together) ran in sequence and outlasted the LLM, TTS and playout stages combined. Nothing played at the 1,100 ms filler deadline, and the turn was already past its 2,500 ms SLO when the borrower spoke into the silence at 2,990 ms. The thin gray bars are the 10 ms orchestration hops.
What does latency mean on a voice call?
This section uses nine named intervals and leads with the platform turn gap: from caller speech end (the VAD end, back-dated by the hangover) to the first agent audio frame at media egress, computed per turn from spans. Mouth-to-ear latency adds the carrier legs and comes from the dual-channel recording.
| Term | Starts | Ends | Notes |
|---|
| End-of-turn delay | Caller speech end (VAD end, back-dated by the VAD hangover) | eot_commit | Includes the silence budget and detector wait |
| STT final delay | Caller speech end | Final transcript segment | Overlaps end-of-turn delay |
| LLM TTFT | LLM request sent | First token of any type | Also report time to first answer token for reasoning models |
| Tool time | Tool call start | Tool result | Sequential vs parallel matters (T04) |
| TTS TTFB | First text chunk sent to TTS | First audio byte back | TTFB on these pages always means this |
| Playout delay | First audio byte | First audio frame at media egress | Buffering and 160-byte framing |
| Orchestration | One stage hands off | The next stage starts | 10 ms per hop in Steelyard's stack; a tool turn with two tool calls has four hops (T04) |
| Turn gap (platform) | Caller speech end | Agent audio starts at media egress | The headline per-turn number, from spans |
| Turn gap (mouth-to-ear) | Caller stops speaking at the handset | Agent audio reaches the caller | Measured on the dual-channel recording; adds the carrier legs |
Every turn carries one of seven types: greeting, verification, amount-quote, readback, tool, objection, closing.
The frameworks and testing tools you will wire in each attach the word "latency" to a different interval. Translate every field into the terms above before you put two of them on one chart:
| Source | Field or metric | What the docs say it measures | Our term |
|---|
| LiveKit Agents | end_of_turn_delay, transcription_delay | Speech end to the end-of-turn decision; speech end to the transcript | End-of-turn delay; STT final delay |
| LiveKit Agents | llm_node_ttft, tts_node_ttfb | One value per stage; empty rather than zero when the trace is speech-to-speech | LLM TTFT; TTS TTFB |
| LiveKit Agents | e2e_latency | From the user stopping speaking to the agent starting its response | Platform turn gap, by our reading: it starts at the agent's own end-of-speech detection, so no carrier leg is inside it |
| Pipecat | TTFB per service, TTFAT | First byte from any service, the LLM included; first answer token | TTS TTFB only when filtered to the TTS service; first answer token |
| Deepgram Voice Agent API | total_latency, ttt_token_latency | End of the user's utterance to the first audio byte; first token of any type | Our mapping: a server-side gap that stops at the first audio byte, before playout; LLM TTFT |
| Cekura | Latency, P25 to P99 | VAD run over the stereo recording | Recording-side turn gap |
| Coval | Average silence gap | A mean over gaps of 0.5 s or longer only | No equivalent: it is a mean, and it discards the short gaps |
| LiveKit audio simulations | Caller-heard latency, p50 to p99 | A negative value means the agent spoke over the caller | Signed recording-side turn gap |
We read every row on 2026-11-20: LiveKit's data hooks 1, Pipecat's metrics page 2, Deepgram's latency report 3, Cekura's metric list 4, Coval's statistical metrics 5 and LiveKit's simulations page 6. Watch two of them. A Pipecat "TTFB" column means TTS TTFB only after you filter it to the TTS service, and with report_only_initial_ttfb=True it keeps just the first TTFB of each service, so a P95 over it is not a per-turn P95. Deepgram's docs do not state whether total_latency is the sum of its stage fields, and a number measured on the caller's side can disagree with this server-side one.
The thresholds disagree as much as the definitions do. Hamming's methodology page (self-reported, read 2026-11-20) calls a P50 turn latency under 1,600 ms excellent and under 2,000 ms acceptable, but never says where turn latency starts and stops 7; Cekura's docs (vendor-reported, read 2026-11-20) describe anything under 2,000 ms as generally good 4. Neither number can be held against a platform turn gap until you know both of its endpoints.
CAUTION
Vendor-reported. Treat it as a hypothesis to test on your own calls.
How do the stages add up?
For a cascaded turn, platform turn gap = end-of-turn delay + LLM TTFT + tool time + TTS TTFB + playout delay + orchestration hops. STT final delay sits inside end-of-turn delay, so it is reported and never added. Each stage is a child span of the turn span, and the children have to sum to the parent.
A Hugging Face community playbook from March 2026 uses the same definition of perceived latency (end-of-utterance delay plus LLM TTFT plus TTS TTFB), recorded as nested Langfuse spans per turn. On 23 non-tool turns of a small open stack it measured a P50 of about 1.4 s and a P95 of about 3.4 s, and LLM TTFT had much the widest spread of any stage (P50 566 ms, P95 2,246 ms) 8. Hazel's turns add tool spans, playout and the hops to that shape. Here is T04 as a span tree:
turn (tool) 40,210 -> 44,200 3,990 platform turn gap
end_of_turn 40,210 -> 40,590 380 vad_end to eot_commit
hop 40,590 -> 40,600 10
llm.first_token 40,600 -> 41,820 1,220 request to first token
hop 41,820 -> 41,830 10
tool.get_dues 41,830 -> 42,690 860
hop 42,690 -> 42,700 10
tool.check_payment_status 42,700 -> 43,950 1,250
hop 43,950 -> 43,960 10
tts.first_byte 43,960 -> 44,150 190 request to first audio byte
playout 44,150 -> 44,200 50 first byte to media egress
children + hops 3,990 = parent, residual 0
Write the residual (parent minus the sum of its children) onto every turn and alert when it grows, because time that no span claims is a queue nobody has instrumented yet. Span nesting for text agents is covered in the trace schema; the voice fields and the shared media clock are in the trace chapter. With a reasoning model, also record the first answer token, since the first token may be a thinking token and TTFT alone then makes the wait look shorter than it was.
Why can you not add stage P95s?
Percentiles do not add. With independent stages the slow tails seldom hit the same turn, so summed stage P95s overstate the turn-gap P95. When stages slow down together under shared load, as in T04, the sum comes close. When each stage has a rare spike, the P95 of the total can exceed the sum.
A teaching model makes the first case concrete. It is a generic model, not Steelyard's SLOs: three independent lognormal stages, with end-of-turn delay at P50 450 ms and P95 800 ms, LLM TTFT at 566 and 2,246 ms (the spread from the playbook) and TTS TTFB at 250 and 450 ms. The total comes out at a P50 of about 1.34 s and a P95 of about 3.0 s, yet the three stage P95s add to 3.5 s. Make the stages perfectly correlated, so that a slow turn is slow in every stage, and the P95 of the total equals the sum.
Now the opposite error. Suppose two stages each take 100 ms but carry a 4% chance of an extra 2,000 ms. Each stage P95 is then 100 ms, and the two add to 200 ms. The chance that a turn misses both spikes is 0.96 x 0.96 = 0.9216, which is under 0.95, so the P95 of the total is 2,100 ms: over ten times the sum of the parts. Cold starts, retries and waits for a pooled connection all look like this, rare inside any one stage and frequent across a whole turn.
T04 belongs to the shared-load case. Past about 1,800 concurrent calls the CRM tail and the LLM tail widen on the same turns, so they line up. The working rule follows: measure the total straight from the spans, set the budget on the total, and only then hand out stage budgets as diagnostics. A stage SLO tells you where to look; only the turn-gap SLO tells you whether the borrower waited too long. How load creates these effects is the subject of load and cost.
What is fast enough?
The target comes from your own calls: chart early hang-ups, "are you there?" re-prompts and talk-over against the turn gap and find where they bend upward. Published numbers only bound the range: people leave a most common gap of about 200 ms, and 800 ms and 1.5 s are rules of thumb, not measurements.
The research numbers describe people talking to people. Across languages, the most common gap between turns is about 200 ms 9. A study of answers to yes/no questions in 10 languages found a mean response offset of 208 ms and a median of 100 ms, counting overlaps as negative offsets, with language means running from 7 ms in Japanese to 469 ms in Danish 10. Practitioner guides tend to call gaps beyond roughly 800 ms awkward and beyond about 1.5 s broken, and they are rules of thumb, not measurements 11 12.
Twilio's guide from November 2025 sets median launch targets of 1,115 ms (upper limit 1,400 ms) for the mouth-to-ear turn gap and 885 ms (upper limit 1,100 ms) for the platform turn gap, the second leaving out the internet or PSTN leg. Per component it targets STT at 350 ms, LLM TTFT at 375 ms and TTS TTFB at 100 ms (upper limits 500, 750 and 250 ms), and it names end-of-turn detection as the usual long pole, because systems are tuned to avoid false positives 13. These are a vendor's targets, not measurements. The 230 ms that separates the two medians is our own subtraction, which we read as the network's share at target; Twilio does not give it.
For Steelyard this has two consequences. The silence budget sets a floor under every turn gap: AUTO's 400 ms is already twice the human modal gap before the LLM has received a single token, so an agent that endpoints on silence cannot keep human timing, and every millisecond taken off the budget is paid for in false cutoffs (the curve belongs to turn-taking and barge-in). And because a filled pause and dead air feel different to the borrower, Hazel's filler line ("One second, I'm checking that.") has its own deadline, the silence budget + 800 ms, gated apart from the answer.
To find your own target, put platform turn gaps into 250 ms bins for each turn type and plot three rates per bin: turns followed by a re-prompt ("hello?", "are you still there?", or on a Spanish call "hola? sigue ahi?" [hello? are you still there?]), turns where the caller starts speaking inside the gap, and calls that end before the reply finishes. Plot each language group (English, Spanish-dominant, Spanglish) on its own, because human gap norms vary by language. The bend in those rates is the target, and it needs re-plotting after every change to a silence budget.
Failing turns are a minority, and a median over the whole session averages them out of sight. Tool turns carry the CRM round trips and usually a longer LLM call, so the tail is theirs. Report P50 and P95 for each turn type against that type's own SLO, and let the worst type decide the gate.
The worked example uses 200 synthetic turns: 25 AUTO-early-nov calls of 8 Hazel turns each at today's 400 ms silence budget, with two tool turns per call and two sequential CRM calls in each of those. The same script also runs the lognormal teaching model and a shared-load variant, in which a single shock on 12% of turns doubles the LLM time and both tool calls.
"""Turn-gap percentiles per stage and per turn type from 200 synthetic
turns, the sum-of-stage-P95s check, and a gate that resamples whole calls.
Standard library only. Synthetic Steelyard data."""
import math
import random
rng = random.Random(20261120)
Z95 = 1.6449
def pct(xs, p):
s = sorted(xs)
k = (len(s) - 1) * p
f = math.floor(k)
return s[f] + (s[min(f + 1, len(s) - 1)] - s[f]) * (k - f)
def logn(p50, p95):
return p50 * math.exp(rng.gauss(0, 1) * math.log(p95 / p50) / Z95)
# Part 1. Generic teaching model: three independent lognormal stages,
# each given as (P50, P95) in ms. Not Steelyard's SLOs.
STAGES = {"eot": (450, 800), "ttft": (566, 2246), "ttfb": (250, 450)}
tot = [sum(logn(*v) for v in STAGES.values()) for _ in range(200_000)]
print(f"model: total P50 {pct(tot, .5):,.0f} total P95 {pct(tot, .95):,.0f} "
f"sum of stage P95s {sum(v[1] for v in STAGES.values()):,}")
# Part 2. AUTO-early-nov: 25 calls x 8 Hazel turns = 200 turns at the
# 400 ms silence budget (600 ms after an amount-asked or date-asked turn).
LAYOUT = [("greeting", 0), ("verification", 0), ("amount-quote", 0), ("objection", 0),
("tool", 2), ("readback", 0), ("tool", 2), ("closing", 0)]
SLO = {"readback": 2100, "tool": 2600} # budget + 1,500 or + 2,200 (WG11)
def turn(ttype, n_tools, shock):
budget = 600 if ttype == "readback" else 400
eot = budget + logn(60, 140)
ttft = logn(520 if n_tools else 430, 900 if n_tools else 650) * shock
tools = sum(logn(330, 700) * shock for _ in range(n_tools))
ttfb, play = logn(170, 240), rng.uniform(40, 60)
hops = 10 * (2 + n_tools)
return {"type": ttype, "eot": eot, "ttft": ttft, "tool": tools, "ttfb": ttfb,
"play": play, "gap": eot + ttft + tools + ttfb + play + hops}
def calls(shared_load, n=25):
out = []
for _ in range(n):
out.append([turn(t, n, 2.0 if shared_load and rng.random() < 0.12 else 1.0)
for t, n in LAYOUT])
return out
def report(cs, label):
turns = [t for c in cs for t in c]
print(f"\n{label}: {len(turns)} turns; session median turn gap "
f"{pct([t['gap'] for t in turns], .5):,.0f}")
for ttype in dict(LAYOUT):
g = [t["gap"] for t in turns if t["type"] == ttype]
print(f" {ttype:<13} n={len(g):>3} P50 {pct(g, .5):>5,.0f} P95 {pct(g, .95):>5,.0f}"
f" SLO {SLO.get(ttype, 1900):,}")
tool = [t for t in turns if t["type"] == "tool"]
stage95 = {k: pct([t[k] for t in tool], .95) for k in ("eot", "ttft", "tool", "ttfb", "play")}
print(" tool-turn stage P95s: " + ", ".join(f"{k} {v:,.0f}" for k, v in stage95.items()))
print(f" sum of stage P95s + 40 ms hops {sum(stage95.values()) + 40:,.0f}"
f" vs tool-turn gap P95 {pct([t['gap'] for t in tool], .95):,.0f}")
return cs
def gate(cs, slo=2600, reps=2000):
boot = []
for _ in range(reps):
pick = [cs[rng.randrange(len(cs))] for _ in cs] # whole calls, with replacement
boot.append(pct([t["gap"] for c in pick for t in c if t["type"] == "tool"], .95))
lo, hi = pct(boot, .025), pct(boot, .975)
verdict = "BREACH" if lo > slo else "pass" if hi <= slo else "unclear"
point = pct([t["gap"] for c in cs for t in c if t["type"] == "tool"], .95)
print(f" gate: tool-turn P95 {point:,.0f}, 95% interval {lo:,.0f} to {hi:,.0f}, "
f"SLO {slo:,} -> {verdict}")
base = report(calls(False), "independent stages")
gate(base)
gate(report(calls(True), "shared load (one shock slows the LLM and both tools)"))
print("\nshared load, 200 calls (400 tool turns):")
gate(calls(True, n=200))
Under Python 3.11 the script prints:
model: total P50 1,337 total P95 3,034 sum of stage P95s 3,496
independent stages: 200 turns; session median turn gap 1,208
greeting n= 25 P50 1,130 P95 1,246 SLO 1,900
verification n= 25 P50 1,140 P95 1,416 SLO 1,900
amount-quote n= 25 P50 1,102 P95 1,296 SLO 1,900
objection n= 25 P50 1,117 P95 1,257 SLO 1,900
tool n= 50 P50 1,927 P95 2,478 SLO 2,600
readback n= 25 P50 1,360 P95 1,578 SLO 2,100
closing n= 25 P50 1,147 P95 1,364 SLO 1,900
tool-turn stage P95s: eot 519, ttft 805, tool 1,044, ttfb 239, play 59
sum of stage P95s + 40 ms hops 2,706 vs tool-turn gap P95 2,478
gate: tool-turn P95 2,478, 95% interval 2,242 to 2,659, SLO 2,600 -> unclear
shared load (one shock slows the LLM and both tools): 200 turns; session median turn gap 1,275
greeting n= 25 P50 1,144 P95 1,669 SLO 1,900
verification n= 25 P50 1,168 P95 1,612 SLO 1,900
amount-quote n= 25 P50 1,218 P95 1,417 SLO 1,900
objection n= 25 P50 1,172 P95 1,466 SLO 1,900
tool n= 50 P50 1,960 P95 2,928 SLO 2,600
readback n= 25 P50 1,333 P95 1,869 SLO 2,100
closing n= 25 P50 1,119 P95 1,380 SLO 1,900
tool-turn stage P95s: eot 534, ttft 919, tool 1,260, ttfb 235, play 59
sum of stage P95s + 40 ms hops 3,046 vs tool-turn gap P95 2,928
gate: tool-turn P95 2,928, 95% interval 2,433 to 3,282, SLO 2,600 -> unclear
shared load, 200 calls (400 tool turns):
gate: tool-turn P95 3,452, 95% interval 3,181 to 3,907, SLO 2,600 -> BREACH
Figure 2: P50 and P95 platform turn gap per turn type for 25 synthetic AUTO-early-nov calls of 8 Hazel turns at the 400 ms silence budget (seed 20261120). Synthetic Steelyard data. Tool turns make up a quarter of all turns and hold the whole tail, while the session median sits below every P95 and tells you nothing about them. Readback turns come after a caller turn in the amount-asked or date-asked state, so they run on the 600 ms budget and a 2,100 ms SLO.
Three readings from the output:
- A session median cannot see the tail. It is 1,208 ms. Five of the non-tool types have P95s below 1,420 ms against an SLO of 1,900, and readback sits at 1,578 against 2,100. Tool turns run at P50 1,927 and P95 2,478 against 2,600: one turn in four, and every bit of the risk.
- The stage sum is too pessimistic, by less under load. On the independent tool turns, stage P95s plus hops add to 2,706 ms while the measured P95 is 2,478, so the sum overshoots by 228 ms. Under shared load the overshoot falls to 118 ms (3,046 against 2,928), because the slow LLM and the slow CRM now hit the same turns.
- Fifty tool turns are too few to settle a P95. Both 25-call runs end unclear, with intervals that straddle 2,600 ms. With 200 shared-load calls the P95 is 3,452 ms and the entire interval lies above the SLO. Size the run before you trust a P95 (reliability statistics).
Then cut the tail by its likely causes: prompt length, number of tool calls, cache hit or miss, and call language. A P95 that rises only on cache misses points at the prompt prefix; one that rises only on two-tool turns points at missing parallelism.
How do you measure what the caller heard?
Run an energy VAD over each channel of the dual-channel recording, take the signed gap from caller speech end to agent audio onset (negative when the agent spoke over the caller), and match onsets to spans to recover the carrier offset. Spans give the platform gap; only the recording shows what the borrower heard.
The steps below are implemented in the latency waterfall recipe:
- Find speech on each channel separately. Use short frames and a fixed energy threshold, and bridge very short gaps. A mono mix cannot yield a signed gap.
- Solve for the offset. Let A be the median of (agent onset on the recording minus
play_start_ms), which equals the carrier offset plus the outbound leg, and B the median of (vad_start_ms minus caller onset on the recording), which equals the inbound leg minus the offset. Then offset = (A - B) / 2 and round trip = A + B, on the NTP assumption that both legs are equal. Work from VAD starts rather than ends, because the hangover never moves an onset.
- Take the signed gap. Agent audio onset minus the end of the caller's last speech, both read off the recording. LiveKit's audio simulations sign caller-heard latency the same way 6.
- Reconcile with the spans. On clean turns, the median of (recording gap minus span gap) should come out equal to the round trip; anything beyond that is time the spans cannot see.
Step 4 is where instrumentation bugs surface. Steelyard writes vad_end_ms back-dated: the raw VAD end fires one hangover later (200 ms in the trace schema), and the pipeline subtracts it before storing the field. A pipeline that skips the back-dating keeps the hangover outside every span-based number, so end-of-turn delay and the platform turn gap both read 200 ms better than the borrower's experience. Only the recording reveals it, as a step-4 residual the size of the hangover. The same reconciliation also finds turns where the borrower paused, the turn was committed, and the borrower kept talking: a negative recording gap next to a positive span gap marks a cutoff problem, not a speed problem.
Steelyard captures the recording on its own side of the carrier, so the access network and the handset are still missing from it. For true mouth-to-ear numbers, place scheduled test calls from real handsets on each of the three mobile carriers and the VoIP leg, record at the handset, and store each carrier's remainder with the date it was measured; it shifts whenever a carrier reroutes.
How do you set per-stage SLOs?
Set the turn-gap SLO per turn type and silence budget first, then allocate stage budgets under it. Write both as the silence budget plus an allowance (150 ms of detector time for end-of-turn delay, a fixed amount for the gap), because end-of-turn delay contains the silence budget, so a longer budget moves the SLO.
Steelyard's SLOs are team decisions, not industry thresholds:
| Stage | P95 budget |
|---|
| End-of-turn delay | Silence budget + 150 ms detector allowance, per campaign and call state: AUTO, PIL and POS 550 ms (750 ms in amount-asked and date-asked states); SBL 600 ms (800); SDL 750 ms (950). Before T05's fix AUTO ran a 300 ms budget, so its SLO was 450 ms; T01 (370 ms) and T04 (380 ms) were inside it |
| STT final delay | 300 ms |
| LLM TTFT | 700 ms non-tool turns, 900 ms tool turns |
| Tool time (parallel where possible) | 600 ms |
| TTS TTFB | 250 ms |
| Playout | 60 ms |
| Platform turn gap | Silence budget + 1,500 ms on non-tool turns and + 2,200 ms on tool turns: 1,900 and 2,600 ms at AUTO's 400 ms budget; SDL default 2,100 and 2,800 ms. Filler must start by the budget + 800 ms (1,200 ms at AUTO) |
Hold T04 against the SLOs that applied on 13 November (300 ms budget: 450, 900, 600, 250 and 60 ms for the stages, 2,500 ms for the tool-turn gap). End-of-turn delay, TTS TTFB and playout were within budget. LLM TTFT at 1,220 ms and tool time at 2,110 ms, run in sequence, were not. The stage table names the repairs, running the two lookups in parallel and fixing the prefix order; the gap SLO decides whether the repairs were enough.
Adding up the stage budgets does not give you an SLO. For an AUTO tool turn they come to 550 + 900 + 600 + 250 + 60 + 40 ms of hops = 2,400 ms, inside the 2,600 ms gap SLO. That is the shape an allocation should have, since even with every tail lined up by shared load, stages running at budget still fit. It proves nothing, though: a stage can meet its P95 while carrying rare spikes that push the turn beyond 2,600 ms, exactly as in the spike example earlier. Only the measured gap settles it.
Guardrails draw on the same budget. A synchronous check on the TTS input text, C13 for instance (the amount Hazel says must equal the get_dues result), runs between the first answer token and the TTS request, and when TTS is streamed one sentence at a time, each sentence has to clear the check before it is spoken. Every check you put inline spends milliseconds that TTS no longer has. Keep the deterministic text checks inline, run model judges asynchronously, and give each guardrail its own span so its time never disappears into "LLM" (guardrails versus evals; for latency columns in a general scorecard, cost and latency evaluation).
What does caching buy you?
Prefix caching lowers cost reliably and LLM TTFT by less, and less predictably. Put static policy and scripts first and the per-call account block last, check that the cached prefix clears the provider's minimum length, and compare TTFT on hits against misses: a cached-token count proves a hit, not a speed-up.
Response caching, which replays a stored answer for a fixed line instead of generating it, is a separate technique with a separate test (is the stored line still right in this call state?); the India edition examines one vendor's claim about it.
The independent evidence on agents is modest. A PwC preprint ran over 500 agent sessions with 10,000-token system prompts across four models: prompt caching cut cost by 41% to 80%, but TTFT improved by only 6% to 31%, and caching the full context made GPT-4o's TTFT 8.8% worse. The authors suggest that caching dynamic content can add overhead that cancels the latency benefit 14. Their workload was a research agent, not a voice agent. On the vendor side, Anthropic's 2024 launch post reported TTFT falling from 1.6 s to 1.1 s (-31%) on a 10,000-token prompt and from 11.5 s to 2.4 s on a 100,000-token one 15, and the current docs give no latency figure at all. The docs do say that a prompt under the model's minimum (512 to 4,096 tokens) is processed without caching and without any error 16. OpenAI's docs require the whole rendered prefix to match for reuse and set a 1,024-token minimum for GPT-5.6 and later 17. Our inference, not a measurement: a voice prompt of a few thousand tokens should expect gains around the 10,000-token row or smaller.
T04 is the failure worth designing against. On that call the account block sat ahead of the policy text, so the prefix changed on every call and every request ran cold; that turn's TTFT was 1,220 ms against a 900 ms SLO. The repair is the order of the blocks. Hazel also carries both an English and a Spanish script, and keeping both inside the static prefix means a borrower who switches into Spanish mid-call does not change the prefix (a design choice of ours, not a measured effect). Test the reorder as a paired comparison: run the same scenarios with both prompt orders and compare TTFT per turn type with an interval clustered by call. Since caching fails without a sound below the minimum, alert on cache-hit rate for each prompt version, using the cached_tokens field in the trace.
Does speech-to-speech remove the problem?
No. A speech-to-speech model removes the per-stage spans, so the trace has no LLM TTFT or TTS TTFB, and the platform turn gap plus mouth-to-ear from the recording are the only latency numbers left. At the June 2026 launch of a public speech-to-speech index, the fastest models were not the most capable.
When Artificial Analysis launched its Speech to Speech Index on 23 June 2026, Deepslate Opal had the fastest average time to first audio, 0.44 s, with a score of 62.1%, while GPT-Realtime-2 (High) took 2.33 s and scored 77.2%. The live leaderboard has moved since, and its timings come from a benchmark harness rather than a phone line 18. Everything else that changes when hazel-s2s-pilot replaces the cascade is covered in the platform chapter.
What gate would have caught it?
A gate on the P95 platform turn gap of tool turns over the replay set, with a bootstrap interval resampling whole calls, plus two deterministic checks on every call: duplicate responses and the filler deadline. BREACH blocks the release, unclear means more calls, and a Wilson interval is never used on a percentile.
- Tool-turn P95 against the SLO for its call state. Resample whole calls, because turns inside one call share a network path, a CRM shard and a prompt history 19. The verdict is BREACH when the lower bound is above the SLO, pass when the upper bound is at or below it, and unclear in between.
- Duplicate response. Flag any caller
eot_commit that arrives while an agent reply is still pending, between the previous commit and play_start_ms. In T04 the second commit at 43,580 fell inside the 40,590 to 44,200 window. The check needs only spans, runs on 100% of calls, and maps to failure code LAT.DUP.
- Filler deadline. On every turn predicted to need a tool, the filler's
play_start_ms must come no later than the silence budget + 800 ms after caller speech end, which is 1,200 ms on AUTO today. T04 had no filler at all.
The replay set exposes the cold prefix and the sequential lookups at any load, but the shared-load tail appears only near projected peak (2,400 concurrent calls today), so the same gate also runs inside the pre-release load test (load and cost). Daily P95 dashboards are part of production monitoring.
What goes in the decision memo?
The memo turns latency SLOs per turn type into a release rule: which numbers gate, which only diagnose, and what reopens the decision. Here is Steelyard's version, addressed to the engineering manager who signs Hazel's releases:
- Decision. Hazel releases gate on the P95 platform turn gap of each turn type. SLOs are stated as the campaign's silence budget + 1,500 ms for non-tool turns and + 2,200 ms for tool turns. Stage SLOs are diagnostic allocations and are never added up into a gate.
- Context. T04 on 13 November: a tool turn on AUTO-early-nov took 3,990 ms at peak and Hazel replied twice. In the synthetic study tool turns are one in four, so a session median or a single target for all turns hides them.
- Options considered. (1) One P95 target across all turn types: easy to run, and it passes while tool turns fail. (2) Gating on the sum of stage P95s: too pessimistic when tails are independent, and silent about shared load. (3) A P95 per turn type and silence budget, measured directly from spans. We chose (3).
- Evidence. The T04 span tree; the 200-turn synthetic study (seed 20261120), where the stage P95s summed 228 ms above the measured P95; release replays on gold-v7 scenarios with intervals that resample whole calls; a weekly reconciliation of recordings against spans.
- Owner and cadence. The voice platform team owns it; P95 per turn type is reviewed every week; SLOs are re-derived whenever any campaign changes its silence budget.
- Tripwires. AUTO tool-turn P95 above 2,600 ms for one day. The median recording-minus-span residual drifting away from the measured round trip. A fall in cache-hit rate after a prompt release.
- Dated facts. Twilio's launch targets (November 2025); the 800 ms and 1.5 s rules of thumb; the Speech to Speech Index launch figures (June 2026).
What to do this week
Begin with the vocabulary map and the span residual. If one field maps to the wrong interval, or one queue has no span, every P95 you compute afterwards is wrong.
- Translate each of your vendor's latency fields into the table above, and mark the fields with no match.
- Compute the span residual on 100 turns; every turn with unclaimed time points at a queue with no span.
- Pull P50 and P95 per turn type for one campaign, and put the tool-turn P95 next to the session median.
- Reconcile 20 dual-channel recordings against their spans and confirm the residual matches the round trip; include Spanish-dominant and Spanglish calls in the 20.
- Move the per-call account block to the end of the prompt and run the paired TTFT comparison per turn type.
- Turn on the duplicate-response and filler-deadline checks for 100% of calls.
- Hold your SLOs against the voice playbook, then run the latency waterfall recipe on your own spans.
NOTE
What will date on this page. Field names and definitions in LiveKit Agents, Pipecat and Deepgram's latency report, and the latency metrics and thresholds of Cekura, Coval and Hamming, all read on 2026-11-20: re-check them every quarter and at every SDK upgrade. Twilio's launch targets are dated 2025-11-17. The Speech to Speech Index figures are from its launch on 2026-06-23, and the live leaderboard had already moved by November 2026. The caching minimums (Anthropic 512 to 4,096 tokens; OpenAI 1,024 for GPT-5.6 and later) are as of 2026-11-20, and the PwC preprint is v2 from January 2026, so look for a peer-reviewed version.