On 13 November 2026, at 17:52 and peak load, a borrower asked Asha how much he owed in total and heard 3,990 ms of silence; he said "hello? hello?", and Asha answered twice, the second reply starting over the first (trace T04). This chapter defines the one latency vocabulary the section uses, from end-of-turn delay to the turn gap, and shows how to measure it, budget it and gate a release on it.
What happened on the call?
A tool turn stacked a slow first token, two sequential CRM calls and no filler into a 3,990 ms gap, and the caller's "hello?" became a second turn.
t_ms ch event / text
40210 caller vad_end REF: "kitna baaki hai total?"
40590 agent eot_commit (end-of-turn delay 380 ms)
40600 agent llm request (orchestration hop 10 ms); first token 41820 (TTFT 1,220 ms: cold prefix, account block placed before the policy text)
41830 agent tool get_dues (hop 10 ms), end 42690 (860 ms; CRM P95 under load)
42700 agent tool check_payment_status (hop 10 ms), end 43950 (1,250 ms; sequential, not parallel)
43200 caller REF: "hello? hello?" (2,990 ms of dead air so far)
43580 agent eot_commit on "hello hello" -> a second LLM request starts
43960 agent tts request (hop 10 ms)
44150 agent TTS first byte (TTFB 190 ms) -> play_start 44200 (playout 50 ms; turn gap 3,990 ms)
44900 agent the second reply starts over the first: "Ji, main yahin hoon. Aapka total..."
Diagnosis: 380 + 1,220 + 860 + 1,250 + 190 + 50 = 3,950 ms of stage time plus four 10 ms orchestration hops. TW-early-nov still ran its 300 ms silence budget that day, so the tool-turn SLO was 2,500 ms and the filler deadline 1,100 ms. It happens only above about 1,800 concurrent calls, when the CRM P95 doubles and the LLM first-token tail widens at the same time.
Figure 1: The T04 tool turn as a waterfall. Synthetic Tarazu data. The two tool calls alone (2,110 ms) took longer than the whole LLM, TTS and playout path, and they ran one after the other. The 1,100 ms filler deadline passed with nothing played, and by the time the caller spoke into the silence at 2,990 ms the turn had already passed its 2,500 ms SLO. Grey slivers are 10 ms orchestration hops.
What does latency mean on a voice call?
On this site, voice latency is one of nine named intervals, and the headline number is the platform turn gap: caller speech end (VAD end, back-dated by the hangover) to agent audio at media egress, computed per turn from spans. Mouth-to-ear adds the carrier legs and comes from the dual-channel recording.
| Term | Starts | Ends | Notes |
|---|
| End-of-turn delay | Caller speech end (VAD end, back-dated by the VAD hangover) | eot_commit | Includes the silence budget and detector wait |
| STT final delay | Caller speech end | Final transcript segment | Overlaps end-of-turn delay |
| LLM TTFT | LLM request sent | First token of any type | Also report time to first answer token for reasoning models |
| Tool time | Tool call start | Tool result | Sequential vs parallel matters (T04) |
| TTS TTFB | First text chunk sent to TTS | First audio byte back | TTFB on these pages always means this |
| Playout delay | First audio byte | First audio frame at media egress | Buffering and 160-byte framing |
| Orchestration | One stage hands off | The next stage starts | 10 ms per hop in Tarazu's stack; a tool turn with two tool calls has four hops |
| Turn gap (platform) | Caller speech end | Agent audio starts at media egress | The headline per-turn number, from spans |
| Turn gap (mouth-to-ear) | Caller stops speaking at the handset | Agent audio reaches the caller | Measured on the dual-channel recording; adds the carrier legs |
Turn types: greeting, verification, amount-quote, readback, tool, objection, closing.
Every tool you will plug in measures a different interval under the word "latency". Map each field onto these terms before two numbers share a chart:
| Source | Field or metric | What it measures | Closest term here |
|---|
| LiveKit Agents | end_of_turn_delay, transcription_delay | End of speech to the end-of-turn decision; to the transcript | End-of-turn delay; STT final delay |
| LiveKit Agents | llm_node_ttft, tts_node_ttfb | Per stage; empty, not zero, in a speech-to-speech trace | LLM TTFT; TTS TTFB |
| LiveKit Agents | e2e_latency | User stopped speaking to agent began responding | Platform turn gap (starts at the agent's own end-of-speech detection, so no carrier leg: our reading) |
| Pipecat | TTFB per service, TTFAT | First byte from any service, the LLM included; first answer token | TTS TTFB only on the TTS service; first answer token |
| Deepgram Voice Agent API | total_latency, ttt_token_latency | End of the user's utterance to the first audio byte; first token of any type | Server-sent gap ending at the first audio byte, before playout (our mapping); LLM TTFT |
| Cekura | Latency, P25 to P99 | VAD applied to the stereo recording | Recording-side turn gap |
| Coval | Average silence gap | Mean of gaps of at least 0.5 s only | No equivalent: a mean that drops the short gaps |
| LiveKit audio simulations | Caller-heard latency, p50 to p99 | Negative means the agent talked over the caller | Signed recording-side turn gap |
Sources for the rows: LiveKit's data hooks 1, Pipecat's metrics page 2, Deepgram's latency report 3, Cekura 4, Coval 5 and LiveKit's simulations page 6, all read on 2026-11-20. Two traps: a Pipecat "TTFB" column is TTS TTFB only when filtered to the TTS service, and with report_only_initial_ttfb=True it holds only the first TTFB per service, so its P95 is not a per-turn P95. Deepgram's docs do not say whether total_latency equals the sum of its stages, and a caller-side measurement can differ from this server-sent number.
CAUTION
Vendor-reported. Treat it as a hypothesis to test on your own calls.
Thresholds differ as much as definitions. Hamming's methodology page rates P50 turn latency under 1,600 ms excellent and under 2,000 ms acceptable without defining turn latency 7; Cekura calls under 2,000 ms generally good. Neither compares with a platform turn gap until you know both endpoints.
How do the stages add up?
On a cascaded turn, the platform turn gap is end-of-turn delay + LLM TTFT + tool time + TTS TTFB + playout delay + the orchestration hops. STT final delay runs inside end-of-turn delay, so report it and never add it. Each stage is a child span of the turn, and the children must sum to the parent.
A March 2026 Hugging Face community playbook defines perceived latency the same way (end-of-utterance delay plus LLM TTFT plus TTS TTFB), traced per turn as nested Langfuse spans; across 23 non-tool turns on a small open stack it found P50 about 1.4 s and P95 about 3.4 s, with LLM TTFT showing by far the widest spread (P50 566 ms, P95 2,246 ms) 8. Tarazu adds tool spans, playout and the hops. T04 as a span tree:
turn (tool) 40,210 -> 44,200 3,990 platform turn gap
end_of_turn 40,210 -> 40,590 380 vad_end to eot_commit
hop 40,590 -> 40,600 10
llm.first_token 40,600 -> 41,820 1,220 request to first token
hop 41,820 -> 41,830 10
tool.get_dues 41,830 -> 42,690 860
hop 42,690 -> 42,700 10
tool.check_payment_status 42,700 -> 43,950 1,250
hop 43,950 -> 43,960 10
tts.first_byte 43,960 -> 44,150 190 request to first audio byte
playout 44,150 -> 44,200 50 first byte to media egress
children + hops 3,990 = parent, residual 0
Store the residual on every turn and alert when it grows: unexplained time between spans is a queue you have not instrumented. The trace schema covers span nesting for text agents; the trace chapter covers the voice fields and the media clock. For reasoning models record the first answer token too: the first token can be a thinking token, so TTFT alone understates the wait.
Why can you not add stage P95s?
Because percentiles are not additive. When stages are independent, their slow tails rarely land on the same turn, so the sum of stage P95s overstates the turn-gap P95. When stages slow down together, as under shared load in T04, the sum gets close. When each stage has a rare spike, the total's P95 can exceed the sum.
Take three independent lognormal stages, a generic teaching model rather than Tarazu's SLOs: end-of-turn delay with P50 450 ms and P95 800 ms, LLM TTFT 566 and 2,246 ms (the playbook's spread), TTS TTFB 250 and 450 ms. The total has P50 about 1.34 s and P95 about 3.0 s, while the stage P95s sum to 3.5 s. With perfectly correlated stages (the same slow turn is slow everywhere), the P95 of the total equals the sum.
The error also runs the other way. Give two stages each a 4% chance of a 2,000 ms spike on top of 100 ms: each stage P95 is 100 ms and the sum is 200 ms. A turn escapes both spikes with probability 0.96 x 0.96 = 0.9216, below 0.95, so the P95 of the total is 2,100 ms, more than ten times the sum. Cold starts, retries and connection-pool waits behave like this: rare in every stage, common in the turn.
T04 is the shared-load case: above about 1,800 concurrent calls the CRM and LLM tails widen on the same turns and line up. So measure the total directly from spans, budget the total, then allocate stage budgets as diagnostics. A stage SLO says where to look; only the turn-gap SLO says whether the caller waited too long. Load effects belong to load and cost.
What is fast enough?
Fast enough is a target you derive from your own calls: plot early hang-ups, "hello?" re-prompts and talk-over against the turn gap and find where they bend. Published numbers frame the range. Human conversation has a modal gap of about 200 ms; 800 ms and 1.5 s are practitioner rules of thumb, not measured thresholds.
The research numbers are about human pairs. The most common gap between conversational turns is around 200 ms across languages 9. In a 10-language study of answers to yes/no questions, the mean response offset was 208 ms and the median 100 ms, with overlaps counted as negative offsets and language means from 7 ms (Japanese) to 469 ms (Danish) 10. Practitioner guides commonly treat response gaps above roughly 800 ms as awkward and above about 1.5 s as broken; these are rules of thumb 11 12.
Twilio's November 2025 guide gives median launch targets of 1,115 ms (upper limit 1,400 ms) for the mouth-to-ear turn gap and 885 ms (upper limit 1,100 ms) for the platform turn gap, which excludes the internet or PSTN leg. Its component targets are STT 350 ms, LLM TTFT 375 ms and TTS TTFB 100 ms (upper limits 500, 750 and 250 ms), and it says end-of-turn detection is usually the long pole because systems are tuned to avoid false positives 13. These are vendor targets, not measurements. The 230 ms between the two medians is our arithmetic, read as the network share at target; Twilio does not state it.
Two consequences for Tarazu. First, the silence budget is a floor under the turn gap: TW's 400 ms budget is twice the human modal gap before the LLM sees a token, so an agent that endpoints on silence cannot match human timing, and lowering the budget buys latency with false cutoffs (turn-taking and barge-in owns that curve). Second, Tarazu treats a filled pause and dead air as different experiences, which is why it gates the filler start (silence budget + 800 ms) separately from the answer.
To set your own target, bin platform turn gaps in 250 ms steps per turn type and plot the share of turns followed by a re-prompt ("hello?", "aap sun rahe ho?"), the share where caller speech starts inside the gap, and the share of calls that end before the reply finishes. Where those rates bend is your target; re-plot after every silence-budget change.
Because the turns that fail are a minority, and a session median averages them away. Tool turns carry CRM round trips and often a longer LLM call, so they own the tail. Report P50 and P95 per turn type against that type's SLO, and gate on the worst.
A worked example on 200 synthetic turns: 25 TW-early-nov calls of 8 agent turns at the current 400 ms silence budget, two per call being tool turns with two sequential CRM calls. The script also runs the lognormal model above and a shared-load variant where one shock on 12% of turns doubles the LLM and both tool calls.
"""Percentiles by stage and turn type from 200 synthetic turn spans, the
sum-of-P95s check, and a call-clustered gate. Dependency-free.
Synthetic Tarazu data."""
import math
import random
rng = random.Random(20261120)
Z95 = 1.6449
def pct(xs, p):
s = sorted(xs)
k = (len(s) - 1) * p
f = math.floor(k)
return s[f] + (s[min(f + 1, len(s) - 1)] - s[f]) * (k - f)
def logn(p50, p95):
return p50 * math.exp(rng.gauss(0, 1) * math.log(p95 / p50) / Z95)
# 1. Teaching model: three independent lognormal stages (P50, P95 in ms).
STAGES = {"eot": (450, 800), "ttft": (566, 2246), "ttfb": (250, 450)}
tot = [sum(logn(*v) for v in STAGES.values()) for _ in range(200_000)]
print(f"model: total P50 {pct(tot, .5):,.0f} total P95 {pct(tot, .95):,.0f} "
f"sum of stage P95s {sum(v[1] for v in STAGES.values()):,}")
# 2. 25 TW-early-nov calls x 8 agent turns = 200 turns, silence budget 400 ms
# (600 ms in the amount-asked and date-asked states).
LAYOUT = [("greeting", 0), ("verification", 0), ("amount-quote", 0), ("objection", 0),
("tool", 2), ("readback", 0), ("tool", 2), ("closing", 0)]
SLO = {"readback": 2100, "tool": 2600} # budget + 1,500 or + 2,200 (W11)
def turn(ttype, n_tools, shock):
budget = 600 if ttype == "readback" else 400
eot = budget + logn(60, 140)
ttft = logn(520 if n_tools else 430, 900 if n_tools else 650) * shock
tools = sum(logn(330, 700) * shock for _ in range(n_tools))
ttfb, play = logn(170, 240), rng.uniform(40, 60)
hops = 10 * (2 + n_tools)
return {"type": ttype, "eot": eot, "ttft": ttft, "tool": tools, "ttfb": ttfb,
"play": play, "gap": eot + ttft + tools + ttfb + play + hops}
def calls(shared_load, n=25):
out = []
for _ in range(n):
out.append([turn(t, n, 2.0 if shared_load and rng.random() < 0.12 else 1.0)
for t, n in LAYOUT])
return out
def report(cs, label):
turns = [t for c in cs for t in c]
print(f"\n{label}: {len(turns)} turns; session median turn gap "
f"{pct([t['gap'] for t in turns], .5):,.0f}")
for ttype in dict(LAYOUT):
g = [t["gap"] for t in turns if t["type"] == ttype]
print(f" {ttype:<13} n={len(g):>3} P50 {pct(g, .5):>5,.0f} P95 {pct(g, .95):>5,.0f}"
f" SLO {SLO.get(ttype, 1900):,}")
tool = [t for t in turns if t["type"] == "tool"]
stage95 = {k: pct([t[k] for t in tool], .95) for k in ("eot", "ttft", "tool", "ttfb", "play")}
print(" tool-turn stage P95s: " + ", ".join(f"{k} {v:,.0f}" for k, v in stage95.items()))
print(f" sum of stage P95s + 40 ms hops {sum(stage95.values()) + 40:,.0f}"
f" vs tool-turn gap P95 {pct([t['gap'] for t in tool], .95):,.0f}")
return cs
def gate(cs, slo=2600, reps=2000):
boot = []
for _ in range(reps):
pick = [cs[rng.randrange(len(cs))] for _ in cs] # resample whole calls
boot.append(pct([t["gap"] for c in pick for t in c if t["type"] == "tool"], .95))
lo, hi = pct(boot, .025), pct(boot, .975)
verdict = "BREACH" if lo > slo else "pass" if hi <= slo else "unclear"
point = pct([t["gap"] for c in cs for t in c if t["type"] == "tool"], .95)
print(f" gate: tool-turn P95 {point:,.0f}, 95% interval {lo:,.0f} to {hi:,.0f}, "
f"SLO {slo:,} -> {verdict}")
base = report(calls(False), "independent stages")
gate(base)
gate(report(calls(True), "shared load (one shock slows the LLM and both tools)"))
print("\nshared load, 200 calls (400 tool turns):")
gate(calls(True, n=200))
Running it with Python 3.11 prints:
model: total P50 1,337 total P95 3,034 sum of stage P95s 3,496
independent stages: 200 turns; session median turn gap 1,208
greeting n= 25 P50 1,130 P95 1,246 SLO 1,900
verification n= 25 P50 1,140 P95 1,416 SLO 1,900
amount-quote n= 25 P50 1,102 P95 1,296 SLO 1,900
objection n= 25 P50 1,117 P95 1,257 SLO 1,900
tool n= 50 P50 1,927 P95 2,478 SLO 2,600
readback n= 25 P50 1,360 P95 1,578 SLO 2,100
closing n= 25 P50 1,147 P95 1,364 SLO 1,900
tool-turn stage P95s: eot 519, ttft 805, tool 1,044, ttfb 239, play 59
sum of stage P95s + 40 ms hops 2,706 vs tool-turn gap P95 2,478
gate: tool-turn P95 2,478, 95% interval 2,242 to 2,659, SLO 2,600 -> unclear
shared load (one shock slows the LLM and both tools): 200 turns; session median turn gap 1,275
greeting n= 25 P50 1,144 P95 1,669 SLO 1,900
verification n= 25 P50 1,168 P95 1,612 SLO 1,900
amount-quote n= 25 P50 1,218 P95 1,417 SLO 1,900
objection n= 25 P50 1,172 P95 1,466 SLO 1,900
tool n= 50 P50 1,960 P95 2,928 SLO 2,600
readback n= 25 P50 1,333 P95 1,869 SLO 2,100
closing n= 25 P50 1,119 P95 1,380 SLO 1,900
tool-turn stage P95s: eot 534, ttft 919, tool 1,260, ttfb 235, play 59
sum of stage P95s + 40 ms hops 3,046 vs tool-turn gap P95 2,928
gate: tool-turn P95 2,928, 95% interval 2,433 to 3,282, SLO 2,600 -> unclear
shared load, 200 calls (400 tool turns):
gate: tool-turn P95 3,452, 95% interval 3,181 to 3,907, SLO 2,600 -> BREACH
Figure 2: P50 and P95 platform turn gap by turn type, 25 synthetic TW-early-nov calls of 8 turns at the 400 ms silence budget (seed 20261120). Synthetic Tarazu data. Tool turns are a quarter of the turns and own the tail; the session median sits under every P95 and says nothing about them. Readback turns follow a caller turn in the amount-asked or date-asked state, so they get the 600 ms budget and a 2,100 ms SLO.
What to read from it:
- The median hides the tail. The session median is 1,208 ms. Five non-tool types have P95s under 1,420 ms against 1,900, and readback 1,578 against 2,100. Tool turns have P50 1,927 and P95 2,478 against 2,600: a quarter of the turns and all of the risk.
- The stage sum overstates, then less so. On independent tool turns the stage P95s plus hops sum to 2,706 ms against a measured 2,478, 228 ms too pessimistic. Under shared load the gap shrinks to 118 ms (3,046 against 2,928): the slow LLM and slow CRM land on the same turns.
- Fifty turns cannot decide a P95. Both 25-call runs come out unclear, their intervals straddling 2,600 ms. On 200 shared-load calls the P95 is 3,452 ms and the whole interval sits above the SLO. Size the run before you read a P95 (reliability statistics).
Segment where the tail lives: prompt length, number of tool calls, cache hit or miss. A P95 that rises only on misses is a prefix problem; one that rises only on two-tool turns is a parallelism problem.
How do you measure what the caller heard?
Run an energy VAD on each channel of the dual-channel recording, take the signed gap from caller speech end to agent audio onset (negative: the agent talked over the caller), and reconcile onsets with spans to recover the carrier offset. Spans give the platform turn gap; only the recording shows what the caller heard.
The method, implemented step by step in the latency waterfall recipe:
- Detect speech per channel. Short frames, a fixed energy threshold, tiny gaps bridged. A mono mix cannot give a signed gap.
- Recover the offset. Let A be the median of (agent onset on the recording minus
play_start_ms), which is the carrier offset plus the outbound leg, and B the median of (vad_start_ms minus caller onset on the recording), which is the inbound leg minus the offset. Then offset = (A - B) / 2 and round trip = A + B, assuming equal legs as NTP does. Use VAD starts, not ends: the hangover never touches an onset.
- Compute the signed gap. Agent audio onset minus the end of the caller's last speech, both read on the recording. LiveKit's audio simulations use the same sign convention for caller-heard latency 6.
- Compare with spans. On clean turns the median of (recording gap minus span gap) should equal the round trip; the rest is time the spans do not see.
Step 4 catches instrumentation bugs. Tarazu stores vad_end_ms back-dated: the raw VAD end fires a hangover later (200 ms in the trace schema) and the pipeline subtracts it. A pipeline that does not back-date leaves the hangover outside every span number, so end-of-turn delay and the platform turn gap look 200 ms better than the caller's experience, and only the recording shows it, as a step-4 residual equal to the hangover. The same comparison finds turns where the caller paused, the turn was committed and the caller resumed: a negative recording gap beside a positive span gap, which is a cutoff problem, not a speed problem.
The recording is captured on Tarazu's side of the carrier, so it still leaves out the access network and the handset. For true mouth-to-ear latency, place scheduled test calls from real handsets on each carrier, record at the handset, and store the per-carrier remainder with a date; it moves when a carrier changes its routing.
How do you set per-stage SLOs?
Set the turn-gap SLO per turn type and silence budget, then allocate stage budgets under it. Write both as the budget plus an allowance (150 ms of detector time for end-of-turn delay, a fixed amount for the gap): end-of-turn delay contains the silence budget by definition, so a longer budget must move the SLO with it.
Tarazu's SLOs (team decisions, not industry thresholds):
| Stage | P95 budget |
|---|
| End-of-turn delay | Silence budget + 150 ms detector allowance: TW, PL and CD 550 ms (750 ms in amount-asked and date-asked states); SBL 600 ms (800); MFI 750 ms (950). Before T05's fix TW ran a 300 ms budget, so its SLO was 450 ms; T01 (370 ms) and T04 (380 ms) were inside it |
| STT final delay | 300 ms |
| LLM TTFT | 700 ms non-tool turns, 900 ms tool turns |
| Tool time (parallel where possible) | 600 ms |
| TTS TTFB | 250 ms |
| Playout | 60 ms |
| Platform turn gap | Silence budget + 1,500 ms on non-tool turns and + 2,200 ms on tool turns: 1,900 and 2,600 ms at TW's 400 ms budget; MFI default 2,100 and 2,800 ms. Filler must start by the budget + 800 ms (1,200 ms at TW) |
Against the SLOs in force that day (300 ms budget: 450, 900, 600, 250, 60, and 2,500 for the tool-turn gap), T04's end-of-turn delay, TTS TTFB and playout passed; LLM TTFT (1,220 ms) and tool time (2,110 ms, sequential) broke it. The stage table says what to fix: parallelize the lookups and repair the prefix. The gap SLO says whether the fix was enough.
Do not add the stage budgets and call the result an SLO. For a TW tool turn they sum to 550 + 900 + 600 + 250 + 60 + 40 ms of hops = 2,400 ms, under the 2,600 ms gap SLO. That is the right shape for an allocation: even if shared load lines every tail up, stages at budget still fit. It is not proof. A stage can pass its P95 and still carry rare spikes that push the turn past 2,600 ms, as in the spike example above. Only the measured gap decides.
Guardrails spend the same budget. A synchronous check on the TTS input text, such as C13 (the spoken amount equals the get_dues result), sits between the first answer token and the TTS request, and a pipeline that streams TTS sentence by sentence must clear each sentence before it is spoken. That is the trade-off: every check you run inline is milliseconds TTS does not get. Run deterministic text checks inline, keep model judges asynchronous, and give the guardrail its own span so it never hides inside "LLM" (guardrails versus evals; for latency columns in a general scorecard, cost and latency evaluation).
What does caching buy you?
Prefix caching cuts cost reliably and LLM TTFT by less, and less predictably. Put the static policy and script first and the per-call account block last, check that the cached prefix clears the provider's minimum, and compare TTFT on hits and misses: a cached-token count proves a hit, not a speed-up.
A correction first: Gnani has published no KV or prefix caching work. It describes response caching of repeated or fixed utterances, which its homepage says serves them 300 to 400 ms faster (self-reported) 14. That skips generation for a canned line, and you evaluate it by checking the cached line is still right for the call state.
The evidence for agents is modest. In a PwC preprint covering over 500 agent sessions with 10,000-token system prompts on four models, prompt caching cut cost by 41% to 80% but improved TTFT by only 6% to 31%, and caching the full context made GPT-4o's TTFT 8.8% worse; the authors suggest that caching dynamic content can add overhead that negates the latency benefit 15. The workload was a research agent, not voice. Anthropic's 2024 launch post reported TTFT falling from 1.6 s to 1.1 s (-31%) for a 10,000-token prompt and from 11.5 s to 2.4 s for a 100,000-token one; these are vendor figures 16. Current docs give no latency number. Anthropic's docs say prompts below the model's minimum (512 to 4,096 tokens) are processed without caching and with no error 17. OpenAI's say reuse needs the entire rendered prefix to match, with a 1,024-token minimum for GPT-5.6 and later 18. Our inference, not a measured result: a voice prompt of a few thousand tokens should expect gains near the 10,000-token row or less.
T04 shows the failure that matters most. The account block was placed before the policy text, so the prefix differed on every call and the request ran cold; TTFT on that turn was 1,220 ms against a 900 ms SLO. The fix is ordering, and the test is paired: run the same scenarios on both prompt orders and compare TTFT per turn type with a call-clustered interval. Because caching fails silently below the minimum, alert on cache-hit rate per prompt version.
Does speech-to-speech remove the problem?
No. A speech-to-speech model removes the per-stage spans, so LLM TTFT and TTS TTFB are absent from the trace and the platform turn gap and mouth-to-ear from the recording become the only latency numbers. And when a public speech-to-speech index launched in June 2026, its fastest models were not its most capable.
At the launch of the Artificial Analysis Speech to Speech Index on 23 June 2026, Deepslate Opal had the fastest average time to first audio at 0.44 s with a 62.1% score, while GPT-Realtime-2 (High) took 2.33 s and scored 77.2%; the live leaderboard has since changed, and its timings come from a benchmark harness, not a phone line 19. What else changes when Tarazu's asha-s2s-pilot replaces the cascade is owned by the platform chapter.
What gate would have caught it?
A release gate on the P95 platform turn gap of tool turns, computed on the replay set with a call-clustered bootstrap interval, plus two deterministic checks on every call: a duplicate-response check and a filler deadline. Block on BREACH, collect more calls on unclear, and never use a Wilson interval on a percentile.
- Tool-turn P95 against the SLO of its call state. Resample whole calls, because turns in one call share a network path, a CRM shard and a prompt history 20. BREACH when the lower bound is above the SLO, pass when the upper bound is at or below it, unclear otherwise.
- Duplicate response. Flag any caller
eot_commit that lands while an agent reply is pending, between the previous commit and play_start_ms. In T04 the second commit at 43,580 fell inside 40,590 to 44,200. This runs on spans alone, on 100% of calls, and maps to LAT.DUP.
- Filler deadline. On every turn predicted to need a tool, the filler's
play_start_ms must be at most the silence budget + 800 ms after caller speech end (1,200 ms at TW today). T04 had no filler at all.
The replay set catches the prefix and the sequential calls at any load; the shared-load tail exists only at projected peak (2,400 concurrent calls today), so the same gate also runs inside the pre-release load test (load and cost). Daily P95 dashboards belong to production monitoring.
What goes in the decision memo?
The memo makes per-turn-type latency SLOs a release rule: what gates a release, what is only a diagnostic, and what reopens the decision. Tarazu's version, for the engineering manager who signs releases:
- Decision. Gate releases on P95 platform turn gap per turn type, with SLOs written as silence budget + 1,500 ms (non-tool) and + 2,200 ms (tool). Stage SLOs are diagnostic allocations, never summed into a gate.
- Context. T04: a 3,990 ms tool-turn gap at peak load and a double reply. Tool turns are a quarter of turns, so a session median or a global target averages them away.
- Options considered. (1) One global P95 target for every turn: simple, but passes while tool turns fail. (2) A sum of stage P95s: overstates independent tails and says nothing under shared load. (3) Per turn type and per silence budget, measured directly from spans. Chosen: (3).
- Evidence. The T04 trace; the 200-turn synthetic study (seed 20261120: stage P95s summed 228 ms above the measured P95); release replays on gold-v7 scenarios with call-clustered intervals; weekly recording reconciliation.
- Owner and cadence. Voice platform team; P95 by turn type reviewed weekly; SLOs re-derived whenever a campaign's silence budget changes.
- Tripwires. TW tool-turn P95 above 2,600 ms for a day. The median recording-minus-span residual moving away from the measured round trip. A drop in cache-hit rate after a prompt change.
- Dated facts. Twilio's launch targets (November 2025); the 800 ms and 1.5 s rules of thumb; the Speech to Speech Index launch figures (June 2026).
What to do this week
Start with the vocabulary map and the span residual. A field mapped to the wrong interval, or a queue with no span, makes every P95 computed after it wrong.
- Map your vendor's latency fields onto the table above; mark the ones with no equivalent.
- Compute the span residual on 100 turns; any turn with unexplained time is an uninstrumented queue.
- Pull P50 and P95 by turn type for one campaign and compare the tool-turn P95 with the session median.
- Reconcile 20 dual-channel recordings with their spans and check that the residual equals the round trip.
- Move the per-call account block to the end of the prompt and run the paired TTFT comparison.
- Put the duplicate-response and filler-deadline checks on 100% of calls.
- Compare your SLOs with the voice playbook, and run the latency waterfall recipe on your own spans.
NOTE
What will date on this page. Field names and definitions in LiveKit Agents, Pipecat and Deepgram's latency report, and Cekura's, Coval's and Hamming's latency metrics and thresholds, all read 2026-11-20: re-check quarterly and at SDK upgrades. Twilio's launch targets, dated 2025-11-17. The Speech to Speech Index launch figures (2026-06-23); the live leaderboard had already changed by November 2026. Caching minimums (Anthropic 512 to 4,096 tokens; OpenAI 1,024 for GPT-5.6 and later) as of 2026-11-20, and the PwC preprint (v2, January 2026; look for a reviewed version).