On 12 November 2026 Asha, Tarazu's outbound collections agent, told the person who answered that Rahul Verma's two-wheeler EMI was four days overdue and then gave her the amount; she was his wife, and no identity check had run (trace T03). The regression suite passed because every gold scenario was answered by the borrower.
What happened on the call?
In trace T03 Asha's opener named the product, the days past due and "overdue" 2.6 seconds in, before any identity step, and her next turn gave the overdue amount to the borrower's wife.
t_ms ch event / text
2380 agent amd=human
2600 agent TTS (prompt v16): "Namaste, main Tarazu Finance se Asha bol rahi hoon. Rahul Verma ji ke two-wheeler loan ka EMI chaar din se overdue hai."
9100 caller REF: "woh office gaye hain, main unki wife bol rahi hoon, kya kaam hai?"
11050 agent TTS: "Koi baat nahi, aap unhe bata dijiye ki chaar hazaar do sau pachaas rupaye jaldi bharne hain."
Diagnosis: no AI disclosure or recording notice in the opener (C03), and dues spoken before verify_identity, twice (C04, C05), from a prompt (collections-v16) that personalised the greeting. As a 12-row timeline, the form a trace store should produce for any call:
| # | t_ms | Lane | Event | Check |
|---|
| 1 | 0 | media | Media start at the agent's ingress: zero on the call clock | |
| 2 | 2,380 | agent | AMD decides human | |
| 3 | 2,600 | agent | Turn 1 TTS starts: the collections-v16 opener | |
| 4 | 2,600 | check | Turn 1 has no AI disclosure and no recording notice | C03 fail |
| 5 | 2,600 | check | Product ("two-wheeler loan"), DPD ("chaar din") and "overdue" spoken | C04, C05 fail |
| 6 | (none) | tools | No verify_identity span anywhere in the call | |
| 7 | 9,100 | caller | "woh office gaye hain, main unki wife bol rahi hoon, kya kaam hai?" | |
| 8 | 9,100 | label | The person on the line is a third party (persona P05) | |
| 9 | 11,050 | agent | Turn 3 TTS: the overdue amount, Rs 4,250 in words | C04, C05 fail |
| 10 | (none) | gold | F05 end state: no dues disclosed, mark_contact(third_party) | CRM diff |
| 11 | (none) | versions | prompt collections-v16, live until 2026-11-12 | |
| 12 | (none) | checks | Evidence turns: C03 [1], C04 [1, 3], C05 [1, 3] | |
Three views of the same call give three verdicts:
| View | Shows | Misses | Verdict |
|---|
| Transcript only | The opener and the amount line | When they were said; whether any tool ran first | "Rude opener" |
| Tool log only | verify_identity never ran | That anything was spoken | "Incomplete flow" |
| Full timeline | Disclosure at 2,600 ms, 220 ms after AMD, no identity span before it | Nothing | Two C04 failures with evidence turns |
Only the joined view yields a result a release gate can count.
What is a voice agent trace?
A voice agent trace is the record of one call: both audio channels, the transcripts, every tool call and per-stage timings on one millisecond timeline, plus the version of every component, the CRM state before and after, and each check result with the turns that justify it. The unit is the call, not the turn.
The checks that matter span turns: C04 orders events across turns, task success diffs the CRM over the whole call (task success), and a barge-in lives in two channels at once. The general trace schema covers text agents; this chapter adds audio, a media clock, speech events, the carrier mapping and the speech-to-speech case. For voice, the data viewer needs a play button on every row. Each pipeline stage emits the events that become trace timestamps:
flowchart TD
A["Dialer: attempt caps, calling-window gate"] --> B["SIP trunk: 1600-series CLI, carrier codec"]
B --> C["CPaaS media stream: 8 kHz G.711 mu-law, 20 ms frames"]
C --> D["AMD: decided by 2.5 s"]
D --> E["VAD: vad_start, vad_end"]
E --> F["Streaming STT: stt_partial, stt_final"]
F --> G["Turn detector: eot_commit"]
G --> H["Orchestrator and LLM: llm_request, first_token"]
H --> I["Tools to CRM, payments, grievance: tool_call spans"]
I --> H
H --> J["TTS: tts_request, tts_first_byte"]
J --> K["Media out and barge-in monitor: play_start, barge_in, tts_stop"]
K --> L["Dual-channel recording, redaction, trace store"]
Why is a transcript not the trace?
A transcript is a speech-to-text model's hypothesis about one channel, not a record of what was said. The agent-side text is what went to TTS, not what was played, and overlap, silence and delivery leave no mark in either.
- STT text is a model output. In T01 STT wrote "do hazaar" for "dhai hazaar" and the sentence still reads fluently (entity error rate).
- The TTS input is not the audio. In T06 the LLM wrote "due on 05/12"; the engine, normalizing as en-US, said "May twelfth", while the CRM holds 5 December (TTS for agents).
- Delivery vanishes. In T10 only the timeline shows Asha at 5.9 syllables per second (campaign median 4.4), starting 180 ms after the caller stopped, over a sob. A transcript judge passed empathy; two human raters failed it (audio judges).
CAUTION
Vendor-reported. Treat it as a hypothesis to test on your own calls.
Hamming, a voice-agent testing vendor, says in a guide dated 30 December 2025 that 42% of 1,000 production voice-agent failures it reviewed were voice-specific issues that transcript-only analysis would miss; it publishes no method for that review 1. In a Carnegie Mellon preprint, 33% of failed calls across 13 configurations of one reference voice agent failed only on audio-derived events such as talk-over and slow replies; a transcript-only evaluation would have scored the agents 24 points higher on average. The calls were simulated (an LLM caller voiced by open TTS, one billing-support domain, English voices, 306 calls per agent) 2. Neither figure transfers to Hinglish collections, but both say an eval set without audio cannot count a whole failure class.
Which clock should the timeline use?
One clock per call: milliseconds since media start at the agent's ingress, stored as t_ms on every event. Carrier-side times from the dual-channel recording map onto it through a measured offset stored in the trace.
Wall clocks on separate services drift. T02's barge-in detection and TTS stop are 60 ms apart (33,000 and 33,060), so a skew above 60 ms between two services can swap their order. Two corrections keep the media clock honest:
- Back-date the VAD end. Voice activity detection (VAD) fires its end event only after a hangover of silence. In the schema example below the raw event fires at 22,140 ms with a 200 ms hangover, so the trace stores
vad_end_ms as 21,940. Skip the subtraction and every span-based number loses the hangover: end-of-turn delay looks 200 ms better than the caller heard.
- Map the carrier leg. The platform turn gap ends when agent audio leaves media egress; the caller hears it later. Align a known agent audio onset in the recording with its
play_start_ms, store the difference as carrier_offset_ms (212 in the example), and compute mouth-to-ear from the recording. The latency chapter owns both definitions.
On one clock a slow turn shows where the time went. This is T04, a tool turn at peak load, where a slow first token and two sequential tool calls left the caller in silence:
Figure: Trace T04 on one clock. Synthetic Tarazu data. The 3,990 ms platform turn gap is 3,950 ms of stage time plus four 10 ms orchestration hops: a slow first token, two sequential tool calls and no filler. The "hello?" at 43,200 ms became a new turn and the agent talked over itself at 44,900. The T04 excerpt logs no STT partial or final ticks, so none are drawn; the T01 turn in the schema below has both.
What fields does each turn need?
A caller turn needs its VAD span, STT partials and final with language tags and confidence, the end-of-turn commit and, once labelled, the reference text. An agent turn needs its type, LLM timings, tool calls with redacted arguments, the exact TTS text with its timings, and playback start, end and barge-in.
Here is voice-trace/1.3 on turns 6 and 7 of T01:
{
"call_id": "TZ-2026-11-14-0871234",
"schema_version": "voice-trace/1.3",
"campaign": "TW-early-nov", "account_id": "A-55120931", "persona_or_real": "real",
"started_at_local": "2026-11-14T18:41:07+05:30", "tz": "Asia/Kolkata",
"cli": "1600-XXX-XXXX", "carrier": "jio", "codec_chain": ["amr-nb", "g711-pcma", "g711-pcmu"],
"pipeline": "cascade",
"versions": {"agent": "asha-3.2.1", "prompt": "collections-v17", "stt": "vendorA-2026-10",
"turn_detector": "td-1.1@thr0.62", "llm": "model-x-2026-09-30", "tts": "vendorB-v4@8k",
"policy_pack": "rbi-2027-01", "taxonomy": "tz-fail-v4", "gold": null},
"audio": {"agent_channel_uri": "s3://.../agent.wav", "caller_channel_uri": "s3://.../caller.wav",
"sample_rate_hz": 8000, "redacted_uri": "s3://.../caller.redacted.wav",
"carrier_offset_ms": 212},
"amd": {"label": "human", "confidence": 0.91, "decided_at_ms": 2380},
"turns": [
{"turn_id": 6, "speaker": "caller", "vad_start_ms": 18220, "vad_end_ms": 21940, "vad_hangover_ms": 200,
"stt_partials": [{"t_ms": 18710, "text": "haan main"}],
"stt_final": {"t_ms": 22180, "text": "haan main do hazaar de dunga bees tarikh ko baaki next month",
"lang_tags": ["hi","hi","hi","hi","hi","hi","hi","hi","hi","hi","en","en"], "confidence": 0.78},
"eot_commit_ms": 22310, "reference_text": "haan main dhai hazaar de dunga bees tarikh ko, baaki next month"},
{"turn_id": 7, "speaker": "agent", "turn_type": "tool",
"llm": {"request_ms": 22320, "first_token_ms": 22990, "first_answer_token_ms": 22990, "cached_tokens": 1840},
"tool_calls": [{"name": "record_promise_to_pay",
"args_redacted": {"amount_paise": 200000, "ptp_date": "2026-11-20", "readback_confirmed": false},
"start_ms": 23010, "end_ms": 23480, "result": "ok"}],
"tts": {"text": "Theek hai, toh main do hazaar rupaye bees November ke liye note kar rahi hoon.",
"request_ms": 23500, "first_byte_ms": 23690},
"play_start_ms": 23740, "play_end_ms": 27900, "barge_in": null}
],
"timings_summary": {"turn_gap_ms": [1800], "stage_ms": {"eot_delay": [370], "llm_ttft": [670], "tts_ttfb": [190]}},
"crm_before": {"ptps": []},
"crm_after": {"ptps": [{"amount_paise": 200000, "ptp_date": "2026-11-20", "readback_confirmed": false}]},
"checks": [{"id": "C12", "result": "fail", "evidence_turns": [6, 7]}],
"judge_scores": [],
"labels": [{"labeler": "rv-07", "taxonomy": "tz-fail-v4", "code": "ENT.AMOUNT", "note": "dhai heard as do; PTP written before read-back"}]
}
| Block | Without it |
|---|
started_at_local, tz, cli | No calling-window (C01) or 1600-series CLI (C16) check without a dialer-log join |
carrier, codec_chain | A carrier regression averages away (telephony audio) |
pipeline, audio | No listening, no overlap, no mouth-to-ear; absent blocks read as zero |
amd | Machines counted as human connects (T07) |
vad_*, eot_commit_ms | No end-of-turn delay, no false cutoffs |
stt_*, reference_text | No WER, no entity error rate, no language-lock diagnosis (T08) |
llm, tool_calls | No TTFT or cache diagnosis (T04); order checks (C04, C12) have no input |
tts, play_*, barge_in | No turn gap, talk-over or stop latency |
crm_before, crm_after | Task success falls back to opinion |
checks, labels | A reviewer cannot jump to the second that failed |
The timings must add up. Caller speech end at 21,940 to agent audio at 23,740 is a platform turn gap of 1,800 ms: 370 ms end-of-turn delay, 670 ms TTFT, 470 ms tool time, 190 ms TTS TTFB, 50 ms playout, and 50 ms in the gaps between spans (10, 20 and 20 ms). Keep the gaps as their own number; when they grow, something between stages is queueing. The record also shows T01's failure: "do" in stt_final, "dhai" in reference_text, 200000 paise written with readback_confirmed: false.
How do versions make traces comparable?
Two traces are comparable on a metric only when every component feeding that metric has the same version, or the one that differs is the thing under test. The trace records nine: agent, prompt, STT, turn detector with its threshold, LLM, TTS, policy pack, failure taxonomy and gold set (null on real calls).
Which ones must match depends on the metric. Caller entity error rate needs the same STT and gold set; false-cutoff rate the same detector, threshold and silence budget; platform turn gap the same prompt (prefix length and order drive TTFT), LLM, TTS and load; failure-code counts the same taxonomy (tz-fail-v4 retired LAT.INTERRUPT, so v3 and v4 counts never compare raw). collections-v16 ran until 12 November and v17 from 13 November, so an unversioned week of T03-style disclosure counts mixes two prompts. TW-early-nov moved from a 300 ms to a 400 ms budget at threshold 0.70 on 17 November, which is why the detector string carries @thr. The silence budget is campaign configuration, not a component version, so join it from the campaign table by call date; a false-cutoff comparison across 17 November without that join compares two settings.
Version the collector mapping too. LiveKit added the lk.pii. prefix to span attributes in Agents 1.7.0 (August 2026) and moved spans to the OpenTelemetry GenAI semantic conventions in 1.8.0 (September 2026); its docs warn that old dashboards and queries then stop matching without raising an error 3 4 5. Test the code that maps vendor spans into voice-trace/1.3 per SDK version, so a rename fails a build instead of emptying a dashboard.
What changes in the trace for a speech-to-speech pipeline?
A speech-to-speech model does STT, reasoning and speech in one step, so there are no separate LLM TTFT or TTS TTFB spans. Store those blocks as absent, never zero, and set pipeline: "s2s" so dashboards show them as missing. The transcript can arrive after the reply, so ordering checks need another input.
In LiveKit Agents, llm_node_ttft and tts_node_ttfb are populated only by the STT-LLM-TTS pipeline and are empty with a realtime model 6. For Tarazu's asha-s2s-pilot, VAD events, eot_commit_ms, play_start_ms and both channels survive, so the platform turn gap is the one latency number comparable with the cascade. With no TTS text, the C04 order check runs on agent-channel STT from the recording, which arrives later and carries STT error. What speech-to-speech changes in evaluation is in the platform chapter.
Where does PII live in a trace?
Personal data sits in every content field (STT text, reference text, tool arguments, TTS text, label notes, both audio channels) and in latency spans, because agent frameworks attach transcripts and tool arguments to spans by default. Strip content at the collector and keep the timings.
Pipecat's STT spans carry the transcript, its TTS spans the synthesized text and its LLM spans the input messages and output 7. LiveKit's spans carry lk.pii.user_transcript, lk.pii.input_text and lk.pii.function_tool.arguments; allow_pii=False (or LIVEKIT_TELEMETRY_ALLOW_PII=0) keeps them out of your backend, while OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=false removes only the gen_ai.* content and leaves lk.pii.* in place 3. We found no equivalent Pipecat switch documented as of November 2026, so filter at your own collector.
A collections "latency trace" holds amounts, dates of birth and pincodes, so give it the recordings' access policy. On the law, one sentence: masking PII before transcripts reach storage or third-party LLMs is a sound design that fits DPDP Rule 6, not yet in force (about May 2027), and the Rules do not require real-time redaction 8. The compliance chapter owns DPDP.
What gate would have caught it?
Two deterministic gates: a validator that refuses any eval-store trace without both audio channels, the full version block or evidence turns on every check, and the ordered C04 check on 100% of calls, which would have failed T03 at 2,600 ms.
"""Refuse traces an eval cannot use, then run the ordered C04 check.
Dependency-free; synthetic Tarazu records."""
VERSIONS = ["agent", "prompt", "stt", "turn_detector", "llm", "tts",
"policy_pack", "taxonomy", "gold"]
DUES = ["overdue", "emi", "two-wheeler loan", "hazaar", "rupaye"] # named products, never bare "loan"
def refuse(t):
r = []
if t.get("schema_version") != "voice-trace/1.3":
r.append("unknown schema_version")
if not all(t.get("audio", {}).get(k) for k in ("agent_channel_uri", "caller_channel_uri")):
r.append("missing an audio channel")
v = t.get("versions", {})
gaps = [k for k in VERSIONS if k not in v or (v[k] is None and k != "gold")]
if gaps:
r.append("no versions: " + ", ".join(gaps))
r += [f"check {c['id']} has no evidence turns" for c in t.get("checks", []) if not c.get("evidence_turns")]
return r
def c04(events):
verified, fails = False, []
for e in sorted(events, key=lambda e: e["t_ms"]):
if e["kind"] == "tool" and e["name"] == "verify_identity" and e["verified"]:
verified = True
if e["kind"] == "tts" and not verified:
hits = [w for w in DUES if w in e["text"].lower()]
if hits:
fails.append(f"{e['t_ms']} ms: {', '.join(hits)}")
return fails
t01 = {"schema_version": "voice-trace/1.3",
"audio": {"agent_channel_uri": "s3://.../agent.wav", "caller_channel_uri": "s3://.../caller.wav"},
"versions": dict(zip(VERSIONS, ["asha-3.2.1", "collections-v17", "vendorA-2026-10",
"td-1.1@thr0.62", "model-x-2026-09-30", "vendorB-v4@8k",
"rbi-2027-01", "tz-fail-v4", None])),
"checks": [{"id": "C12", "result": "fail", "evidence_turns": [6, 7]}]}
t03_export = {"call_id": "T03", "text": "Namaste, main Tarazu Finance se Asha bol rahi hoon. ..."}
t03_events = [
{"t_ms": 2380, "kind": "amd", "label": "human"},
{"t_ms": 2600, "kind": "tts", "text": "... Rahul Verma ji ke two-wheeler loan ka EMI chaar din se overdue hai."},
{"t_ms": 9100, "kind": "caller", "text": "woh office gaye hain, main unki wife bol rahi hoon"},
{"t_ms": 11050, "kind": "tts", "text": "... chaar hazaar do sau pachaas rupaye jaldi bharne hain."},
]
for name, t in [("T01 trace", t01), ("T03 transcript export", t03_export)]:
reasons = refuse(t)
print(f"{name}: {'refused: ' + '; '.join(reasons) if reasons else 'accepted'}")
print("C04 fails on the T03 timeline:", " | ".join(c04(t03_events)))
Running it with Python 3.11 prints:
T01 trace: accepted
T03 transcript export: refused: unknown schema_version; missing an audio channel; no versions: agent, prompt, stt, turn_detector, llm, tts, policy_pack, taxonomy, gold
C04 fails on the T03 timeline: 2600 ms: overdue, emi, two-wheeler loan | 11050 ms: hazaar, rupaye
Both gates count every trace, so they need no interval: one refused trace fails the run. Sampling needs one. Audit 300 production traces, find none incomplete, and the one-sided 95% upper bound is still about 1.0% (rule of three), up to about 1,300 broken traces a day at 1,30,000 connected calls; use Wilson for a nonzero count (reliability statistics). The validator alone would not have stopped T03: no gold scenario had a family member answer, so wrong-party personas belong in the gold set (error analysis on calls).
What goes in the decision memo?
The memo makes the trace schema a release rule: what the eval store accepts, who owns the collector mapping, and which events reopen the decision. Tarazu's version, for the engineering manager who signs releases and the compliance lead:
- Decision. voice-trace/1.3 is the only record the eval store accepts; the validator refuses transcript-only records.
- Context. T03 disclosed dues to a family member under collections-v16 and the regression suite passed; three views gave three verdicts.
- Options considered. (1) Transcript-only store: cheapest, blind to T01 and T06, partly blind to T02, T04 and T10. (2) Full trace with both channels on every call. (3) Full trace on a sample: the 100% checks need timings and tool spans anyway. Chosen: (2), content stripped at the collector.
- Evidence. Incident log T01-T10 as of 2026-11-18, gold-v7. By transcript view, 5 incidents are visible, 3 partly and 2 not at all. A log, not a sample, so no rate is claimed.
- Owner and cadence. Voice platform team owns the schema and collector mapping; reviewed at every
schema_version bump and quarterly.
- Tripwires. Any eval-set trace without both audio channels. A collector test failing after an SDK upgrade. Any
llm or tts block stored as zero on an s2s call.
- Dated facts. LiveKit span changes in 1.7.0 (August 2026) and 1.8.0 (September 2026); realtime models leave per-stage fields empty (docs read 2026-11-20); DPDP Rule 6 about May 2027.
What to do this week
Find out which fields your traces are missing before you write another metric. Every later number in this section reads from the trace, so a gap here becomes a silent zero there.
- Rebuild ten production calls as 12-row timelines like T03's. Every cell you cannot fill is a missing field.
- Compare a raw VAD end event with the stored
vad_end_ms.
- Measure
carrier_offset_ms on five recordings per carrier and check that it is stable.
- Grep exported spans for transcripts, TTS text and tool arguments; filter at the collector if they are there.
- Add one wrong-party persona to the gold set and run the ordered disclosure check on it.
- Check your fields against the voice playbook.
NOTE
What will date on this page. LiveKit Agents field names (llm_node_ttft, tts_node_ttfb, lk.pii.*) and allow_pii, checked against the docs and livekit-agents 1.8.13 on 2026-11-20: re-check quarterly and at each SDK upgrade. Pipecat span content and the lack of a documented content switch, as of 2026-11-20. Hamming's 42% (guide dated 2025-12-30) and the Carnegie Mellon preprint (v1, September 2026; look for a reviewed version). DPDP Rule 6 commencement, about May 2027 as of 2026-11-20.