On 12 November 2026 Hazel, Steelyard Credit's AI collections agent, called Ryan Mitchell, reached his brother, and told him the auto loan payment was four days past due and how much was owed, with no identity check in the call (trace T03). Every gold scenario had the borrower pick up, so the regression suite stayed green.
What happened on the call?
In T03 the opening line, 2.6 seconds after media start, named the product, the days past due and the words "past due" before any identity step, and the next agent turn asked the brother to pass on the dollar amount.
t_ms ch event / text
2380 agent amd=human
2600 agent TTS (prompt v16): "Hi, this is Hazel from Steelyard Credit. I'm calling about Ryan Mitchell's auto loan payment, which is four days past due."
9100 caller REF: "He's at work. This is his brother, he left his phone here. What's this about?"
11050 agent TTS: "No problem. Can you let him know he needs to pay the four hundred twenty-five dollars as soon as possible?"
Diagnosis: the opener led with the agent, not the business, and had no AI disclosure or recording notice (C03); dues went out twice before verify_identity (C04, C05), from a prompt (collections-v16) that "personalised" the greeting. A trace store should rebuild any call as a timeline like this:
| # | t_ms | Lane | Event | Check |
|---|
| 1 | 0 | media | Media start at the agent's ingress, zero on the call clock | |
| 2 | 2,380 | agent | AMD labels the answer human | |
| 3 | 2,600 | agent | Turn 1 TTS begins: the collections-v16 opener | |
| 4 | 2,600 | check | No business identity first, no AI disclosure, no recording notice | C03 fail |
| 5 | 2,600 | check | Product ("auto loan"), DPD ("four days") and "past due" spoken | C04, C05 fail |
| 6 | (none) | tools | No verify_identity span exists in the call | |
| 7 | 9,100 | caller | "He's at work. This is his brother, he left his phone here. What's this about?" | |
| 8 | 9,100 | label | A third party is on the line (persona P05) | |
| 9 | 11,050 | agent | Turn 3 TTS: the past-due amount, $425.00 in words | C04, C05 fail |
| 10 | (none) | gold | F05 end state: no debt disclosed, mark_contact(third_party) | CRM diff |
| 11 | (none) | versions | prompt collections-v16, live until 2026-11-12 | |
| 12 | (none) | checks | Evidence turns: C03 [1], C04 [1, 3], C05 [1, 3] | |
Three views of the call, three verdicts:
| View | Shows | Misses | Verdict |
|---|
| Transcript only | The opener and the amount request | The timing, and whether any tool ran before either line | "Awkward opener" |
| Tool log only | verify_identity never ran | That anything about the debt was said | "Flow cut short" |
| Full timeline | Dues at 2,600 ms, 220 ms after AMD, with no identity span before them | Nothing | Two C04 failures, each with its evidence turn |
A release gate can count only the third verdict.
What is a voice agent trace?
A voice agent trace is the record of one call: both audio channels, the transcripts, every tool call and per-stage timings on one millisecond timeline, plus every component version, the CRM state before and after, the dial outcome and each check result with its evidence turns. The call is the unit, not the turn.
The checks worth running cross turns: C04 orders events over the whole call, task success diffs the CRM across it, and a barge-in exists only as overlap between two channels. The trace schema page covers text agents; voice adds audio, a media clock, speech events, the carrier mapping, dial outcomes and the speech-to-speech case, and its data viewer needs a play button on every row. Each stage of Steelyard's pipeline emits the events that become timestamps:
flowchart TD
A["Dialer: attempt counter per person per debt, calling-window gate"] --> B["SIP trunk: STIR/SHAKEN-signed caller ID, AMR or Opus leg"]
B --> C["CPaaS media stream: 8 kHz G.711 mu-law, 20 ms frames"]
C --> D["AMD: six classes, decided by 2.5 s"]
D --> E["VAD: vad_start, vad_end"]
E --> F["Streaming STT, English-Spanish: stt_partial, stt_final"]
F --> G["Turn detector: eot_commit"]
G --> H["Orchestrator and LLM: llm_request, first_token"]
H --> I["Tools to CRM, payments, disclosures: tool_call spans"]
I --> H
H --> J["TTS, en-US or es-US: tts_request, tts_first_byte"]
J --> K["Media out and barge-in monitor: play_start, barge_in, tts_stop"]
K --> L["Dual-channel recording, redaction, trace store"]
Why is a transcript not the trace?
A transcript is a speech-to-text model's guess at one channel; on the agent side it is the text sent to synthesis, not the audio that played. Overlap, silence, pace and tone leave no mark in it.
- STT output is a hypothesis. In T01 Ryan said "two fifty" and STT wrote "two fifteen"; the sentence still reads cleanly (entity error rate).
- TTS input is not the audio. In T06 the English stack wrote "Su proximo pago de $680.00 vence el 12/03" [Your next payment of $680.00 is due on 12/03], month first. The es-US voice read short dates day first and said "el doce de marzo" [the twelfth of March], while the CRM held 3 December (TTS for agents).
- Delivery disappears. In T10 only the timeline shows Hazel speaking at 5.9 syllables per second against a campaign median of 4.4, starting 180 ms after the borrower stopped, over crying. The transcript judge passed empathy; two human raters failed it (audio judges).
CAUTION
Vendor-reported. Treat it as a hypothesis to test on your own calls.
Hamming, a voice-agent testing vendor, wrote in a guide dated 30 December 2025 that 42% of 1,000 production failures it reviewed were voice-specific issues a transcript-only review would miss, with no published method 1. In a Carnegie Mellon preprint, 33% of failed calls across 13 configurations of one reference agent failed only on audio-derived events such as talk-over and slow replies; transcript-only scoring rated the agents 24 points higher on average. Those calls were simulated (an LLM caller voiced by open TTS, one billing-support domain, English voices, 306 calls per agent) 2. Neither study covers collections or Spanish speakers, but both point to a failure class that an eval set without audio cannot count.
Which clock should the timeline use?
Use one clock per call: milliseconds since media start at the agent's ingress, written as t_ms on every event. Times taken from the carrier side of the dual-channel recording join that clock through a measured offset that the trace stores.
Wall clocks on separate services drift. In T02 the barge-in detection and the TTS stop are 60 ms apart (33,000 and 33,060), so skew above 60 ms between the two services is enough to reverse them. Two corrections keep the media clock trustworthy:
- Back-date the end of speech. Voice activity detection (VAD) reports speech end only after a hangover of silence. Below, the raw event fires at 22,140 ms with a 200 ms hangover, so
vad_end_ms is stored as 21,940. Skip the subtraction and end-of-turn delay, like every span-based number, reads 200 ms better than the caller heard.
- Map the carrier leg. The platform turn gap ends when agent audio leaves media egress; the caller hears it later. Align a known agent onset in the recording with its
play_start_ms, store the difference as carrier_offset_ms (212 here), and compute mouth-to-ear from the recording. The latency chapter owns both definitions.
With one clock, a slow turn shows exactly where its time went. T04 is a tool turn at peak load in which a slow first token and two tool calls in series left the caller in silence:
Figure: T04 on a single clock. Synthetic Steelyard data. The 3,990 ms platform turn gap breaks down into 3,950 ms of stage time and four 10 ms orchestration hops: a cold first token, two tool calls in series and no filler line. The "hello?" at 43,200 ms opened a new turn, so Hazel talked over her own reply at 44,900. No STT partial or final ticks appear because the T04 excerpt logs none; the T01 turn in the schema below carries both.
What fields does each turn need?
A caller turn needs its VAD span, the STT partials and final with language tags and confidence, the end-of-turn commit and, once labeled, the reference text. An agent turn needs its turn type, LLM timings, tool calls with redacted arguments, the exact TTS text with its timings, and playback start, end and barge-in.
Here is voice-trace/1.3 for turns 6 and 7 of T01:
{
"call_id": "SY-2026-11-14-0871234",
"schema_version": "voice-trace/1.3",
"campaign": "AUTO-early-nov", "account_id": "A-55120931", "persona_or_real": "real",
"started_at_local": "2026-11-14T18:41:07-07:00", "tz": "America/Phoenix",
"location_signals": {"area_code": "America/Phoenix", "address": "America/Phoenix"},
"dial": {"attempts_7d_before": 2, "connect_outcome": "answered", "counts_toward_cap": true, "conversation": true},
"cli": "+1-520-555-01XX", "attestation": "A", "carrier": "carrier-b-mobile", "codec_chain": ["amr-wb", "g711-pcmu"],
"pipeline": "cascade",
"versions": {"agent": "hazel-3.2.1", "prompt": "collections-v17", "stt": "vendorA-2026-10",
"turn_detector": "td-1.1@thr0.62", "llm": "model-x-2026-09-30", "tts": "vendorB-v4@8k",
"policy_pack": "us-regf-2026-11", "taxonomy": "gl-fail-v4", "gold": null},
"audio": {"agent_channel_uri": "s3://.../agent.wav", "caller_channel_uri": "s3://.../caller.wav",
"sample_rate_hz": 8000, "redacted_uri": "s3://.../caller.redacted.wav",
"carrier_offset_ms": 212},
"amd": {"label": "human", "confidence": 0.91, "decided_at_ms": 2380},
"turns": [
{"turn_id": 6, "speaker": "caller", "vad_start_ms": 18220, "vad_end_ms": 21940, "vad_hangover_ms": 200,
"stt_partials": [{"t_ms": 18710, "text": "yeah I can"}],
"stt_final": {"t_ms": 22180, "text": "yeah I can do two fifteen on the twentieth rest next month",
"lang_tags": ["en","en","en","en","en","en","en","en","en","en","en","en"], "confidence": 0.78},
"eot_commit_ms": 22310, "reference_text": "yeah I can do two fifty on the twentieth, rest next month"},
{"turn_id": 7, "speaker": "agent", "turn_type": "tool",
"llm": {"request_ms": 22320, "first_token_ms": 22990, "first_answer_token_ms": 22990, "cached_tokens": 1840},
"tool_calls": [{"name": "record_promise_to_pay",
"args_redacted": {"amount_cents": 21500, "ptp_date": "2026-11-20", "readback_confirmed": false},
"start_ms": 23010, "end_ms": 23480, "result": "ok"}],
"tts": {"text": "Okay, so I'm noting two hundred fifteen dollars for November twentieth.",
"request_ms": 23500, "first_byte_ms": 23690},
"play_start_ms": 23740, "play_end_ms": 27900, "barge_in": null}
],
"timings_summary": {"turn_gap_ms": [1800], "stage_ms": {"eot_delay": [370], "llm_ttft": [670], "tts_ttfb": [190]}},
"crm_before": {"ptps": []},
"crm_after": {"ptps": [{"amount_cents": 21500, "ptp_date": "2026-11-20", "readback_confirmed": false}]},
"checks": [{"id": "C12", "result": "fail", "evidence_turns": [6, 7]}],
"judge_scores": [],
"labels": [{"labeler": "rv-07", "taxonomy": "gl-fail-v4", "code": "ENT.AMOUNT", "note": "fifty heard as fifteen; PTP written before read-back"}]
}
| Block | What breaks without it |
|---|
started_at_local, tz, location_signals | No calling-window check (C01) across every location signal without a dialer-log join |
dial | No attempt cap (C11): unanswered calls and voicemails go uncounted |
cli, attestation | No caller ID check (C16) |
carrier, codec_chain | A regression on one carrier averages away (telephony audio) |
pipeline, audio | No listening, no overlap, no mouth-to-ear; missing blocks read as zero |
amd | Machines counted as human connects (T07) |
vad_*, eot_commit_ms | No end-of-turn delay, no false cutoffs |
stt_*, reference_text | No WER, no entity error rate, no way to see a language lock (T08) |
llm, tool_calls | No TTFT or cache diagnosis (T04); the order checks (C04, C12) have nothing to read |
tts, play_*, barge_in | No turn gap, talk-over or stop latency |
crm_before, crm_after | Task success becomes a matter of opinion |
checks, labels | A reviewer cannot jump to the second that failed |
Text-agent schemas lack the dial block. Steelyard collects its own loans in its own name, so the FDCPA and Regulation F largely do not bind it: a creditor's employees collecting in the creditor's name are outside both definitions of debt collector 3 4. It adopts their conduct rules as policy, and state law, the TCPA and the CFPB's ban on unfair, deceptive or abusive practices still apply. Under the Regulation F commentary, unanswered calls and calls that reach voicemail count toward the 7-in-7 limit, while busy and out-of-service calls do not 5. A telephone conversation about the debt starts a 7-day period with no further calls, counting the conversation date as day one 6. So the trace logs every attempt with connect_outcome, counts_toward_cap and conversation. In T07 a call-screening assistant was logged as a human connect and a conversation, and the dialer froze the account for a week of contact that never happened.
The timings have to add up. From caller speech end at 21,940 to agent audio at 23,740 the platform turn gap is 1,800 ms: 370 ms end-of-turn delay, 670 ms TTFT, 470 ms tool time, 190 ms TTS TTFB, 50 ms playout, and 50 ms of handoff between spans (10, 20 and 20 ms). Report the handoff as its own number; when it grows, something between stages is queueing. The same record also holds T01's failure: "fifteen" in stt_final, "fifty" in reference_text, and 21500 cents written with readback_confirmed: false, $35.00 short of what Ryan offered.
Assemble the record by call_id, not by appending events as they arrive. The spans land during the call, the recording and carrier_offset_ms minutes later, and reference_text, labels and the kept-or-broken PTP outcome days later. Treat each late block as a versioned update to the same call, and have every eval run record which revision of each trace it read; otherwise a rerun after labeling scores different data under the same name.
How do versions make traces comparable?
Two traces can be compared on a metric only if every component that feeds the metric has the same version, or the one that differs is the thing being tested. The trace stores nine versions: agent, prompt, STT, turn detector with its threshold, LLM, TTS, policy pack, failure taxonomy and gold set (null on real calls).
The set depends on the metric. Caller entity error rate needs the same STT and gold set; false-cutoff rate the same detector, threshold and silence budget; platform turn gap the same prompt (prefix length and order drive TTFT), LLM, TTS and load; failure-code counts the same taxonomy (gl-fail-v4 retired LAT.INTERRUPT, so raw v3 and v4 counts never compare). Prompt collections-v16 ran until 12 November and v17 from 13 November, so a week of T03-style disclosure counts with no prompt version mixes two prompts. AUTO-early-nov moved from a 300 ms to a 400 ms silence budget, at threshold 0.70, on 17 November, which is why the detector string carries @thr. The silence budget is campaign configuration, not a component, so join it from the campaign table by call date; skip the join and a false-cutoff comparison across 17 November compares two settings.
The collector mapping needs a version as well. LiveKit added the lk.pii. prefix to span attributes in Agents 1.7.0 (August 2026) and moved its spans to the OpenTelemetry GenAI semantic conventions in 1.8.0 (September 2026); its docs warn that existing dashboards and queries then stop matching, with no error raised 7 8 9. Test the span mapping into voice-trace/1.3 per SDK version, so a rename fails a build instead of emptying a dashboard.
What changes in the trace for a speech-to-speech pipeline?
A speech-to-speech model does recognition, reasoning and speech in one step, so separate LLM TTFT and TTS TTFB spans do not exist. Store those blocks as absent, never zero, and set pipeline: "s2s" so dashboards mark them missing. The transcript may arrive after the reply, so ordering checks need another input.
LiveKit Agents fills llm_node_ttft and tts_node_ttfb only for the STT-LLM-TTS pipeline; with a realtime model they stay empty 10. In hazel-s2s-pilot, VAD events, eot_commit_ms, play_start_ms and both channels remain, so the platform turn gap is the one latency number that compares with the cascade, the paired control. With no TTS text, the C04 order check reads agent-channel STT from the recording, which arrives later and adds recognition error. The platform chapter covers what speech-to-speech changes for evaluation.
Where does PII live in a trace?
Personal data sits in every content field (STT text, reference text, tool arguments, TTS text, label notes, both audio channels) and also in latency spans, because agent frameworks attach transcripts and tool arguments to spans by default. Strip content at the collector and keep the timings.
Pipecat's STT spans carry the transcript, its TTS spans the synthesized text, and its LLM spans the input messages and the output 11. LiveKit's spans carry lk.pii.user_transcript, lk.pii.input_text and lk.pii.function_tool.arguments. Setting allow_pii=False (or LIVEKIT_TELEMETRY_ALLOW_PII=0) keeps them out of your backend, while OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=false removes only the gen_ai.* content and leaves lk.pii.* in place 7. We found no matching Pipecat switch documented as of November 2026, so filter at your own collector.
A collections "latency trace" holds amounts, dates of birth and ZIP codes, so give it the recordings' access policy. For the UK and EU books, recordings and transcripts are personal data under UK and EU GDPR, and voice characteristics extracted to identify a speaker are special category biometric data, so Hazel never authenticates by voice 12. The ICO says that guidance is under review after the Data (Use and Access) Act 2025. Recording notices, retention and the US privacy rules belong to the compliance chapter.
What gate would have caught it?
Two deterministic gates: a validator that refuses any eval-store trace missing an audio channel, the version block, the dial outcome or evidence turns on a check, and the ordered C04 check on 100% of calls, which fails T03 at 2,600 ms.
"""Refuse traces an eval cannot use, then run the ordered C04 check.
Dependency-free; synthetic Steelyard records."""
VERSIONS = ["agent", "prompt", "stt", "turn_detector", "llm", "tts",
"policy_pack", "taxonomy", "gold"]
DIAL = ["attempts_7d_before", "connect_outcome", "counts_toward_cap", "conversation"]
# Toy lexicon; production matches canonicalized amount and date entities too.
DUES = ["past due", "auto loan", "payment", "dollars", "owe"]
def refuse(t):
r = []
if t.get("schema_version") != "voice-trace/1.3":
r.append("unknown schema_version")
if not all(t.get("audio", {}).get(k) for k in ("agent_channel_uri", "caller_channel_uri")):
r.append("missing an audio channel")
v = t.get("versions", {})
gaps = [k for k in VERSIONS if k not in v or (v[k] is None and k != "gold")]
if gaps:
r.append("no versions: " + ", ".join(gaps))
d = t.get("dial", {})
if any(k not in d for k in DIAL):
r.append("no dial outcome")
r += [f"check {c['id']} has no evidence turns" for c in t.get("checks", []) if not c.get("evidence_turns")]
return r
def c04(events):
verified, fails = False, []
for e in sorted(events, key=lambda e: e["t_ms"]):
if e["kind"] == "tool" and e["name"] == "verify_identity" and e["verified"]:
verified = True
if e["kind"] == "tts" and not verified:
hits = [w for w in DUES if w in e["text"].lower()]
if hits:
fails.append(f"{e['t_ms']} ms: {', '.join(hits)}")
return fails
t01 = {"schema_version": "voice-trace/1.3",
"audio": {"agent_channel_uri": "s3://.../agent.wav", "caller_channel_uri": "s3://.../caller.wav"},
"versions": dict(zip(VERSIONS, ["hazel-3.2.1", "collections-v17", "vendorA-2026-10",
"td-1.1@thr0.62", "model-x-2026-09-30", "vendorB-v4@8k",
"us-regf-2026-11", "gl-fail-v4", None])),
"dial": {"attempts_7d_before": 2, "connect_outcome": "answered",
"counts_toward_cap": True, "conversation": True},
"checks": [{"id": "C12", "result": "fail", "evidence_turns": [6, 7]}]}
t03_export = {"call_id": "T03", "text": "Hi, this is Hazel from Steelyard Credit. ..."}
t03_events = [
{"t_ms": 2380, "kind": "amd", "label": "human"},
{"t_ms": 2600, "kind": "tts", "text": "... I'm calling about Ryan Mitchell's auto loan payment, which is four days past due."},
{"t_ms": 9100, "kind": "caller", "text": "He's at work. This is his brother, he left his phone here."},
{"t_ms": 11050, "kind": "tts", "text": "... he needs to pay the four hundred twenty-five dollars as soon as possible?"},
]
for name, t in [("T01 trace", t01), ("T03 transcript export", t03_export)]:
reasons = refuse(t)
print(f"{name}: {'refused: ' + '; '.join(reasons) if reasons else 'accepted'}")
print("C04 fails on the T03 timeline:", " | ".join(c04(t03_events)))
Running it with Python 3.11 prints:
T01 trace: accepted
T03 transcript export: refused: unknown schema_version; missing an audio channel; no versions: agent, prompt, stt, turn_detector, llm, tts, policy_pack, taxonomy, gold; no dial outcome
C04 fails on the T03 timeline: 2600 ms: past due, auto loan, payment | 11050 ms: dollars
The lexicon is a stand-in: the second T03 turn trips only on "dollars", so the production check also matches canonicalized amount and date entities. Both gates see every trace, so neither needs an interval: one refused trace fails the run. Sampling does. Audit 300 production traces, find none incomplete, and the one-sided 95% upper bound is still about 1.0% (rule of three), up to about 1,300 broken traces a day at 130,000 connected calls; use Wilson for a nonzero count (reliability statistics). The validator alone would not have stopped T03: no gold scenario had a relative answer, so wrong-party personas belong in the gold set (error analysis on calls).
What goes in the decision memo?
The memo makes the trace schema a release rule: what the eval store accepts, who owns the collector mapping, what reopens it. Steelyard's version, for the engineering manager who signs releases and the compliance lead:
- Decision. voice-trace/1.3 is the only record the eval store accepts; the validator refuses transcript-only records and records with no dial outcome.
- Context. Under collections-v16, T03 disclosed a debt to the borrower's brother and the regression suite passed; three views gave three verdicts.
- Options considered. (1) Transcripts only: cheapest, blind to T01 and T06, partly blind to T02, T04 and T10. (2) Full trace, both channels, every call. (3) Full trace on a sample: the 100% checks need timings, tool spans and dial outcomes on every call anyway. Chosen: (2), content stripped at the collector.
- Evidence. Incident log T01-T10 as of 2026-11-18, gold-v7. In the transcript view, 5 incidents are visible, 3 partly and 2 not at all. This is a log, not a sample, so no rate is claimed.
- Owner and cadence. The voice platform team owns the schema and the collector mapping; reviewed at every
schema_version change and each quarter.
- Tripwires. Any eval-set trace missing an audio channel or a
dial block. A collector test that fails after an SDK upgrade. Any llm or tts block stored as zero on an s2s call.
- Dated facts. LiveKit span changes in 1.7.0 (August 2026) and 1.8.0 (September 2026); realtime models leave per-stage fields empty (docs read 2026-11-20); ICO biometric guidance under review as of 2026-11-20.
What to do this week
Find the fields your traces lack before you write another metric; a gap here becomes a silent zero in every later chapter.
- Rebuild ten production calls as 12-row timelines like T03's. Each cell you cannot fill marks a missing field.
- Pull one raw VAD end event and compare it with the stored
vad_end_ms.
- Measure
carrier_offset_ms on five recordings per carrier and check that it holds steady.
- Count a week of dial attempts by
connect_outcome and compare with what your attempt cap counted.
- Search exported spans for transcripts, TTS text and tool arguments; filter at the collector if any turn up.
- Add a wrong-party persona, such as a relative who answers, to the gold set and run the ordered disclosure check on it.
- Check your fields against the voice playbook.
NOTE
What will date on this page. LiveKit Agents field names (llm_node_ttft, tts_node_ttfb, lk.pii.*) and allow_pii, checked against livekit-agents 1.8.13 on 2026-11-20: re-check quarterly and at each SDK upgrade. Pipecat span content and its missing content switch, as of 2026-11-20. Hamming's 42% (guide dated 2025-12-30) and the Carnegie Mellon preprint (v1, September 2026; look for a reviewed version). ICO biometric guidance, under review after the Data (Use and Access) Act 2025 (2026-11-20). Regulation F counting rules, read in the eCFR on 2026-11-20.