Tarazu's first structured call review read 40 calls from transcripts and found 11 failures; relistening to the same 40 calls found 7 more, every one of them audio-only, including a T02-type 900 ms talk-over and a T01-type "dhai" written down as "do". The method that came out of that week: listen first, note the first failure per call, group notes into a versioned taxonomy, and stop only when new calls add no new modes.
What happened in the first review?
Tarazu pulled 120 outbound collections calls for its first error analysis. The first 40 were reviewed from the STT transcript and the TTS input text, and 11 failures were found. Relistening to the same 40 found 7 more, all of them audio-only. The other 80 calls were listened to from the start.
One of the seven was a call like T01. In the transcript view the call looks clean, because the STT text agrees with itself:
t_ms ch event / text
14100 agent TTS: "Rahul ji, aapka overdue chaar hazaar do sau pachaas rupaye hai. Aap kab tak bhar paayenge?"
18220 caller vad_start
caller REF: "haan main dhai hazaar de dunga bees tarikh ko, baaki next month"
caller STT: "haan main do hazaar de dunga bees tarikh ko baaki next month"
21940 caller vad_end
22310 agent eot_commit (end-of-turn delay 370 ms)
22990 agent llm first token (TTFT 670 ms)
23010 agent tool record_promise_to_pay(amount_paise=200000, ptp_date=2026-11-20, readback_confirmed=false)
23740 agent TTS: "Theek hai, toh main do hazaar rupaye bees November ke liye note kar rahi hoon."
28120 caller REF: "haan haan theek hai" (a backchannel; he did not register the changed amount)
The transcript reader saw a borrower promising Rs 2,000 and an agent confirming Rs 2,000; only the audio has "dhai". In a T02-type call the words show the ignored "already paid" claim, but the 900 ms of agent speech over the borrower exists only on the timeline. Diagnosis: the method missed 7 of 18 failures, because the unit under review was a text file and the failures lived in sound and time.
How is error analysis different for calls?
The unit is the call: dual-channel audio plus the timeline, not the transcript. Listen at 1x for the first 30 calls, write one timestamped plain-word note per call on the first failure, and only then look at transcripts, checks and scores. A transcript is a model output, so reading it reviews STT as much as the agent.
The method is Husain and Shankar's, adapted from grounded theory: a domain expert writes free-form notes on traces (open coding), then groups the notes into a failure taxonomy and counts each category (axial coding) 1. The general procedure is the open-coding workflow. Four things change on a phone call.
- The reviewer listens. Talk-over, a false cutoff on a hesitation, a rushed reply over a sob and a misheard amount are invisible or ambiguous in text. Chapter 1 explains why a transcript is not the trace.
- The first 30 calls are at 1x. Pauses are where turn-taking failures live, and a 720 ms hold pause plays as 480 ms at 1.5x. After 30 calls, use 1.5x only on stretches with no caller speech.
- One note per call, about the first failure. Failures cascade. In T01 the misheard amount produced a wrong tool call and a wrong confirmation that the borrower accepted with a backchannel. Coding all three sends three teams after one bug. Code the earliest failure and mention the rest in the note. If two failures land in the same turn, code the one whose fix is mandatory: a policy breach over a delivery problem.
- Notes carry timestamps. "18,220 ms: caller said dhai, STT wrote do" takes the engineer to the evidence in two seconds; "bad call" takes nobody anywhere.
Open coding stays open: do not start from a category list. Hamming, a voice-agent testing vendor, lists seven speech-recognition failure modes it sees in production: noise-driven omissions, substituted intents, formatting drift, truncation and endpointing errors, hallucinated content, accent and dialect variability, and silent regressions. It draws them from its post-incident reviews, QA audits and monitoring across customer deployments, calls the list not exhaustive, and gives no counts or rates for any mode 2. Use such a list after open coding, to ask what you have not seen, never as the picker reviewers choose from. Tarazu reads substituted intents as "haan" and "nahi" flips and formatting drift as rupee amounts and dates; that mapping is Tarazu's, and Hamming's own examples are English.
How many calls should you review?
Start from a pool of about 100 diverse calls, review the first 30 by hand at 1x, review at least 100, then keep going until 25 more calls add no new failure mode. Stratify the pool by flow, language mix, DPD bucket, hour and carrier, and oversample the rare risky paths, or a uniform sample will not contain them.
Those numbers are Husain and Shankar's: a pool of about 100 diverse traces, at least the first 30 annotated by hand, at least 100 reviewed, then onward until new traces stop revealing new failure modes or changing existing ones. They call that point theoretical saturation, a term from Glaser and Strauss (1967); the 100 figure is their heuristic, not a grounded-theory requirement 1 3. Tarazu makes the stop mechanical: after call 100, stop at the first call where the last 25 calls added no mode and changed no definition.
Figure: Synthetic Tarazu data. Cumulative distinct failure modes against calls reviewed. Most modes appear in the first 30 calls; the curve flattens near 110, and the review stops at 135 because the 25 calls after the last new mode added none. A flat curve says the common modes are found, not that rare ones are absent.
Stratify, or the pool lies. A random pull from yesterday is mostly TW-early-nov in Hinglish at peak hours. Stratify by flow (F01 to F10), language mix, DPD bucket, hour and carrier, using dimensional sampling. A 120-call pass gives 10 calls to each of Tarazu's 12 gated segments (3 language groups by 4 flow groups).
Oversample the risky paths. At 1 angry call in 600 connects, a uniform 120-call sample contains even one angry call with probability only 18%, since 1 - (599/600)^120 = 0.181 [C]. The T09 threat ran at about 1 in 24,000 connected calls; a uniform 120-call review sees it with probability 0.5% [C]. Pull angry, complaint, dispute, hardship and window-edge calls in on purpose, and weight counts back before reporting rates.
Know what saturation can and cannot tell you. If a mode occurs on a share p of calls, the chance that 25 calls show it at least once is 1 - (1 - p)^25: 72% at p = 5%, 40% at 2%, 22% at 1% [C]. So "25 calls, no new mode" is strong evidence that nothing at 10% or more is missing (93% detection) and weak evidence about anything under 2%. Saturation finds the modes worth building checks for; it never clears a rare policy failure. That is the job of checks on 100% of calls and the stratified judge sample in production monitoring.
What does a good open-coding note look like for audio?
A good note says what happened in plain words, where on the timeline, and what the caller heard, with no category name. It names the first failure, mentions downstream effects in passing, and records whether the failure was visible in the transcript view. Write it before you look at check results or judge scores.
Four notes from Tarazu's review, one per incident type:
| Trace | Note (as written) | Transcript view |
|---|
| T01 | "18,220 to 21,940 ms: caller says dhai hazaar, STT has do hazaar. Agent writes PTP for Rs 2,000 at 23,010 before any read-back. Caller's 'haan haan' at 28,120 is a backchannel, not a yes to 2,000." | Not visible |
| T02 | "32,160 ms: borrower says already paid on the 3rd by PhonePe while Asha is still reading dues. Asha keeps talking until 33,060 (900 ms over her). Then asks for payment today. No payment status check." | Partly visible |
| T05 | "52,950 ms: caller pauses after 'agle mahine ki'. Agent takes the floor at 53,250 and starts talking at 53,900 about same-day payment; caller had resumed at 53,670 with the date. Date lost under agent audio." | Visible |
| T10 | "Borrower says she lost her job, cries about 4 s. Asha starts 180 ms after her speech ends, over a sob, speaks fast, asks for Rs 1,000 this week. Money request after hardship." | Partly visible |
Each note quotes what the caller heard, not the STT text, and uses the trace's t_ms clock. None carries a code yet: writing POL.HARDSHIP on T10 at this stage would stop the reviewer noticing the pacing problem. The last column measures what transcript-only review misses, call by call, with no separate study. Notes that fail: "STT error" (which word, when?), "agent rude" (said what, how fast?), "bad latency" (which turn?). Latency notes use the latency chapter vocabulary: "platform turn gap 3,990 ms on a tool turn".
How do notes become a taxonomy?
Group the notes by root cause into 4 to 7 categories, give each category codes with written include and exclude rules, then count. Tarazu's tz-fail-v4 has seven categories and 19 codes. A good category's members share a fix owner and a check.
Axial coding is a second pass over the notes, not a new review: cluster them, name each cluster by what broke, and write the boundary cases down. The site convention is 4 to 7 categories; fewer hides distinct fixes, more spreads thin counts across labels nobody can separate.
Worked example: 12 notes, 5 categories, one count. Twelve first-failure notes from one review block, with the code assigned in the axial pass:
| # | Note (shortened) | Code |
|---|
| 1 | T01: dhai heard as do, PTP written early | ENT.AMOUNT |
| 2 | T06: "05/12" read aloud as May twelfth; CRM holds 5 Dec | ENT.DATE |
| 3 | Amount said as "sava lakh", stored as one lakh | ENT.AMOUNT |
| 4 | "nahi" heard as "haan" on the read-back question | ENT.CONFIRM |
| 5 | T02: already-paid claim talked over, plan unchanged | TURN.BARGEIN |
| 6 | T05: agent takes the floor in a 720 ms pause before a date | TURN.CUTOFF |
| 7 | Agent speaks over the caller's pincode digits | TURN.TALKOVER |
| 8 | T03: dues stated to the borrower's wife before verification | POL.DISCLOSE |
| 9 | T10: money request right after "meri naukri chali gayi" | POL.HARDSHIP |
| 10 | T04: 3,990 ms platform turn gap, then a double reply | LAT.TAIL |
| 11 | Tool turn with no filler, caller says "hello?" | LAT.TAIL |
| 12 | T08: STT locks to English mid-utterance, three clarifications | STT.LID |
Counts by category: ENT 4, TURN 3, POL 2, LAT 2, STT 1 [C]. The top three categories hold 9 of 12 notes, 75%, inside the 60-80% rule [C]; POL and LAT tie for third, so the 75% holds either way. Note 10 is LAT.TAIL, not LAT.DUP, because the slow turn caused the duplicate; note 12 is STT.LID, not LOOP.CLARIFY, for the same reason. Note 9 shows the same-turn tie-break: the reply is also rushed (TONE.PACE), but the policy breach must be fixed. Two warnings: twelve notes can reorder with the next 25, and frequency alone is the wrong priority. POL has 2 notes and is the category that can stop the campaign, so rank by frequency and severity together and treat any POL code as severity one.
The codebook. Each code gets a row that a second reviewer can apply without asking the first. An excerpt of tz-fail-v4:
| Code | Definition | Include | Exclude | Example trace | First version |
|---|
| ENT.AMOUNT | An amount was misheard, misspoken or mis-stored | STT substitution on an amount; TTS reads a different amount; tool args differ from what was confirmed | Amount right but recorded without a closed read-back (a C12 check failure, not an entity error) | T01 | Before v4 |
| TURN.CUTOFF | Agent takes the floor during a hold pause | Pause mid-sentence, before a date or amount | Caller finished and the agent replied fast | T05 | v4 |
| TURN.BARGEIN | Missed or unrecovered barge-in | Agent keeps speaking over new information; agent stops but does not revise its plan | Backchannel ("haan haan") during agent speech | T02 | v4 |
| LAT.TAIL | Turn gap over budget | Platform turn gap over the campaign SLO for that turn type | Duplicate reply after a normal gap (LAT.DUP) | T04 | Before v4 |
| LOOP.CLARIFY | More than two clarifications on one slot | Third clarification on the same amount or date | Clarifications on different slots | T08 | v4 |
Write the exclude column first for a new code. Disagreements cluster at boundaries (barge-in versus backchannel, cutoff versus fast reply), and the exclude column settles them once instead of every week.
How much do transcripts miss?
Too much for transcript-only review. Hamming, a testing vendor, says 42% of 1,000 production failures it reviewed were voice-specific (December 2025, no published method). A Carnegie Mellon preprint found 33% of failed simulated calls failed only on audio events. Tarazu's illustrative review missed 7 of 18.
CAUTION
Vendor-reported. Treat it as a hypothesis to test on your own calls.
Hamming, a voice-agent testing vendor, says that when it reviewed 1,000 production voice-agent failures, 42% were voice-specific issues (latency, speech recognition errors, interruptions, noise) that transcript-only analysis would miss. It publishes no method for that review, and elsewhere it states the figure differently, as text-based tools missing about 40% of voice-specific failures 4. The two statements have different denominators; quote the 42%-of-1,000 version and do not round it.
In a Carnegie Mellon preprint (Baali and Raj, Inquesto Score v0.1, arXiv, September 2026), 33% of failed calls across 13 configurations of one reference billing-support voice agent failed only on audio-derived events such as talk-over and delayed replies. A transcript-only evaluation would have passed them and scored the agents 24 points higher on average. The calls were simulated English calls in a controlled setup (30 scenarios, three acoustic conditions, four US/UK speaker groups, 306 calls per configuration), not production traffic 5. It is not peer reviewed, and it says nothing about Hinglish or Indian telephony.
Tarazu's own number from the opening review is illustrative: 7 of 18 failures were audio-only, 38.9%, with a Wilson 95% interval of 20.3% to 61.4% [C]. Seven of the 40 calls, 17.5%, carried a failure the transcript review passed [C]. Three things matter more than any one percentage.
- The share depends on the mix. Tight silence budgets and loud kitchens produce more audio-only failures than an inbound balance line. Measure yours with the transcript-view column.
- "Partly visible" is the dangerous class. In T02 and T10 a transcript reviewer codes the visible part and files the call under the wrong owner. T10 read as text is a policy problem; listened to, it is also a pacing problem no transcript judge will score.
- The miss is systematic. Transcript review loses whole codes (talk-over, cutoffs, self-consistent STT misses), concentrated in TURN and ENT, so a taxonomy built from transcripts gets the wrong top three.
The staff engineer owns the review tool and the cadence as infrastructure; a bilingual collections expert owns the quality call. Tarazu runs a weekly 25-call stratified top-up, a full pass of 100 or more calls every 2 to 4 weeks, and two reviewers coding 30 shared calls for kappa.
Error analysis that depends on someone's spare afternoon stops in the second month. Run it like a build pipeline, with a tool, a schedule, a queue and an owner.
The listening tool. The general case is the data viewer. For calls it needs:
- a waveform with agent and caller lanes on the trace clock, VAD spans and the end-of-turn commit drawn on it;
- STT and TTS input text aligned to the lanes, the REF lane hidden until the note is saved;
- jump-to-turn, so a check pointing at turns 6 and 7 opens there;
- a note box that stamps
t_ms itself, and a code picker that unlocks only after the note;
- check results and judge scores hidden until the note is saved; a judge's "empathy: pass" shown before T10 is heard anchors the reviewer.
Recordings are PII; the compliance chapter owns access rules.
The cadence (Tarazu policy). Weekly, a 25-call stratified top-up: two from each of the 12 gated segments plus one from the risk strata. Every 2 to 4 weeks, and after any prompt, STT, turn detector or TTS change, a full pass of 100 or more. Each cycle, both reviewers code the same 30 calls.
What it costs. In the course's cost model a bilingual reviewer works at 2.5x real time, so a 150 s call takes 6.25 minutes and Rs 62.50 [C]. The weekly top-up is about 2.6 reviewer-hours (Rs 1,562.50), a 120-call full pass 12.5 hours (Rs 7,500), and the 30 shared calls 3.1 more hours (Rs 1,875) per cycle [C]. Unbudgeted, the top-up is the first thing dropped in a busy week.
Two-reviewer agreement. Compute Cohen's kappa on the first-failure category of the 30 shared calls, with "no failure" as a label, and again at code level, which will be lower. The site's bands are in inter-rater agreement: below 0.4 the codebook is broken, 0.4 to 0.6 is borderline, 0.6 to 0.8 is usable, above 0.8 is well specified. Those cutoffs are a judgment-based convention, and kappa shifts with class prevalence 6; treat 0.6 as team policy. Thirty calls give a wide interval, so report a bootstrap interval resampled over calls and read every disagreement; the fix is usually a new exclude rule, not a retrained reviewer. Reviewer agreement is not judge validation, which audio judges covers.
How do you version a taxonomy without breaking trend lines?
Treat the taxonomy as a release with a version, a change log and a mapping table. Splits, merges and renames get new code ids; retired ids are never reused; every label records the taxonomy version it was coded under. Trend lines are then drawn only across versions the mapping table can bridge.
Tarazu's v3 to v4 log has three entries: TURN split out of LAT, LOOP.CLARIFY added after T08, and LAT.INTERRUPT retired and mapped to TURN.TALKOVER and TURN.BARGEIN. Each change type has a rule.
| Change | Example | What happens to old labels | Trend line |
|---|
| Add | LOOP.CLARIFY (after T08) | None carry it; earlier loops sit under other codes or none | Starts at v4; no back-fill without relabeling |
| Split | LAT.INTERRUPT -> TURN.TALKOVER + TURN.BARGEIN | Old id retired; label keeps v3 and the old id | The sum is continuous; each part starts at v4 |
| Merge | Two codes reviewers could not separate | Both old ids map to the new id | Continuous, as the sum of the old ids |
| Rename | Wording change, same boundary | Keep the id; change only the name | Continuous |
| Boundary change | Exclude rule moves a case to another code | Same id, but counts are not comparable | Break the line; annotate the chart |
The rules behind the table:
- Ids are permanent. If LAT.INTERRUPT came back in v6 meaning something else, every query over v3 labels would silently mix two definitions.
- Labels carry the version. The label record stores the taxonomy version beside the code, so a dashboard can refuse to add counts the mapping table does not bridge.
- Splits do not back-fill. Under v4, the v3 LAT.INTERRUPT history exists only as the combined TURN.TALKOVER plus TURN.BARGEIN line. Splitting it needs a listening relabel, worth doing for the last full pass only.
- Category trends move too. Because TURN came out of LAT, v3 LAT and v4 LAT measure different things; Tarazu's dashboard shows v3 LAT as v4 LAT plus TURN, labelled.
- Version on a schedule. Cut taxonomy releases after a full pass, never mid-review, or half the cycle becomes uncountable.
What gate would have caught it?
A review-coverage gate: no release ships unless every flow it changes has a listened sample, reviewed by the current codebook, with every new mode either coded or ticketed. It catches discovery failures, the modes nobody has a check for yet. It does not estimate rates.
Tarazu's gate, applied to every release candidate:
- List the flows the change touches: a prompt edit touches every flow the prompt serves; a turn detector threshold touches every state of its campaigns.
- For each touched flow, at least 25 listened calls from canary or audio simulation across its language groups, coded by a reviewer who did not build the change.
- A note that fits no existing code blocks the release until it is coded, ticketed with an owner, or shown not to be a failure.
- Every new code gets a check, a judge candidate or a recorded decision not to check it.
T03 is the case for step 1. Prompt v16 "personalised" the greeting, the regression suite passed because every gold scenario was answered by the borrower, and a wrong-party call disclosed dues to the borrower's wife. A gate that listed F05 as touched by an opener change would have demanded wrong-party calls first.
The interval rule is what keeps the gate honest. If 25 listened calls show no instance of a mode, a one-sided 95% upper bound on its rate is 11.3% (exact; the rule of three gives 3/25 = 12%) [C]. Zero in 25 means "not common", never "fixed". Rates and release thresholds come from checks and the gold set, with the methods in reliability statistics. The audio-only share in one review is a single proportion, so Wilson applies; comparing it across two cycles is a difference of proportions and needs Newcombe's method.
What goes in the decision memo?
The memo records the review program as a decision with an owner: what is reviewed, how often, by whom, what it costs, and what event forces a change. Tarazu's version follows.
Decision. Adopt review program v2: listening-first open coding, one first-failure note per call, taxonomy tz-fail-v4 as the axial frame, transcript-only review retired.
Context. Transcript review missed 7 of 18 failures in the first structured review (illustrative counts), all of them audio-only.
Options considered. (a) Transcript review plus a transcript judge: cheapest, blind to audio-only failures. (b) Listening review of a uniform sample: misses rare paths. (c) Listening review of a stratified pool, risk strata oversampled: chosen.
Evidence. Audio-only share 7 of 18, Wilson 95% 20.3% to 61.4%; 120-call review across 12 gated segments; taxonomy tz-fail-v4; external figures as dated below.
Owner and cadence. Staff engineer, eval platform: tool, queue, taxonomy releases. Collections QA lead: codebook boundaries. Weekly 25-call top-up (about 2.6 reviewer-hours), full pass of 100+ every 2 to 4 weeks and after component changes, kappa on 30 shared calls per cycle.
Tripwires. Any block of 25 calls adds a new code: run a full pass and cut a taxonomy release. Category-level kappa below 0.6: freeze codes and revise exclude rules. A release ships without its coverage sample: escalate to the release owner.
Dated facts. Hamming 42% (blog, December 2025, no method); CMU Inquesto Score 33% (preprint v1, September 2026, simulated English calls).
What to do this week
Pull 40 recent calls stratified by flow and language, listen to them at 1x before reading any transcript, and write one timestamped note per call about the first failure. Mark each note's transcript view. By Friday you will know your own audio-only share and whether your current categories fit.
- Build or borrow the two-lane listening view; hide check results and judge scores until the note is saved.
- Add a taxonomy version field to every stored label, and a mapping row for every retired code.
- Book 30 shared calls for two reviewers and compute kappa at category and code level.
- Put the weekly 25-call top-up on a named person's calendar, hours budgeted.
- Add the coverage gate to the release checklist.
NOTE
What will date on this page. Hamming's 42% of 1,000 failures and its "about 40%" framing (blog posts of December 2025 and February 2026, checked 2026-11-20). The Carnegie Mellon Inquesto Score figure (arXiv preprint v1, 24 September 2026, not peer reviewed; a later version or review may change it). Hamming's seven failure modes (post of 16 December 2025, checked 2026-11-20). Husain and Shankar's 100-trace heuristic and cadence (FAQ, checked 2026-11-20).