Steelyard's first structured call review read 40 calls from transcripts and found 11 failures; listening to the same 40 found 7 more, all of them audio-only, among them a T02-type 900 ms talk-over and a T01-type "two fifty" that the transcript records as "two fifteen". The method that came out of it: listen before reading, note the earliest failure per call, sort the notes into a versioned taxonomy, and stop only when fresh calls add no new modes.
What happened in the first review?
The team pulled 120 outbound collections calls. For the first 40, reviewers read the STT transcript and the TTS input text and logged 11 failures; listening to the same 40 added 7, all audio-only. The other 80 were listened to from the start.
One of the seven matched T01. Read as text, the call has no visible problem, because the STT output never contradicts itself:
t_ms ch event / text
14100 agent TTS: "Ryan, your past-due amount is four hundred twenty-five dollars. When do you think you can pay it?"
18220 caller vad_start
caller REF: "yeah I can do two fifty on the twentieth, rest next month"
caller STT: "yeah I can do two fifteen on the twentieth rest next month"
21940 caller vad_end
22310 agent eot_commit (end-of-turn delay 370 ms)
22990 agent llm first token (TTFT 670 ms)
23010 agent tool record_promise_to_pay(amount_cents=21500, ptp_date=2026-11-20, readback_confirmed=false)
23740 agent TTS: "Okay, so I'm noting two hundred fifteen dollars for November twentieth."
28120 caller REF: "uh-huh, okay" (a backchannel; he did not register the changed amount)
A transcript reader sees Ryan offer $215 and Hazel confirm $215; "fifty" exists only in the recording. In the T02-type call the text shows Sofia Ramirez saying she already paid, but Hazel's 900 ms over her exists only on the timeline. Diagnosis: the review missed 7 of 18 failures because it examined text, and those failures were made of sound and timing.
How is error analysis different for calls?
On a phone agent you review the call, two audio channels on one clock, not its transcript. Play the first 30 calls at normal speed, write one timestamped plain-words note on the first thing that went wrong, and only then open transcripts, checks and scores. The transcript is STT output, so reading it reviews the STT.
The method comes from Husain and Shankar, who adapted it from grounded theory: a domain expert writes free-form notes on traces (open coding), then clusters the notes into a failure taxonomy and counts each cluster (axial coding) 1. The site's open-coding workflow covers the general procedure. A phone call changes four parts of it.
- Reviewers listen instead of reading. A reply that lands on top of the caller, a turn closed during a hesitation, a fast answer over crying and an amount the STT got wrong all look clean or ambiguous on the page. Chapter 1 sets out why the transcript is not the trace.
- The first 30 calls play at 1x. Turn-taking failures sit inside pauses, and at 1.5x a 720 ms hold pause lasts 480 ms. After call 30, speed up only through stretches where the caller is silent.
- Each call gets one note, about its earliest failure. Failures chain. In T01 one misheard word led to a wrong tool call, which led to a wrong confirmation, which the borrower waved through with a backchannel. Coding all three would open three tickets for one defect. Code the first one and describe the knock-on effects inside the note. When two failures share a turn, code the one whose fix is not optional: a policy breach outranks a delivery problem.
- Every note carries a time. "18,220 ms: caller said two fifty, STT wrote two fifteen" puts an engineer on the evidence at once. "Bad call" helps nobody.
Spanish and Spanglish calls add a fifth rule: the reviewer understands both languages. Steelyard's book is 64% English, 18% Spanish-dominant and 14% Spanglish by account preference (the other 4% prefer other languages and go to human agents), so about a third of the pool goes to a bilingual English-Spanish reviewer. An English-only reviewer cannot tell that "aja" [uh-huh] was a backchannel and "eso es" [that's it] a yes, and will miss Hazel saying "estoy seguro" [I am sure, masculine] instead of her persona's "estoy segura" [I am sure, feminine] (TONE.GENDER). A call's language comes from the account preference and the call itself, never from a surname.
Keep open coding open: reviewers start without a category list. Hamming, a voice-agent testing vendor, names seven speech-recognition failure modes it sees in production: noise-driven omissions, substituted intents, formatting drift, truncation and endpointing errors, hallucinated content, accent and dialect variability, and silent regressions. It draws them from post-incident reviews, QA audits and monitoring across customer deployments, says the list is not exhaustive, and gives no counts or rates 2. Use such a list after open coding, to ask what your notes have not shown, never as a menu. Steelyard reads substituted intents as "si" [yes] and "no" flips and formatting drift as dollar amounts and dates like "12/03"; that mapping is Steelyard's, and Hamming's examples are English.
How many calls should you review?
Build a pool of about 100 varied calls, hand-review the first 30 at 1x, review at least 100, then keep going until 25 more calls add no new failure mode. Stratify by flow, language group, DPD bucket, hour, state and carrier, and oversample rare risky paths, which a uniform sample usually misses.
The figures are Husain and Shankar's: about 100 diverse traces, at least the first 30 annotated by hand, at least 100 reviewed, then onward until new traces stop revealing new failure modes or changing existing ones. They call that point theoretical saturation, a term from Glaser and Strauss (1967); the 100 is their heuristic, not a grounded-theory requirement 1 3. Steelyard makes the stop mechanical: after call 100, stop at the first call where the previous 25 added no mode and changed no definition.
Figure: Synthetic Steelyard data. Distinct failure modes found, cumulative, against calls reviewed across the three language groups. Most modes show up inside the first 30 calls; the curve goes flat at call 110, and the review ends at call 135 because the 25 calls after the last new mode added nothing. A flat curve means the common modes have been found. It does not mean the rare ones are absent.
An unstratified pool misleads. A random pull from yesterday is mostly AUTO-early-nov, in English, at peak hours. Stratify by flow (F01 to F10), language group, DPD bucket, hour, state and carrier with dimensional sampling. Steelyard gates releases on 12 segments, 3 language groups (English, Spanish-dominant, Spanglish) by 4 flow groups, so a 120-call pass puts 10 calls in each.
Oversample on purpose. Angry borrowers turn up once in 600 connected calls, so a uniform 120-call sample has only an 18% chance of holding even one: 1 - (599/600)^120 = 0.181. The T09 threat ("send someone out to your house") ran at about 1 in 24,000 connected calls, which a uniform 120-call review would catch with probability 0.5%. Pull angry, complaint, dispute, hardship, stop-request and window-edge calls in on purpose, and reweight before reporting any rate.
Saturation has a blind spot. For a mode that appears on a share p of calls, the chance that a block of 25 calls shows it at least once is 1 - (1 - p)^25. That is 72% at p = 5%, 40% at 2% and 22% at 1%. A clean block of 25 is good evidence that nothing at 10% or more is missing (93% detection) and poor evidence about anything under 2%. Saturation finds the modes worth a check; it never clears a rare policy failure. That is the job of checks on every call and the stratified judge sample in production monitoring.
What does a good open-coding note look like for audio?
A useful note says in plain words what happened, where on the timeline, and what the caller heard, with no category name. It names the earliest failure, mentions what followed in a clause, and records whether a transcript reader could have seen it. It is written before any check result or judge score is opened.
Four notes from Steelyard's review, one for each incident type:
| Trace | Note (as written) | Transcript view |
|---|
| T01 | "18,220 to 21,940 ms: caller says two fifty, STT has two fifteen. Hazel writes the PTP for $215 at 23,010, before any read-back. His 'uh-huh, okay' at 28,120 is a backchannel, not agreement to $215." | Not visible |
| T02 | "32,160 ms: Sofia says 'ya lo pague, el tres, con mi tarjeta' [I already paid it, on the third, with my card] while Hazel is still reading the $130 payment in Spanish. Hazel talks on until 33,060, 900 ms over her, then asks for a payment today. No payment status check." | Partly visible |
| T05 | "52,950 ms: Ryan pauses after 'next month on the'. Hazel commits the turn at 53,250 and starts speaking at 53,900 about paying today; Ryan had already resumed at 53,670 with 'fifth, I get paid then'. The date is lost under Hazel's audio." | Visible |
| T10 | "Daniela says she lost her job and cries for about 4 s, then 'ahorita no puedo pagar nada' [right now I can't pay anything]. Hazel starts 180 ms after her speech ends, over a sob, talks fast, and asks for $100 this week. Money request after hardship." | Partly visible |
Each note quotes the audio, not the STT text, on the trace's t_ms clock, and Spanish goes in as spoken with an English gloss. None carries a code yet: tagging T10 POL.HARDSHIP this early would have stopped the reviewer hearing the pacing. The last column measures, call by call, what transcript-only review misses, with no separate study. Notes that fail: "STT error" (which word, when?), "agent rude" (said what, how fast?), "slow" (which turn?). Latency notes use the latency chapter vocabulary: "platform turn gap 3,990 ms on a tool turn".
How do notes become a taxonomy?
Cluster the notes by root cause into 4 to 7 categories, give each category codes with written include and exclude rules, and then count. Steelyard's gl-fail-v4 has seven categories and 19 codes. A category earns its place when its members share one owner for the fix and one check that catches them.
Axial coding is a second pass over existing notes, not a fresh review: group them, name each group after what broke, and write down the boundary cases. The site convention is 4 to 7 categories; fewer hides distinct fixes, more spreads small counts over labels nobody can separate.
Worked example: 12 notes, 5 categories, one count. Twelve first-failure notes from one review block, each with the code it received in the axial pass:
| # | Note (shortened) | Code |
|---|
| 1 | T01: "two fifty" transcribed as "two fifteen"; PTP written before read-back | ENT.AMOUNT |
| 2 | T06: "12/03" read by the es-US voice as "doce de marzo" [twelfth of March]; CRM holds 3 December | ENT.DATE |
| 3 | Borrower says "mil quinientos" [one thousand five hundred]; STT keeps only "mil" and $1,000 is stored | ENT.AMOUNT |
| 4 | "Uh-uh" (no) to the read-back question transcribed as "uh-huh" and taken as a yes | ENT.CONFIRM |
| 5 | T02: "ya lo pague" [I already paid it] talked over; plan unchanged | TURN.BARGEIN |
| 6 | T05: Hazel takes the floor in a 720 ms pause just before a date | TURN.CUTOFF |
| 7 | Hazel speaks over the caller reading out his ZIP code | TURN.TALKOVER |
| 8 | T03: product, DPD and amount told to the borrower's brother before verification | POL.DISCLOSE |
| 9 | T10: money request straight after "I lost my job" | POL.HARDSHIP |
| 10 | T04: 3,990 ms platform turn gap, then a double reply | LAT.TAIL |
| 11 | Tool turn with no filler; caller says "hello?" | LAT.TAIL |
| 12 | T08: STT locks to English mid-utterance; three clarifications | STT.LID |
By category: ENT 4, TURN 3, POL 2, LAT 2, STT 1. The three largest hold 9 of 12 notes, 75%, inside the 60-80% rule; POL and LAT tie for third, so the 75% holds either way. Note 10 is LAT.TAIL, not LAT.DUP, because the slow turn caused the duplicate; note 12 is STT.LID, not LOOP.CLARIFY, because the loop followed the language lock. Note 9 applies the same-turn tie-break: the reply was also rushed (TONE.PACE), but the policy breach cannot wait. Twelve notes can reorder after the next 25 calls, and frequency alone sets the wrong priority: POL has 2 notes and can halt a campaign, so rank by frequency and severity together and treat every POL code as severity one.
Notes 2, 3 and 5 came from Spanish turns, and only a bilingual reviewer would have written them. A monolingual pass over the same block would have left ENT at 2 and TURN at 2, a four-way tie with no top three worth acting on.
The codebook. Every code gets a row specific enough that a second reviewer can apply it without asking the first. An excerpt of gl-fail-v4:
| Code | Definition | Include | Exclude | Example trace | First version |
|---|
| ENT.AMOUNT | An amount was misheard, misspoken or stored wrong | STT substitution on an amount ("fifty" and "fifteen"); TTS reads a different amount; tool args differ from what was confirmed | Correct amount stored without a closed read-back (a C12 check failure, not an entity error) | T01 | Before v4 |
| TURN.CUTOFF | Agent takes the floor during a hold pause | Pause mid-sentence, before a date or amount, in any language | Caller had finished and the agent answered quickly | T05 | v4 |
| TURN.BARGEIN | Missed or unrecovered barge-in | Agent keeps talking over new information ("ya lo pague" [I already paid it], "wait"); agent stops but does not change its plan | Backchannel during agent speech ("uh-huh", "mm-hm", "aja" [uh-huh], "si si" [yes yes]) | T02 | v4 |
| LAT.TAIL | Turn gap over budget | Platform turn gap over the campaign SLO for that turn type | Duplicate reply after a normal gap (LAT.DUP) | T04 | Before v4 |
| LOOP.CLARIFY | More than two clarifications on one slot | A third clarification on the same amount or date | Clarifications on different slots | T08 | v4 |
| TONE.GENDER | Masculine self-reference in Spanish | "un asistente" [a male assistant], "estoy seguro" [I am sure, masculine] from Hazel | Masculine forms that refer to the caller or a third person | (codebook example) | Before v4 |
For a new code, fill in the exclude column first. Reviewer disagreement gathers at the edges (barge-in against backchannel, cutoff against a quick reply), and a written exclude rule settles each edge once rather than every week.
How much do transcripts miss?
Enough to rule out transcript-only review. Hamming, a testing vendor, says 42% of 1,000 production failures it reviewed were voice-specific (December 2025, no published method). A Carnegie Mellon preprint found 33% of failed simulated calls failed only on audio events. Steelyard's illustrative review missed 7 of 18.
CAUTION
Vendor-reported. Treat it as a hypothesis to test on your own calls.
Hamming says that in a review of 1,000 production voice-agent failures, 42% were voice-specific issues (latency, speech recognition errors, interruptions, noise) that analysis of transcripts alone would miss. It publishes no method for that review, and another of its posts frames the number as text-based tools missing about 40% of voice-specific failures 4. Those two statements divide by different things; quote the 42%-of-1,000 version as written and do not round it.
The Carnegie Mellon preprint (Baali and Raj, Inquesto Score v0.1, arXiv, September 2026) ran 13 configurations of one reference billing-support voice agent; 33% of failed calls failed only on audio-derived events such as talk-over and delayed replies, which a transcript-only evaluation would have passed, scoring the agents 24 points higher on average. The calls were simulated English calls in a controlled setup (30 scenarios, three acoustic conditions, four US and UK speaker groups, 306 calls per configuration), not production traffic 5. The paper is not peer reviewed and contains no Spanish or Spanglish calls.
Steelyard's figure from the opening review is illustrative: 7 of 18 failures were audio-only, 38.9%, with a Wilson 95% interval of 20.3% to 61.4%. Seven of the 40 calls, 17.5%, carried a failure that the transcript review passed. Three points matter more than any of these percentages.
- Your share depends on your mix. Tight silence budgets, speakerphones and TV noise produce more audio-only failures than a quiet inbound balance line. Measure yours with the transcript-view column.
- "Partly visible" is where calls get misfiled. In T02 and T10 a transcript reviewer codes the half they can see and sends the call to the wrong owner. Read as text, T10 is a policy failure. Heard, it is also a pacing failure that no transcript judge scores.
- The miss is not random. Transcript review drops whole codes (talk-over, cutoffs, STT errors that stay self-consistent), and they cluster in TURN and ENT, so a taxonomy built from transcripts ranks the wrong top three. Spanish calls widen the gap: when the STT locks to English, "si, pero" [yes, but] reaches the transcript as "see pedro", and the text alone no longer shows what the borrower meant.
A staff engineer owns the review tool and the schedule as infrastructure; a bilingual collections expert owns what counts as a failure. Steelyard runs a weekly 25-call stratified top-up, a full pass of 100 or more calls every 2 to 4 weeks, and two reviewers coding 30 shared calls for kappa each cycle.
Review that runs on someone's free afternoon stops by the second month, so run it like a build pipeline: a tool, a schedule, a queue and a named owner.
The listening tool. The general version is the data viewer. For calls it also needs:
- a waveform with separate agent and caller lanes on the trace clock, with VAD spans and the end-of-turn commit drawn on them;
- STT text and TTS input text aligned to the lanes, with the REF lane hidden until the note is saved;
- jump-to-turn, so a check that points at turns 6 and 7 opens the audio there;
- a note box that fills in
t_ms by itself, and a code picker that stays disabled until the note is written;
- a language tag on each call in the queue, so Spanish-dominant and Spanglish calls route to bilingual reviewers;
- check results and judge scores kept hidden until the note is saved, because a judge's "empathy: pass" shown before T10 is heard anchors the reviewer.
Recordings carry the borrower's voice and account details, and sometimes a third party's. Access rules for the US, UK and EU books belong to the compliance chapter.
The cadence (Steelyard policy). Every week, a 25-call stratified top-up: two calls from each of the 12 gated segments plus one from the risk strata. Every 2 to 4 weeks, and after any change to the prompt, STT, turn detector or TTS, a full pass of 100 or more calls. Every cycle, both reviewers, bilingual in English and Spanish, code the same 30 calls.
What it costs. The course cost model prices a bilingual English-Spanish reviewer at about $30 an hour working at 2.5x real time, so one 150 s call takes 6.25 minutes and costs $3.125. The weekly top-up comes to about 2.6 reviewer-hours ($78.13), a 120-call full pass to 12.5 hours ($375.00), and the 30 shared calls to 3.1 more hours ($93.75) per cycle. All prices are invented for the fictional world. If nobody budgets the hours, the top-up is the first task dropped in a busy week.
Agreement between two reviewers. Compute Cohen's kappa on the first-failure category of the 30 shared calls, with "no failure" as a label, then at code level, where it will be lower. Look at disagreements by language group too: a codebook can be clear in English and vague in Spanglish. The site's bands are in inter-rater agreement: below 0.4 the codebook is broken, 0.4 to 0.6 is borderline, 0.6 to 0.8 is usable, above 0.8 is well specified. Those cutoffs are a convention based on judgment, and kappa moves with class prevalence 6, so treat 0.6 as team policy. Thirty calls give a wide interval, so report a bootstrap interval resampled over calls and read every disagreement; the fix is usually a new exclude rule, not a retrained reviewer. Agreement between reviewers is not judge validation, which audio judges covers.
How do you version a taxonomy without breaking trend lines?
Ship the taxonomy like a release: a version string, a change log and a mapping table. Splits and merges create new code ids, renames keep theirs, and a retired id is never reused. Every stored label records its taxonomy version, and a trend line spans only versions the mapping table connects.
Steelyard's v3 to v4 change log has three entries: TURN split out of LAT, LOOP.CLARIFY added after T08, and LAT.INTERRUPT retired and mapped to TURN.TALKOVER and TURN.BARGEIN. Each kind of change follows a rule.
| Change | Example | What happens to old labels | Trend line |
|---|
| Add | LOOP.CLARIFY (after T08) | No old label has it; earlier loops sit under other codes or none | Starts at v4; no back-fill without relabeling |
| Split | LAT.INTERRUPT -> TURN.TALKOVER + TURN.BARGEIN | Old id retired; label keeps v3 and the old id | The sum continues; each part starts at v4 |
| Merge | Two codes reviewers could not tell apart | Both old ids map to the new id | Continuous, as the sum of the old ids |
| Rename | Wording change, same boundary | Keep the id; change only the name | Continuous |
| Boundary change | An exclude rule moves a case to another code | Same id, but counts before and after do not compare | Break the line and annotate the chart |
Why the rules look like this:
- An id is permanent. If LAT.INTERRUPT returned in v6 with a new meaning, every query over v3 labels would quietly mix two definitions.
- Every label stores its version. With the taxonomy version next to the code, a dashboard can refuse to add counts that the mapping table does not connect.
- A split does not back-fill. Under v4, v3 LAT.INTERRUPT history exists only as TURN.TALKOVER plus TURN.BARGEIN combined; separating it takes a listening relabel, worth doing for the latest full pass only.
- Categories move too. TURN came out of LAT, so v3 LAT and v4 LAT measure different things; Steelyard's dashboard draws v3 LAT as v4 LAT plus TURN, labeled.
- Releases follow full passes. Cut a taxonomy version after a full pass and never in the middle of a review, or half the cycle cannot be counted.
What gate would have caught it?
A review-coverage gate: a release ships only if every flow it changes has a listened sample, coded under the current codebook, with every new mode either coded or ticketed. It catches discovery failures, the modes no check exists for yet. It does not estimate rates.
Steelyard's gate runs on every release candidate:
- List the flows the change touches. A prompt edit touches every flow that prompt serves; a turn detector threshold touches every call state in its campaigns.
- For each touched flow, at least 25 listened calls from canary traffic or audio simulation, spread across its language groups, coded by a reviewer who did not build the change.
- A note that fits no existing code blocks the release until it is coded, ticketed with an owner, or shown not to be a failure.
- Each new code gets a check, a judge candidate or a written decision not to check it.
T03 shows why step 1 comes first. Prompt v16 "personalised" the greeting. The regression suite passed because the borrower answered every gold scenario. Then a call reached Ryan's brother Kyle, and Hazel told him the product, the days past due and the amount before any identity check. A gate that listed F05 (wrong party) as touched by an opener change would have required wrong-party calls before release.
The interval rule keeps the gate honest. If 25 listened calls show no instance of a mode, the one-sided 95% upper bound on its rate is 11.3% (exact; the rule of three gives 3/25 = 12%). Zero in 25 means "not common", never "fixed". Rates and release thresholds come from checks and the gold set, using the methods in reliability statistics. The audio-only share in one review is a single proportion, so Wilson applies; comparing that share across two cycles is a difference of proportions and needs Newcombe's method.
What goes in the decision memo?
The memo makes the review program a decision with an owner: what is reviewed, how often, by whom, at what cost, and what event forces a change. It also records the options rejected, so the next owner does not reopen transcript-only review without new evidence.
Steelyard's memo for review program v2:
Decision. Adopt review program v2: listening-first open coding, one first-failure note per call, taxonomy gl-fail-v4 as the axial frame, bilingual reviewers for Spanish-dominant and Spanglish calls, transcript-only review retired.
Context. In the first structured review, transcript reading missed 7 of 18 failures (illustrative counts), all of them audio-only, including a "two fifty" stored as $215.
Options considered. (a) Transcript review plus a transcript judge: cheapest, and blind to audio-only failures. (b) Listening review of a uniform sample: misses the rare paths. (c) Listening review of a stratified pool with the risk strata oversampled: chosen.
Evidence. Audio-only share 7 of 18, Wilson 95% 20.3% to 61.4%; 120-call review across 12 gated segments; taxonomy gl-fail-v4; external figures as dated below.
Owner and cadence. Staff engineer, eval platform: tool, queue, language routing, taxonomy releases. Collections QA lead: codebook boundaries. Weekly 25-call top-up (about 2.6 reviewer-hours, $78.13), full pass of 100+ every 2 to 4 weeks and after component changes, kappa on 30 shared calls per cycle.
Tripwires. Any block of 25 calls adds a new code: run a full pass and cut a taxonomy release. Category-level kappa below 0.6: freeze codes and revise exclude rules. A release ships without its coverage sample: escalate to the release owner.
Dated facts. Hamming 42% (blog, December 2025, no method); CMU Inquesto Score 33% (preprint v1, September 2026, simulated English calls).
What to do this week
Pull 40 recent calls stratified by flow and language group, listen to each at 1x before opening its transcript, and write one timestamped first-failure note per call. Mark each note's transcript view and route Spanish and Spanglish calls to a bilingual reviewer. By Friday you will know your audio-only share.
- Build or borrow the two-lane listening view, with check results and judge scores hidden until the note is saved.
- Add a taxonomy version field to every stored label, and a mapping row for every retired code.
- Book 30 shared calls for two reviewers, covering all three language groups, and compute kappa at category and code level.
- Put the weekly 25-call top-up on a named person's calendar, with the hours budgeted.
- Add the coverage gate to the release checklist.
NOTE
What will date on this page. Hamming's 42% of 1,000 failures and its "about 40%" framing (blog posts of December 2025 and February 2026, checked 2026-11-20). The Carnegie Mellon Inquesto Score figure (arXiv preprint v1, 24 September 2026, not peer reviewed; a later version or a review may change it). Hamming's seven failure modes (post of 16 December 2025, checked 2026-11-20). Husain and Shankar's 100-trace heuristic and cadence (FAQ, checked 2026-11-20).