On trace T01, Ryan Mitchell told Hazel he could pay "two fifty", the speech-to-text (STT) output read "two fifteen", and Hazel stored a promise to pay of $215.00 without reading it back. The utterance had twelve words and one of them was wrong, a word error rate (WER) of 8.3%, while the one number the call existed to capture was 100% wrong.
What happened on the call?
The STT heard "fifteen" where Ryan said "fifty", produced a fluent sentence with no visible flaw, and Hazel booked the promise in the CRM before any read-back. A reviewer reading only the STT text and Hazel's replies sees nothing wrong; the error lives in the audio and the human reference lane.
t_ms ch event / text
14100 agent TTS: "Ryan, your past-due amount is four hundred twenty-five dollars. When do you think you can pay it?"
18220 caller vad_start
caller REF: "yeah I can do two fifty on the twentieth, rest next month"
caller STT: "yeah I can do two fifteen on the twentieth rest next month"
21940 caller vad_end
22310 agent eot_commit (end-of-turn delay 370 ms)
22990 agent llm first token (TTFT 670 ms)
23010 agent tool record_promise_to_pay(amount_cents=21500, ptp_date=2026-11-20, readback_confirmed=false)
23740 agent TTS: "Okay, so I'm noting two hundred fifteen dollars for November twentieth."
28120 caller REF: "uh-huh, okay" (a backchannel; he did not register the changed amount)
Diagnosis: one substituted word, utterance WER 1/12 = 8.3%, amount accuracy 0 of 1, and a C12 failure because the write preceded the read-back. The CRM row says 21500 cents for 20 November against a gold value of 25000, so the promise is $35.00 short and the kept check will later mark it broken. The fields in this excerpt are explained in what a voice agent trace contains.
Why can WER look fine while the entities are wrong?
WER gives every word one vote, and the words that carry dollars, dates and consent are a thin slice of any call, so an STT system can get all of them wrong and still post a good average. The filler, verbs and names that dominate the denominator decide the score; the money barely registers in it.
CAUTION
Vendor-reported. Treat it as a hypothesis to test on your own calls.
Soniox, an STT vendor, makes the point with an illustration, not a measurement: when phone numbers are 10 of a transcript's 1,000 words, missing every one of them costs 1 point of WER, and the transcript can read as 97% accurate with 0% of phone numbers right. Exact match makes it starker. A single wrong digit in the only phone number on a call drops phone-number accuracy to 0% and moves WER by 0.1 points 1.
T01 is that illustration at the scale of one turn. "Fifteen" for "fifty" is one substitution. So is "yeah" for "yep". WER cannot tell a $35.00 error from a change of register.
Cresta, a contact-center AI vendor, describes its STT evaluation in four layers: lexical (WER, character error rate (CER), sentence error rate), entity (accuracy on business-critical tokens such as phone numbers, emails, amounts and dates, plus keyword recall), semantic (whether the meaning survives) and task (downstream behavior such as agent success rate, p95 latency and interruption rate). To compare systems it applies hard constraints, then compares WER, and breaks ties on entity, worst-slice and task metrics 2. A collections agent should swap the middle: hard constraints first, entity error rate second, WER only to separate systems that tie on entities. Every amount and date Hazel hears either becomes a CRM write or triggers a read-back, so entities are the product. The voice playbook lists named-entity WER as a secondary STT metric; for an agent that writes money it is the primary one.
What is entity error rate?
Entity error rate for one entity type is the share of gold entities in the reference that the STT hypothesis fails to recover exactly, after both sides are canonicalized. It is computed per type (amount, date, loan tail, confirmation) and never pooled, because each type fails and costs differently.
Each clause in that definition settles an engineering choice:
- The gold entities come from a person reading the reference. An annotator marks every amount, date and yes or no the speaker meant, with its canonical value. Before you score any vendor, run your own extractor over the reference text and confirm it reproduces every gold value. A mismatch at this stage is a bug in the canonicalizer or in the labels; if it stays, the STT gets blamed for a gap in the scorer.
- One canonicalizer for both sides. "Fifteen hundred", "1,500" and "mil quinientos" [one thousand five hundred] are all 150000 cents. Comparison happens on canonical values, never on strings.
- Exact match, no partial credit. $250.00 against $215.00 is two characters apart; $250.00 against $2,500.00 is one inserted digit and ten times the money. Edit distance on digit strings scores a near miss as nearly right when it is simply wrong. If you want severity, report it beside the rate as the absolute dollar error and its direction (short or over).
- Match as multisets, per utterance and type. Recovered is the size of the intersection of gold values and hypothesis values. Missed is gold minus recovered. Hallucinated is hypothesis minus recovered.
From those counts come two rates. Entity error rate is the recall side: missed gold entities over all gold entities of the type. The precision side is the hallucinated entity rate: hypothesis entities with no gold match over all hypothesis entities. A substituted amount counts in both, a dropped amount only as missed, an invented amount only as hallucinated. For collections the invented amount is the worse error, because an agent that hears a number nobody said will try to book it. Report missed and hallucinated counts next to each other for every type.
Value matching ignores what role a value plays. In "two hundred now and fifty next week" the STT can recover both values while the agent books the wrong one as today's payment. That error belongs to the task check, which compares the CRM after the call with the gold end state.
What does scoring eight utterances by hand show?
Scoring eight Steelyard utterances both ways shows that the two metrics disagree about which rows are bad: the two lowest-WER rows each lost an entity, and the highest-WER row lost none. A review queue sorted by WER would open on the rows that did no harm.
The WER column uses lowercasing and punctuation stripping (hyphens become spaces) with no numeral canonicalization; the text below is shown after that step. Dates resolve against the call date, 2026-11-14 (a Saturday), except row 8, which is the first turn of T08 on 2026-11-16 (a Monday). Rows 3 and 7 are Spanish; row 8 is Spanglish.
| # | Reference | STT hypothesis | WER | Gold entities | Entity result |
|---|
| 1 | yeah i can do two fifty on the twentieth rest next month | yeah i can do two fifteen on the twentieth rest next month | 1/12 = 8.3% | amount 25000; date 2026-11-20 | amount missed (21500 heard) |
| 2 | one thousand five hundred by the fifth | 1500 by the fifth | 4/7 = 57.1% | amount 150000; date 2026-12-05 | both recovered |
| 3 | no es correcto [it is not right] | es correcto [it is right] | 1/3 = 33.3% | confirmation: no | polarity flipped to yes |
| 4 | ill pay on the eighth after my check clears | ill pay on the eighteenth after my check clears | 1/9 = 11.1% | date 2026-12-08 | date missed (2026-11-18 heard) |
| 5 | give me one second please | give me one fifty please | 1/5 = 20.0% | none ("one second" asks for a pause) | 1 hallucinated amount, 15000 |
| 6 | last four digits four four one seven | last four digits four four seven one | 2/7 = 28.6% | loan tail 4417 | missed (4471 heard) |
| 7 | el cinco de diciembre [the fifth of December] | el 5 de diciembre | 1/4 = 25.0% | date 2026-12-05 | recovered |
| 8 | si pero my paycheck comes el viernes so after that i can pay like the twenty third [si pero: yes but; el viernes: on Friday] | see pedro my paycheck comes el bee air ness so after that i can pay like the twenty third | 5/17 = 29.4% | confirmation: yes; dates 2026-11-20, 2026-11-23 | dates 1/2 (Friday lost); confirmation missed ("see") |
Corpus WER is total errors over total reference words: 16/64 = 25.0%. The same eight rows by entity type:
| Type | Gold | Missed | Entity error rate | Hallucinated |
|---|
| Amount | 2 | 1 | 50% | 2 (rows 1, 5) |
| Date | 6 | 2 | 33.3% | 1 (row 4) |
| Loan tail | 1 | 1 | 100% | 1 (row 6) |
| Confirmation | 2 | 2 | 100% | 1 (row 3) |
Row 2 has the worst WER and a perfect entity score: the STT wrote the amount in digits, which WER charges as one substitution and three deletions and the canonicalizer reads as exactly $1,500.00. Row 7 is the same story for a date in Spanish. Rows 1 and 4 share the T01 shape, a "-ty" or "-th" word swapped for its "-teen" neighbor at a WER near 10%. Row 5 is the one most scorers get wrong twice: "one second" is a request to wait, so a canonicalizer must not read "one" as an amount or "second" as a date, and the substitution then creates "one fifty", which the amount shorthand turns into $150.00 that nobody offered. Row 8 shows where T08's loop began: "viernes" [Friday] dissolved into three English syllables, taking the payday date with it, and "si" [yes] became "see".
Eight rows say nothing about a vendor. What they show is that the two metrics order utterances differently, so you compute both on every run. The entity error rate recipe runs the same scorer on 1,200 seeded utterances, and reference-based metrics covers the general case.
How do you canonicalize English and Spanish amounts and dates?
Convert every form to one value: amounts to integer cents, dates to ISO dates resolved against the call date in the borrower's time zone, digits to digits, yes and no to a polarity. Parse numbers with an explicit locale, version the rules, test them on a fixed list of forms, and run the same code on both sides.
Steelyard's canonical forms, from its world bible, with the rule that breaks most often:
| Entity type | Spoken examples | Canonical form | Rule that bites |
|---|
| Amount | "two fifty", "two hundred fifty", "$250", "doscientos cincuenta" (all 25000); "four twenty-five", "cuatrocientos veinticinco" (42500); "a hundred twenty thousand" (12000000) | Integer cents | Shorthand: in an amount slot, "X fifty" or "X fifteen" with X from one to nine is X hundred fifty or X hundred fifteen dollars, and the two are one sound apart, so either is always read back |
| Date | "the twentieth", "November twentieth", "el veinte de noviembre" [the twentieth of November], "next Friday", "el viernes" [Friday], "after payday, the twenty-third", "12/03" | ISO date, resolved against the call date | "12/03" is month-first in en-US and day-first in es-US |
| Loan tail | "ending 4417", "four four one seven", "cuatro cuatro uno siete" [four four one seven] | 4 digits | Order matters; the "four" in "last four digits" is not part of the tail |
| Payment confirmation number | 12 digits, read in groups | String of digits | Group boundaries are not digits |
| Confirmation | "yes", "yeah", "that's right", "okay", "si" [yes], "eso es" [that's it], "no", "no, no" | Yes, no or backchannel | "uh-huh", "mm-hm" and "aja" [uh-huh] are backchannels unless they answer a closed read-back question |
| Name | "Ryan Mitchell", "Mr. Mitchell", "Sofia" | Normalized first and last name | Titles such as "Mr." are stripped |
| ZIP code | "eight five seven one nine", "85719" | 5 digits | Score it as its own entity, separate from the address |
| Phone | "five two zero, five five five, zero one nine nine", "(520) 555-0199" | 10 digits | Spoken grouping is not part of the value |
| Email | "advocate at steelyard dot example" | Lowercased address | "at" and "dot" are spoken symbols |
| Address | "1200 Example Avenue, Tucson, AZ 85701" | Normalized tokens; ZIP exact | The street number and the ZIP are both digit strings; keep them apart |
A generic number parser handles "one thousand five hundred" and little else a borrower says. US amounts need the shorthand forms ("two fifty", "fifteen hundred", "four twenty-five"), Spanish number words, which fuse tens and units into single words ("veinticinco" for 25, "quinientos" for 500), and locale-aware separators. The separators are the trap that passes code review. In Unicode CLDR 48 data, es-US, es-MX and es-419 use a period for decimals and a comma for grouping, the same as English, while Spain uses a comma decimal and a period group and leaves four-digit numbers ungrouped by default; the string "1.234" means about one in Mexico and one thousand two hundred thirty-four in Spain 3. The Real Academia Espanola's rules accept either decimal sign depending on country, so text from different regions can write the same amount differently and a normalizer has to parse amounts rather than compare strings 4.
The test that pins all of this is a fixed list. Steelyard's canonicalizer must map all eight forms of $1,500.00 to 150000 cents, and it fails its own test if any one does not:
1 fifteen hundred
2 one thousand five hundred
3 1,500
4 1500
5 $1,500.00
6 mil quinientos
7 mil quinientos dolares
8 1.500 locale es-ES -> 150000 cents
1.500 locale es-MX -> 150 cents ($1.50): same string, other locale
The last pair is the point of the list. A canonicalizer that ignores the locale tag gets one of the two wrong, and which one depends on which country its author grew up in. Steelyard's Spanish calls are es-US, but STT output and LLM text can carry either convention, so the locale has to come from the call record, never from the string.
Spanish words and digits also differ before any locale question arises. Whisper's basic normalizer, the one used for non-English text, does not standardize numbers, so "quince" against "15" counts as an error unless you add rules 5. That is fine for WER as long as the policy is written down; for entities it is the reason canonicalization runs before matching. The full scoring policy for accents, numerals and mixed-language references belongs to normalization and code-switching in multilingual STT.
The canonicalizer is also a model that can be wrong. Sproat and Jaitly found that neural text normalizers reach high overall accuracy yet still make errors that would give a speech application the wrong message 6, and Zhang et al. used finite-state covering grammars to block misreadings of that kind 7. The lesson carries over directly: token accuracy hides the errors that matter, so count message-changing canonicalizer errors separately and treat each as a release blocker.
Dates carry more policy than amounts. Write each decision down:
- Relative dates resolve against the call date in the borrower's time zone, taken from the trace's local start time and never from the scoring machine's clock. "The twentieth" on 2026-11-14 is 2026-11-20; "the fifth" on the same day is 2026-12-05, because 5 November has passed.
- "Next Friday" needs a rule. Steelyard reads it as the first Friday after the call date, 2026-11-20 for a Saturday call. Some speakers mean the Friday of the following week. Pick one reading, version it, and have the agent ask when the difference matters.
- Numeric dates follow the locale of whoever wrote them. "12/03" written by an en-US stack is 3 December; read by an es-US voice it is the twelfth of March, which is T06, covered in evaluating speech output for agents.
When the reference itself is ambiguous ("two or three hundred"), the gold label is "ambiguous", the utterance leaves the amount denominator, and the correct agent behavior is one clarifying question, scored in the conversation checks.
Pin the canonicalizer by hash on every number you report, the same way you pin the WER normalizer. A rule change that moves entity error rate by a point is a scorer release, not an STT improvement.
Why do yes and no need their own metric?
A confirmation is usually one or two words, so a single dropped or swapped word reverses it while WER barely moves, and short utterances are where STT is weakest. Score confirmation polarity as its own entity type and report the two directions of error separately, because they cost different things.
Row 3 above is the plain case: "no es correcto" [it is not right] lost its first word and became "es correcto" [it is right]. In a 1,200-utterance set that is one error among thousands of words, and the polarity was the entire content of the turn.
The two directions:
- No heard as yes turns a refusal into consent. After the read-back, Hazel books the promise, the C12 order check passes because she did hear a yes, and the CRM holds a promise the borrower declined. This is the expensive direction and the one to gate.
- Yes heard as no costs an extra clarifying turn. Repeated, it trips the clarification-loop rule (more than two clarifications on one slot, code LOOP.CLARIFY). T08 shows that loop, though its trigger was a lost date ("viernes"), not a flipped yes.
Spanish adds two traps that a polarity lexicon has to handle. The reference convention keeps accents; this page stays ASCII, so the block below spells the accented yes as s-i-acute:
REF s-i-acute, eso es gloss: yes, that's it polarity: yes
HYP si eso es accent dropped by the STT polarity: yes (accent-folded match in answer position)
REF si puedo, pago el viernes gloss: if I can, I'll pay Friday polarity: none ("si" without the accent is "if")
REF son doscientos, no? gloss: it's two hundred, right? polarity: none (a tag question, not a refusal)
An extractor that maps every "si" to yes and every "no" to no scores the last two rows wrong in both directions. Read polarity only from a turn that answers a closed question, and test the lexicon on references before you charge any error to the STT.
Two things that look like confirmation errors are not STT errors. The first is a backchannel. In T01, "uh-huh, okay" at 28,120 ms answered a statement, not a question, so it confirms nothing, even though "okay" is on the yes list. Separating a yes from a backchannel takes the previous turn and the timing, so that label belongs to turn-taking, barge-in and recovery; the STT metric scores polarity only. The second is a lexicon gap. A yes form missing from the list ("yep", "sure", "claro" [of course]) is scored as a missed entity and blamed on the STT.
Negation hides inside amounts too. "Not four hundred, three hundred" contains two amount strings and one gold amount, $300.00; so does "cuatrocientos no, trescientos" [not four hundred, three hundred]. Annotate the negated value as negated, so an extractor that grabs the first number fails its own reference test instead of passing quietly.
Which slices matter for collections?
Report entity error rate by entity type and surface form, utterance length, negation, language group, code-switch density, carrier and accent region, and gate on the worst slice a campaign depends on. A system that is excellent on digits and weak on "two fifty" has a good average and a broken auto-loan campaign.
- Entity type and surface form. Split amounts into digits, English number words, shorthand ("two fifty", "fifteen hundred") and Spanish number words. In the recipe's synthetic bake-off, the weaker system's amount entity error rate was 2.9% on digit-form amounts, 7.1% on shorthand and 21.1% on Spanish number words; the error injector was written that way, so the ordering says nothing about real systems. The habit of splitting is the lesson.
- Utterance length. Confirmations and short amount answers fill the shortest bin. Voice of India (Bhogale et al., arXiv 2604.19151 v4, 3 July 2026, accepted at Interspeech 2026) is a closed benchmark of unscripted telephonic conversations in 15 languages: 306,230 utterances, 536 hours, 36,691 speakers. Averaged across languages, Amazon's STT error rate rises from 10.45% on utterances longer than 5 s to 18.74% on utterances under 2 s, and Microsoft's from 10.90% to 18.59%; the authors attribute it to limited semantic context. The paper states no sample rate or carrier codec, so these are not 8 kHz PSTN figures, and they are not English or Spanish; treat them as a reason to build a short-utterance bin, not as a number for your calls 8.
- Negation and self-correction. Utterances with "no" or "not" near an entity, and corrected amounts, are a slice of their own.
- Language group. English, Spanish-dominant and Spanglish, the three groups Steelyard gates on. The group comes from the account preference and the call itself, never from a surname. In the same synthetic bake-off, the weaker system's amount entity error rate was 2.4% for English speakers, 6.9% for Spanish-dominant speakers and 15.7% for Spanglish speakers, against 6.1% overall: the mean described none of the three groups.
- Code-switch density. Low, medium and high switching per utterance; how to score it is owned by the normalization chapter. Row 8 is a high-density utterance, and its losses sat exactly on the Spanish words.
- Carrier and codec chain. A mobile leg through AMR-NB and then G.711 is not the same audio as a 16 kHz app recording. Building that test set is covered in testing on telephony audio.
- Accent region. For telephone Spanish, Fisher Spanish Speech (LDC2010S01) is the closest public match: about 163 hours, 819 conversations from 136 native speakers in the continental US and Puerto Rico (91 non-Caribbean, 45 Caribbean), recorded as 8 kHz mu-law two-channel audio. It is licensed through LDC, not free 9. Its speaker split is a template for slicing your own gold set by region.
A slice with few gold entities gives an interval too wide to act on. Mark it "insufficient" rather than print a point estimate that reads like a finding, and grow the gold set until the interval is narrower than the tolerance the decision needs.
How many utterances do you need, and what interval?
Bootstrap by resampling speakers, not utterances, because a speaker's accent, handset and pace are shared across their utterances and their errors cluster. Size the set by gold entities of the gated type, not by utterances, and expect a per-segment gate to need several hundred gold amounts per segment.
A Wilson interval over utterances treats every utterance as an independent draw. In a gold set of 1,200 utterances from 180 speakers that is false: a speaker whose "fifty" comes out as "fifteen" once will do it again. The speaker-clustered bootstrap:
- For each speaker, total the missed gold entities of the type and the gold entities of the type.
- Draw speakers with replacement, as many as there are, 1,000 or more times.
- For each draw, compute the pooled ratio: total missed over total gold.
- Take the 2.5th and 97.5th percentiles as the 95% interval.
The general method, including the design effect DEFF = 1 + (m - 1) x rho for clusters of size m and within-cluster correlation rho, is in clustered standard errors and in Cameron and Miller's practitioner's guide 10.
The entity error rate recipe generates two seeded synthetic sets in the shape of Steelyard's gold-v7 STT set and computes the interval on both (seed 20261120). For the system with the higher amount entity error rate:
| Set | Amount entity error rate | Speaker-clustered 95% interval | Width |
|---|
| 400 utterances, 60 speakers | 5.4% | [2.0%, 9.1%] | 7.1 points |
| 1,200 utterances, 180 speakers | 6.1% | [4.1%, 8.2%] | 4.0 points |
From the smaller set you cannot tell whether that system's amount entity error rate is 2% or 9%. Here the clustered intervals are wider than the utterance-level ones (7.1 against 5.8 points, and 4.0 against 3.5), because each speaker contributes about 4 amounts (765 over 180 speakers) and keeps one language group and one difficulty level across them. In production samples, a borrower who calls back every few days contributes dozens of utterances, and that is where the utterance-level interval goes wrong.
For planning, the normal approximation gives a rough count: n is about 1.96^2 x p(1 - p) / h^2 x DEFF gold entities for a half-width h. At p = 2% and h = 1 point that is 753 gold amounts before the design effect. Split 765 amounts evenly across English, Spanish-dominant and Spanglish and each group holds 255, which widens the half-width at 2% to about 1.7 points. Gating per segment multiplies the gold set, so decide which segments gate and which are only monitored before you pay for transcription.
Comparing two STT systems is a different question from bounding one. Score both on the same utterances and bootstrap the paired difference by speaker, as in bootstrap and paired tests; two intervals that overlap are not that test.
Figure 1: WER and amount entity error rate pick opposite winners. Synthetic Steelyard data, shaped like the gold-v7 STT set, from the entity error rate recipe (1,200 utterances, 180 speakers, seed 20261120): system B has the lower WER and about six times the amount entity error rate, and it flips 6 confirmations to A's 5. Choose by WER and you ship the system that books more wrong promises.
What gate would have caught it?
Two checks: one before release, one per call. Offline, an entity error rate gate on gold-v7 (amount, date, confirmation; per language group) blocks any STT update whose speaker-clustered upper bound is over the campaign threshold. Online, C12 fails any promise written before a closed read-back and an explicit yes.
The offline gate:
gate: stt-entities
gold_set: gold-v7 # STT set: 1,200 reference-transcribed utterances, 180 speakers
scorer:
wer_policy: pinned by name and version # see the normalization chapter
canonicalizer: pinned by hash # locale-aware: en-US, es-US, es-MX, es-ES
segments: [english, spanish-dominant, spanglish]
entities:
amount: {gate: true, slices: [digits, number-words, shorthand, spanish-number-words]}
date: {gate: true}
confirmation: {gate: true, direction: no-heard-as-yes}
loan_tail: {gate: false} # monitored
interval: speaker-clustered bootstrap, 1,000 resamples, 95%
block_if:
- upper_bound > campaign_threshold # per entity, per segment; set by the campaign owner
- paired_difference_upper_bound > margin # candidate minus incumbent, same utterances, by speaker
report_always: [wer, cer, hallucinated_entity_rate, insufficient_slices]
The collections owner signs the threshold and the margin for each campaign before the first vendor call. Thresholds chosen after the bake-off tend to fit whichever vendor looked best.
A WER gate would have waved T01 through at 8.3%. If the STT confuses "fifty" and "fifteen" as a pattern, the shorthand slice of a gold set stratified by surface form shows it before release. A one-off error slips past any sample, which is why the deterministic online check matters: record_promise_to_pay ran at 23,010 ms with readback_confirmed=false, so C12 fails T01 on the spot. With the read-back in place the same STT error becomes "So that's two hundred fifteen dollars by November twentieth. Is that right?", a closed question Ryan can answer with "no, two fifty", and a correction instead of a wrong CRM row. How the verified-PTP metric compares the stored amount with the reference transcript is in layered metrics for voice agents.
Entity scoring also runs in the other direction: the amount or date Hazel speaks can be wrong in the audio while her text is right, as in T06, and the round-trip test in evaluating speech output for agents catches that.
What goes in the decision memo?
This chapter's memo is STT vendor selection, and its rule is the one this page argues for: hard constraints first, entity error rate second, WER only to break ties. Steelyard's worked memo uses the recipe's synthetic bake-off as a stand-in for real vendor results.
Decision. Choose system A for all three language groups: amount entity error rate 1.0% [0.4%, 1.9%] against B's 6.1% [4.1%, 8.2%], even though B has the lower WER (10.8% against 15.6%).
Context. Every collections flow writes amounts and dates taken from STT, and AUTO-early-nov carries the most volume. Audio arrives as 8 kHz G.711 mu-law; 32% of accounts prefer Spanish or code-switch.
Options considered. (1) Pick by WER: B, which books more wrong promises. (2) Pick by amount entity error rate: A, which costs extra clarifying turns on its WER errors. (3) Run both and route by language group: double the integration and monitoring for no gain shown.
Evidence. Synthetic stand-in from the entity error rate recipe (seed 20261120), shaped like gold-v7's STT set: 1,200 utterances, 180 speakers, 765 gold amounts; speaker-clustered 95% intervals; canonicalizer pinned by hash with locale tags. Confirmation flips: A 5, B 6. For the real bake-off the gate statistic is the paired speaker-level difference.
Owner and cadence. The STT owner on the eval platform team; the collections product lead signs. Rerun on every STT model version and every quarter.
Tripwires. A vendor model update; borrowers correcting amounts at read-back more often; any gated slice falling to "insufficient"; a new Spanish locale in the account mix.
Dated facts. Cresta's layer ordering (blog, 29 May 2026); Voice of India v4 (3 July 2026); CLDR 48 separator data; Whisper normalizer behavior as of November 2026.
What to do this week
- Pull 50 calls that contain an amount or a date, half of them Spanish or Spanglish, have a bilingual reviewer write reference transcripts, and annotate gold entities with canonical values.
- Run your canonicalizer on the references and fix every row where it disagrees with the gold labels before you score any STT system.
- Add the eight forms of $1,500.00 (with both locale tags on "1.500"), "one second" as a pause, "son doscientos, no?" as a tag question, and "not four hundred, three hundred" to the canonicalizer's tests.
- Report amount, date and confirmation entity error rate next to WER on your next STT run, with missed and hallucinated counts and a speaker-clustered interval per language group.
- Add the order check: no promise-to-pay write before a closed read-back and an explicit yes.
NOTE
What will date on this page. Voice of India figures are from arXiv 2604.19151 v4 (3 July 2026); check for newer versions. Separator and date-pattern rules are from Unicode CLDR 48 data; later CLDR releases can change them. Whisper's basic normalizer is described as of its main branch in November 2026. Cresta's layer description is from its 29 May 2026 post. The Soniox illustration and the Fisher Spanish catalog entry were checked in November 2026. Recipe numbers are tied to seed 20261120 and change if the generator does.