On trace T01, a borrower said "dhai hazaar" (Rs 2,500), the speech-to-text (STT) output said "do hazaar" (Rs 2,000), and Asha wrote a promise to pay for Rs 2,000 before reading anything back. One word in twelve was wrong, so the utterance scored 8.3% word error rate (WER), and the only number that mattered was 100% wrong.
What happened on the call?
On T01 the STT turned "dhai hazaar" into "do hazaar", the transcript stayed fluent and self-consistent, and the agent stored the wrong amount without a read-back. Nothing in the transcript view shows the error: only the reference lane or the audio has "dhai".
t_ms ch event / text
14100 agent TTS: "Rahul ji, aapka overdue chaar hazaar do sau pachaas rupaye hai. Aap kab tak bhar paayenge?"
18220 caller vad_start
caller REF: "haan main dhai hazaar de dunga bees tarikh ko, baaki next month"
caller STT: "haan main do hazaar de dunga bees tarikh ko baaki next month"
21940 caller vad_end
22310 agent eot_commit (end-of-turn delay 370 ms)
22990 agent llm first token (TTFT 670 ms)
23010 agent tool record_promise_to_pay(amount_paise=200000, ptp_date=2026-11-20, readback_confirmed=false)
23740 agent TTS: "Theek hai, toh main do hazaar rupaye bees November ke liye note kar rahi hoon."
28120 caller REF: "haan haan theek hai" (a backchannel; he did not register the changed amount)
Diagnosis: one substitution, utterance WER 1/12 = 8.3%, amount accuracy 0, and a C12 failure because the write came before the read-back. The CRM holds Rs 2,000 for 20 November; the borrower meant Rs 2,500, so the promise is Rs 500 short and the kept check will later mark it broken. The trace and its fields are described in what a voice agent trace contains.
Why can WER look fine while the entities are wrong?
WER counts every word the same, and the words that carry money, dates and consent are a small share of any transcript, so getting all of them wrong barely moves the average. A WER gate can pass an STT system that misreads every amount, because the denominator is mostly filler, verbs and names.
Soniox puts it as an illustration, not a measurement: if phone numbers are 10 of a transcript's 1,000 words, getting all of them wrong costs 1 point of WER, so the transcript can still be 97% accurate overall with 0% of phone numbers right. On exact match it takes even less: one wrong digit in a call's only phone number makes phone-number accuracy 0% while adding 0.1 points of WER 1.
T01 is the same arithmetic in one turn. "Do" for "dhai" costs one substitution, the same as "to" for "toh". WER has no notion that one moves Rs 500 and the other is a spelling.
Cresta describes its STT evaluation as four layers: lexical (WER, character error rate (CER), sentence error rate), entity (accuracy on business-critical tokens such as phone numbers, emails, amounts and dates, plus keyword recall), semantic (whether meaning is preserved) and task (downstream behaviour: agent success rate, p95 latency and interruption rate). When it compares systems, Cresta applies hard constraints first, then compares on WER, and uses entity, worst-slice and task metrics to break ties 2. For a collections agent, swap the last two steps: after hard constraints, entity error rate decides and WER breaks ties, because every amount and date Asha hears either lands in the CRM or triggers a read-back. The voice playbook lists named-entity WER as a secondary STT metric; for an agent that writes money, entity error rate is the primary one.
What is entity error rate?
Entity error rate, for one entity type, is the share of gold entities in the reference transcripts that the STT hypothesis does not recover exactly after both sides are canonicalized. Compute it per type (amount, date, loan tail, confirmation and the rest) and never pool the types: they fail and cost differently.
The parts of that definition each carry a decision:
- Gold entities come from the reference, annotated by a person. A labeller marks each amount, date or yes/no the speaker meant, with its canonical value. Before scoring any STT system, run your extractor on the reference text and check that it reproduces every gold value. Each disagreement is a canonicalizer bug or a labelling error; leave it in and the STT is charged for the scorer's gap.
- Canonicalize both sides with the same code. "Dedh lakh", "1.5 lakh" and "1,50,000" are the same 15000000 paise. Matching happens on canonical values, not on strings.
- Exact match, not edit distance. Rs 2,500 against Rs 2,000 is one digit apart; Rs 2,500 against Rs 25,000 is one inserted digit and ten times the money. Character-level partial credit on digit strings rewards near misses that are still wrong. If you need severity, report it next to the rate: the absolute rupee error and its direction (short or over).
- Per utterance and type, match as multisets. Recovered is the size of the intersection between the gold values and the hypothesis values. Missed is gold minus recovered. Hallucinated is hypothesis minus recovered.
That rule gives two numbers. Entity error rate is the recall side: missed gold entities over gold entities. The precision side is the hallucinated entity rate: hypothesis entities with no gold match over hypothesis entities. A substituted amount counts once in each, a dropped amount only as missed, and an invented amount only as hallucinated. In collections a hallucinated amount is the more dangerous error, because an agent that hears a number the borrower never said will try to book it. Report missed and hallucinated counts side by side for every type.
Value-only matching ignores roles. In "do hazaar abhi, paanch sau agle hafte" the STT can get both values right while the agent books the wrong one; that belongs to the task check, which diffs the CRM against the gold end state.
What does scoring eight utterances by hand show?
Scoring eight Tarazu utterances both ways shows the two metrics disagreeing on which rows are bad: the two lowest-WER rows each lost an entity, and the highest-WER row lost none. A review queue sorted by WER would start with the harmless rows.
The WER column uses lowercasing and punctuation stripping, with no numeral canonicalization and no transliteration mapping. Dates resolve against the call date, 2026-11-14 (a Saturday), except row 8, which is T08's first turn from 2026-11-16.
| # | Reference | STT hypothesis | WER | Gold entities | Entity result |
|---|
| 1 | haan main dhai hazaar de dunga bees tarikh ko baaki next month | haan main do hazaar de dunga bees tarikh ko baaki next month | 1/12 = 8.3% | amount 250000; date 2026-11-20 | amount missed (200000 heard) |
| 2 | ek lakh pachaas hazaar paanch tarikh tak | 1,50,000 paanch tarikh tak | 4/7 = 57.1% | amount 15000000; date 2026-12-05 | both recovered |
| 3 | sahi nahi hai | sahi hai | 1/3 = 33.3% | confirmation: no | polarity flipped to yes |
| 4 | main aath tarikh ko kar dunga salary ke baad | main saat tarikh ko kar dunga salary ke baad | 1/9 = 11.1% | date 2026-12-08 | date missed (2026-12-07 heard) |
| 5 | salary aane do phir dunga | salary aane do sau phir dunga | 1/5 = 20.0% | none ("do" here is a verb) | 1 hallucinated amount, 20000 |
| 6 | last four digit chaar chaar ek saat | last four digit chaar chaar saat ek | 2/7 = 28.6% | loan tail 4417 | missed (4471 heard) |
| 7 | paanch december ko pakka | panch december ko pakka | 1/4 = 25.0% | date 2026-12-05 | recovered ("panch" is an accepted variant) |
| 8 | haan but meri salary 7 ko aati hai toh after that i can pay around 9th | hum but mary salary seven ko rt hai to after that i can pay around ninth | 6/16 = 37.5% | confirmation: yes; two dates | dates 2/2; confirmation missed ("hum") |
Corpus WER is total errors over total reference words: 17/63 = 27.0%. The entity view, by type:
| Type | Gold | Missed | Entity error rate | Hallucinated |
|---|
| Amount | 2 | 1 | 50% | 2 (rows 1, 5) |
| Date | 6 | 1 | 16.7% | 1 (row 4) |
| Loan tail | 1 | 1 | 100% | 1 (row 6) |
| Confirmation | 2 | 2 | 100% | 1 (row 3) |
Row 2 is the worst row by WER and the best by entities: the STT wrote the amount as digits, which WER scores as one substitution and three deletions and the canonicalizer scores as correct. Row 7 is a spelling variant that WER charges and the entity check accepts. Rows 1 and 4 share the T01 pattern: one number word swapped for a close-sounding neighbour, at WER near 10%. Row 5 is the case most scorers get wrong: bare "do" is a verb ("aane do", let it come), so the canonicalizer must not read it as two, and the inserted "sau" creates an amount nobody said.
Eight rows prove nothing about a vendor; they show the two metrics rank utterances differently, which is why you compute both on every run. The entity error rate recipe runs the same scorer on 1,200 seeded utterances, and reference-based metrics covers the general case.
How do you canonicalize Hinglish amounts and dates?
Map every spoken or written form to one canonical value: amounts to integer paise, dates to ISO dates resolved against the call date, digit strings to digits, yes/no forms to a polarity. Keep the rules in a versioned policy, test them on a fixed list of surface forms, and run the same code on both sides.
Tarazu's canonical forms, from the world bible, with the rule that most often breaks:
| Entity type | Spoken examples | Canonical form | Rule that bites |
|---|
| Amount | "dhai hazaar", "do hazaar paanch sau", "2,500" (all 250000); "paune do lakh" (17500000); "sava lakh" (12500000) | Integer paise | Fractions: sava = 1.25, dedh = 1.5, dhai = 2.5, paune X = X - 0.25, saade X = X + 0.5 from 3 upward |
| Date | "5 tarikh", "paanch December", "agle mangalvaar", "salary ke baad, 8 ko", "05/12" | ISO date, resolved against the call date | A bare day number means its next occurrence; "05/12" is day first under an explicit dmy policy |
| Loan tail | "ending 4417", "chaar chaar ek saat" | 4 digits | Order matters; "saat" (seven) and "saath" (with) differ by one letter |
| UTR | 12 digits, read in groups | String of digits | Group boundaries are not digits |
| Confirmation | "haan", "haan ji", "ji", "nahi", "nahi nahi", "theek hai" | Yes, no or backchannel | Polarity is scored here; backchannel is a dialogue label (next section) |
| Name | "Rahul Verma", "Sunita ji" | Normalized first and last name | Honorifics such as "ji" are stripped |
| Pincode | "chaar ek ek zero chaar paanch", "411045" | 6 digits | Mixed Hindi and English digit words in one string |
| Email | "grievance at tarazu dot example" | Lowercased address | "at" and "dot" are spoken symbols |
| Address | "Tarazu House, Baner Road, Pune 411045" | Normalized tokens; pincode exact | Score the pincode as its own entity |
Indian amounts need four things a Western number parser lacks: the lakh and crore scale words, Indian digit grouping (1,50,000) next to Western grouping (150,000), the fraction words, and Hindi number words, which run as single words up to ninety-nine ("unnees" for 19, "pachaas" for 50) rather than tens plus units. The test that pins all of it is a fixed list. These eight forms of Rs 1,50,000 must all map to 15000000 paise, and the canonicalizer fails its own test if any does not:
1 dedh lakh
2 1.5 lakh
3 1,50,000
4 150000
5 150,000
6 ek lakh pachaas hazaar
7 one lakh fifty thousand
8 १,५०,००० (Devanagari digits)
NVIDIA's open NeMo-text-processing Hindi inverse text normalization (release r1.2.0, 5 June 2026) is a usable base. It lists the Hindi scale words (hazaar, lakh, crore, arab, kharab), has a separate table for "paune" (quarter-less) amounts, accepts spelling variants, and outputs Devanagari digits; its own test cases map "dedh lakh rupaye" to the rupee sign followed by the digits 3.
input डेढ़ लाख रुपए
output rupee sign + १५००००
scale words हज़ार लाख करोड़ अरब खरब
spelling variants पन्द्रह = पंद्रह
Its tables and tests are in Devanagari. Add Hinglish in Roman script, English number words and the spellings your STT actually emits, then convert its Devanagari digits before comparing.
Dates carry more policy than amounts. Write each decision down:
- Relative dates resolve against the call date in the borrower's time zone, from the trace's local start time, never the scoring machine's clock. "Bees tarikh" on 2026-11-14 is 2026-11-20; "paanch tarikh" on the same day is 2026-12-05, because the 5th of November has passed.
- "Agle mangalvaar" needs a rule. Tarazu reads it as the first Tuesday after the call date: 2026-11-17 for a Saturday call. Some speakers mean the Tuesday of next week. Pick one reading, version it, and make the agent ask when it matters.
- "05/12" is day first under an explicit dmy policy. Read month first it is 12 May, the T06 failure in the speech-output direction, covered in evaluating speech output for agents.
When the reference itself is ambiguous ("dhai ya teen hazaar"), the gold label is "ambiguous", the utterance leaves the amount denominator, and the right agent behaviour is one clarification, scored in the conversation checks.
The canonicalizer is part of the metric: pin it by hash on every reported number, as you pin the WER normalizer. Script, matras and transliterated spellings belong to normalization and code-switching in Hinglish STT.
Why do haan and nahi need their own metric?
A yes or no is usually one or two words, so one dropped or swapped word reverses its meaning while barely moving WER, and short utterances are where STT is weakest. Score confirmation polarity as its own entity type and report the two directions of error separately: they cost different things.
Row 3 above is the clearest case: "sahi nahi hai" (it is not right) lost one word and became "sahi hai" (it is right). In a 1,200-utterance corpus that is one error among thousands of words, and the polarity was the whole content of the turn.
The two directions:
- No heard as yes turns a refusal into consent. After a read-back, the agent books the promise, the C12 check passes because the agent did hear a yes, and the CRM holds a promise the borrower declined. This is the costly direction, and it is the one to gate.
- Yes heard as no costs an extra clarification turn. Repeated, it trips the clarification-loop rule (more than two clarifications on one slot, code LOOP.CLARIFY), and T08's "haan" heard as "hum" is the start of exactly that loop.
Two things that look like confirmation errors are not STT errors. The first is a backchannel: in T01, "haan haan theek hai" at 28,120 ms answered a statement, not a closed question, so it confirms nothing. Telling a yes from a backchannel needs the preceding turn and the timing, so that label belongs to turn-taking, barge-in and recovery; the STT metric scores only polarity.
The second is a lexicon gap. Yes forms include "haan", "haan ji", "ji", "bilkul", "sahi hai", "yes" and "correct"; no forms include "nahi", "na", "nahi nahi" and "galat hai". A form missing from the lexicon is scored as a missed entity and blamed on the STT, so test the lexicon on the references first.
Negation also hides inside amounts. "Chaar hazaar nahi, teen hazaar" (not four thousand, three thousand) contains two amount strings, and the gold amount is 3,000. Annotate the negated value as negated, so an extractor that takes the first number it sees fails its own reference test instead of passing silently.
Which slices matter for collections?
Report entity error rate by entity type, surface form, utterance length, negation, carrier, and region or accent, and gate on the worst slice that a campaign depends on, not the mean. An STT system that is excellent on digits and weak on "dhai" and "sava" has a fine average and a broken two-wheeler campaign.
- Entity type and surface form. Split amounts into digit forms, number words and fraction words. In the recipe's synthetic bake-off, one system's amount errors sat almost entirely on spoken numbers (9.3% on fraction forms, 11.3% on other number words, 0 of 280 on digits); the error injector was written that way, so the ordering is not a finding about real systems. The habit of splitting is the point.
- Utterance length. Confirmations and short amount answers live in the shortest bin. Voice of India (Bhogale et al., arXiv 2604.19151 v4, 3 July 2026, accepted at Interspeech 2026) is a closed benchmark of unscripted telephonic conversations: 306,230 utterances, 536 hours, 36,691 speakers, 15 languages, scored with OIWER, which accepts valid spelling variants. Short utterances are hardest: averaged across languages, Amazon's STT error rate rises from 10.45% on utterances longer than 5 s to 18.74% on utterances under 2 s, and Microsoft's from 10.90% to 18.59%. The authors attribute this to limited semantic context. The paper does not state a sample rate or carrier codec, so these are not 8 kHz PSTN figures 4.
- Negation and self-correction. Utterances with "nahi" near an entity, or a corrected amount, are a slice of their own.
- Carrier and codec chain. Jio to trunk through an AMR-NB and G.711 tandem is not the same audio as a 16 kHz app recording. How to build that test set is covered in testing on telephony audio.
- Region and accent. LAHAJA (AI4Bharat, Interspeech 2024) is a 12.5-hour multi-accent Hindi benchmark from 132 speakers in 83 districts. Its named-entity task, in which speakers say five numbers, five dates, five person names and five places, is the worst content category for both models in its Table 5: Google Chirp scores 63.2% WER and the authors' best model, M3, scores 24.4%, against 12.9% for M3 on read Wikipedia sentences 5. M3 is the authors' own model, and the audio is read and extempore speech, not phone calls. That is WER on entity-heavy speech, not entity error rate, but it points the same way.
- Code-switch density. Slice by low, medium and high density of language switching per utterance; how to score density is owned by the normalization chapter.
A slice with few gold entities gives an interval too wide to act on. Mark it "insufficient" instead of printing a point estimate that looks like a finding, and grow the gold set until the interval is narrower than the decision tolerance.
How many utterances do you need, and what interval?
Use a bootstrap that resamples speakers, not utterances: errors cluster by speaker because accent, handset and pace are shared across a speaker's utterances. Size the set by gold entities of the type you gate on, not by utterance count, and expect a per-segment gate to need several hundred gold amounts per segment.
A Wilson interval over utterances assumes each utterance is an independent draw. In a gold set of 1,200 utterances from 180 speakers they are not, and a speaker who drops "dhai" once tends to drop it again. The speaker-clustered bootstrap:
- For each speaker, sum the missed gold entities and the gold entities of the type.
- Resample speakers with replacement, as many as there are, 1,000 or more times.
- For each resample, recompute the pooled ratio: total missed over total gold.
- Take the 2.5th and 97.5th percentiles as the 95% interval.
The general case, including the design effect DEFF = 1 + (m - 1) x rho for clusters of size m and within-cluster correlation rho, is in clustered standard errors and Cameron and Miller's practitioner's guide 6.
The entity error rate recipe generates two seeded sets in the shape of Tarazu's gold-v7 STT set and computes the interval on both (synthetic Tarazu data, seed 20261120). For the system with the higher amount entity error rate:
| Set | Amount entity error rate | Speaker-clustered 95% interval | Width |
|---|
| 400 utterances, 60 speakers | 2.9% | [1.0%, 5.2%] | 4.2 points |
| 1,200 utterances, 180 speakers | 5.9% | [4.2%, 7.7%] | 3.5 points |
The smaller set cannot tell you whether that system's amount entity error rate is 3% or 6%. (The recipe's bake-off prints [4.1%, 7.7%] for the larger set because it draws different resamples; do not read the second decimal.) Here the clustered and utterance-level widths differ by only 0.1 points, because each speaker contributes about 3.5 amounts (632 over 180 speakers). Keep the clustered interval anyway: production samples, where repeat callers contribute dozens of utterances, are where the utterance-level interval goes wrong.
For planning, the normal approximation gives a rough count: n is about 1.96^2 x p(1 - p) / h^2 x DEFF gold entities for a half-width h. At p = 2% and h = 1 point, that is 753 gold amounts before the design effect. If 632 amounts split evenly across three language groups, each holds about 211, and the half-width at 2% becomes about 1.9 points. Per-segment gating multiplies the gold set, so decide which segments gate and which are only monitored before you buy transcription.
Comparing two STT systems is a different question from bounding one. Score both on the same utterances and bootstrap the paired difference by speaker, as in bootstrap and paired tests; overlapping intervals are not that test.
Figure: Rank reversal between WER and amount entity error rate. Synthetic Tarazu data from the entity error rate recipe (1,200 utterances, 180 speakers, seed 20261120): system B has the lower WER and about three times the amount entity error rate, and it flips 8 confirmations against A's 2. Pick by WER and you ship the system that writes more wrong promises.
What gate would have caught it?
Two checks. Offline, an entity error rate gate on gold-v7 (amount, date and confirmation, per language group) blocks any STT release or vendor switch whose speaker-clustered upper bound exceeds the campaign threshold. Online, the C12 order check fails any promise written before a closed read-back and an explicit yes.
The offline gate:
gate: stt-entities
gold_set: gold-v7 # STT set: 1,200 reference-transcribed utterances, 180 speakers
scorer:
wer_policy: nfc+t-wer # pinned with its mapping version; see the normalization chapter
canonicalizer: pinned by hash
segments: [hinglish, hindi-dominant, indian-english]
entities:
amount: {gate: true, slices: [digits, number-words, fraction-words]}
date: {gate: true}
confirmation: {gate: true, direction: no-heard-as-yes}
loan_tail: {gate: false} # monitored
interval: speaker-clustered bootstrap, 1,000 resamples, 95%
block_if:
- upper_bound > campaign_threshold # per entity, per segment; set by the campaign owner
- paired_difference_upper_bound > margin # candidate minus incumbent, same utterances, by speaker
report_always: [wer, cer, hallucinated_entity_rate, insufficient_slices]
The collections owner signs the threshold and margin per campaign, before the first vendor call. Set after the bake-off, they get fitted to whichever vendor looked best.
A WER gate would not have caught T01, which sits at 8.3%. If the STT confuses "dhai" and "do" as a pattern, the fraction-form slice of a gold set stratified by entity type shows it before release. A one-off error slips through any sample, which is why the deterministic online check matters: record_promise_to_pay ran at 23,010 ms with readback_confirmed=false, so C12 fails T01 on the spot. With the read-back in place, the same STT error becomes "Toh do hazaar rupaye, bees November tak. Sahi hai?", a closed question the borrower can say no to, and a correction instead of a wrong CRM row. How verified PTP compares the stored amount with the reference transcript is in layered metrics for voice agents.
Entity scoring also runs the other way: the amount Asha speaks can be wrong in the audio while the text is right (T06); the round-trip test in evaluating speech output for agents catches that.
What goes in the decision memo?
The memo for this chapter is STT vendor selection, and its rule is the one this page argues for: hard constraints first, entity error rate second, WER only as a tie-breaker. Tarazu's worked memo uses the recipe's synthetic bake-off as a stand-in for vendor results.
Decision. Choose system A for all three language groups: amount entity error rate 1.9% [1.0%, 2.9%] against B's 5.9% [4.1%, 7.7%], although B has the lower WER (8.9% against 11.9%).
Context. Every collections flow writes amounts and dates read from STT; TW-early-nov carries the most volume. Inputs are 8 kHz G.711 mu-law streams.
Options considered. (1) Pick by WER: B, which writes more wrong promises. (2) Pick by amount entity error rate: A, which costs extra clarification turns on its WER errors. (3) Run both and route by language: doubles integration and monitoring for no shown gain.
Evidence. Synthetic stand-in (tarazu-calls, seed 20261120), shaped like gold-v7's STT set: 1,200 utterances, 180 speakers, 632 gold amounts; speaker-clustered 95% intervals; scorer policy nfc+t-wer (tarazu-translit-v1). Confirmation flips: A 2, B 8. The paired speaker-level difference is the gate statistic for the real bake-off.
Owner and cadence. STT owner on the eval platform team; the collections product lead signs. Rerun on every STT model version and quarterly.
Tripwires. A vendor model update; borrowers correcting amounts at read-back more often; any gated slice moving to "insufficient".
Dated facts. Cresta's layer ordering (blog, 29 May 2026); Voice of India v4 (3 July 2026); LAHAJA (2024); NeMo-text-processing r1.2.0 (5 June 2026).
What to do this week
- Pull 50 calls with an amount or a date, have a bilingual reviewer write reference transcripts, and annotate gold entities with canonical values.
- Run your canonicalizer on the references and fix every row where it disagrees with the gold labels before you score any STT system.
- Add the eight surface forms of Rs 1,50,000, "paune do lakh", bare "do" as a verb, and "chaar hazaar nahi, teen hazaar" to the canonicalizer's tests.
- Report amount, date and confirmation entity error rate next to WER on your next STT run, with missed and hallucinated counts and a speaker-clustered interval.
- Add the order check: no promise-to-pay write before a closed read-back and an explicit yes.
NOTE
What will date on this page. Voice of India figures are from arXiv 2604.19151 v4 (3 July 2026); recheck for newer versions. LAHAJA is v1 (21 August 2024). NeMo-text-processing Hindi inverse text normalization is described as of release r1.2.0 (5 June 2026); the scale-word and paune tables may change. Cresta's layer description is from its 29 May 2026 post. The Soniox illustration was checked in November 2026. Recipe numbers are tied to seed 20261120 and change if the generator does.