On call T08, Asha's speech-to-text locked onto English halfway through a Hinglish sentence, mangled the borrower's salary date, and looped on clarifications until she hung up. Separately, Tarazu's STT word error rate once fell 4 points overnight with no model change, because the scoring job had changed its normalizer.
Both events have one cause: nobody had written down what a correct Hinglish transcript looks like. This chapter covers that definition. The entity metric built on top of it is in entity error rate.
What happened on the call?
The borrower switched language three times in one sentence, the STT locked onto English after the first switch, and the agent kept asking for the date until the borrower spelled out "nine" and hung up at 71 seconds.
t_ms ch event / text
caller REF: "haan, but meri salary 7 ko aati hai, toh after that I can pay, around 9th"
caller STT (language locked to en after "but"): "hum but mary salary seven ko RT hai to after that I can pay around ninth"
agent TTS: "Sorry, could you please tell me the date you can pay?"
caller REF: "nau tarikh, 9th, after salary"
caller STT: "now three, ninth after salary"
agent TTS: "Just to confirm, is it the third or the ninth?"
caller REF: "arre yaar, nine. N-I-N-E." (third clarification; hangs up at 71 s)
Diagnosis: language identification locked to English (STT.LID), turning "aati" into "RT" and "nau" into "now three", and the agent broke the two-clarification rule (LOOP.CLARIFY). Marcus Chen described the same shape in June 2026: a regression suite of 312 recorded production calls all passed, then a caller who switched between English and Hindi mid-sentence caused a 3am page when speech recognition mis-segmented the audio and the agent looped on clarifications. The pattern appeared zero times in the 312 calls 1. A suite that never contains dense code-switching cannot fail on it.
Why does normalization decide whether the number means anything?
WER is computed on text after a normalizer has rewritten both the reference and the hypothesis, a step called text normalization. Change the normalizer and the number changes with the model untouched. For Hinglish it also decides script, spelling variants and numerals. Treat it as part of the metric.
For Indian languages, single-reference WER overstates error because suffix spellings, word splitting and merging, and code-mixed loanword spellings are not standardized. On IndicVoices, AI4Bharat's Orthographically informed WER (OIWER), which credits permitted spelling variants, cut a fine-tuned Canary model's WER by 6.3 points on average across 22 languages, Hindi from 13.2 to 8.9 (arXiv preprint, March 2026) 2. Those 4.3 Hindi points are a measurement change, not a model gain. Tarazu's 4-point drop was the same kind of event, celebrated as good news.
This chapter uses four scoring policies, the same four as the entity recipe and the entity-error-lab toggles:
| Policy | What it does before alignment |
|---|
| raw | Splits on whitespace; case and punctuation count as errors |
| whisper-basic | Mirrors Whisper's basic normalizer: lowercase, then every mark, symbol and punctuation character becomes a space |
| nfc-only | Unicode NFC, lowercase, punctuation and symbols removed, combining marks kept |
| nfc+t-wer | nfc-only, then a versioned Devanagari-to-Roman mapping (tarazu-translit-v1) |
Worked example: T08's first turn. The reference has 16 words. The logged hypothesis substitutes six of them: haan/hum, meri/mary, 7/seven, aati/RT, toh/to, 9th/ninth.
| Hypothesis | raw | whisper-basic | nfc-only | nfc+t-wer |
|---|
| STT as logged (Roman, locked to en) | 50.0% | 37.5% | 37.5% | 37.5% |
| Same words, Hindi in Devanagari | 50.0% | 56.2% | 43.8% | 0.0% |
Synthetic Tarazu data; R1's scorer, rerun for this page.
In the first row, raw scores 8 of 16 because "hai," and "pay," carry commas the hypothesis lacks. The other policies agree on 6 of 16. Add numeral canonicalization and "7"/"seven" and "9th"/"ninth" stop counting, leaving 4 of 16, or 25.0%: hum, mary, RT and to are the real recognition errors.
The second row heard every word correctly in a different output convention, and scores 0% to 56.2% depending on the policy. None of that is recognition error.
What is transliterated WER?
Transliterated WER (T-WER) counts a word as correct when the hypothesis writes it in the other script, using a curated mapping between scripts. Standard WER accepts only an exact match, so the same word in two scripts counts as an error.
T-WER was introduced for the MUCS 2021 code-switching challenge: an English word in the reference counts as correct whether the system writes it in Latin script or as its transliteration in the native script 3.
Three consequences follow. The mapping is part of the metric, so its version string (tarazu-translit-v1) goes beside every T-WER number; add one entry and every historical number is stale. A mapping covers only the words it contains: against an all-Devanagari reference, the Devanagari-mode T08 hypothesis still scores 50.0% under nfc+t-wer because English words and number words written in Devanagari are not in the map (Figure 1, row C). Log unmapped non-Latin tokens on every run. And direction matters: Tarazu maps Devanagari to Roman because its reference is Roman, while MUCS mapped toward the native script.
A related approach normalizes everything into one script before scoring. SN-WER transliterates reference and hypothesis into one canonical script; its abstract reports a 67% attenuation of artificial romanization-induced WER inflation in stress tests, and up to 12% smaller inflated model gaps on FLEURS 4. Machine transliteration adds its own errors, so test either route on a fixed fixture and pin it.
Which script policy should the reference use?
Pick one convention for the reference, write it down, and pass every hypothesis through the same normalizer before scoring. Tarazu's reference is code-mix: Hindi in Roman, English words in Latin spelling, numerals as digits.
Entities are scored after canonicalization whatever the script, so the script policy moves WER, never the amount check. One STT model can emit several conventions. Sarvam's Saaras V4 returns five output conventions from one model: verbatim, transcribe (numbers and dates normalized), codemix (English words kept in Latin script), translit (all Roman) and translate. In Sarvam's own example the same audio comes out as "T20" in codemix mode, as the Devanagari spelling of "T-20" in transcribe mode and as the Devanagari words for "T twenty" in verbatim mode 5. The docs list the same five modes for the earlier saaras:v3 6. Score a hypothesis against a reference in another convention and it picks up errors that are not recognition errors, so ask every vendor which mode produced its numbers.
Sarvam also reports, unaudited, a language-ID error of 5.22% across all 22 Indian languages and 2.9% across the ten most spoken, on verified IndicVoices utterances (Saaras V4 blog, 25 September 2026) 5. A per-utterance language-ID score cannot show T08's failure, which was a lock inside one utterance, so measure locks yourself (the gate below).
CAUTION
Vendor-reported. Treat it as a hypothesis to test on your own calls.
Benchmarks disagree too. IndicContextEval's references are verbatim native-script transcripts with English named entities transliterated into the native script, and models are told to write numbers as spoken words 7, the opposite of Tarazu's convention. Its WER and a WER on Tarazu's gold set are different quantities. Build the gold set from native Hinglish calls and never translate an English set (non-English RAG evals makes the same argument for text). Inverse text normalization (ITN), which turns spoken numbers into written ones, adds a choice: NVIDIA's open Hindi ITN outputs Devanagari digits, mapping the Devanagari for "dedh lakh rupaye" to the rupee sign and 150000 in Devanagari digits 8. The policy's numeral accept-set says which forms count as equal.
Figure 1: one utterance, four references, one hypothesis that heard every word correctly (Hindi in Devanagari, English in Latin, digits). Synthetic Tarazu data, scored with R1's scorer.
| Reference convention | nfc-only | nfc+t-wer |
|---|
| A. Code-mix Roman, digits (Tarazu) | 43.8% | 0.0% |
| B. Mixed script: Hindi Devanagari, English Latin, digits | 6.2% | 0.0% |
| C. All Devanagari, numbers as words | 56.2% | 50.0% |
| D. All Roman, numbers as words | 56.2% | 12.5% |
hypothesis हाँ but मेरी सैलरी 7 को आती है तो after that I can pay around 9th
A haan, but meri salary 7 ko aati hai, toh after that I can pay, around 9th
B हाँ, but मेरी salary 7 को आती है, तो after that I can pay, around 9th
C हाँ, बट मेरी सैलरी सात को आती है, तो आफ्टर दैट आई कैन पे, अराउंड नाइंथ
D haan, but meri salary seven ko aati hai, toh after that I can pay, around ninth
Row B's 6.2% is one word, "salary" in two scripts. Row D falls to 0.0% once numerals are canonicalized. A recognizer that made no mistakes scores anywhere from 0% to 56.2%, and the convention picks the number.
Which normalizer bugs should you check first?
Check what the normalizer does to Hindi vowel signs before you trust any Hindi number: Whisper's basic normalizer breaks Devanagari words into consonant fragments and makes Hindi WER look better than it is. Then check Unicode normalization, because one word typed two ways is otherwise two words.
Whisper's basic (non-English) text normalizer replaces every Unicode mark-class character, including Hindi vowel signs (matras) and the virama, with a space. As of November 2026 the BasicTextNormalizer in the openai/whisper repository still maps mark-class characters to spaces 9. Manohar, Pillai and Sherly found this cut Whisper-small's Hindi WER on the FLEURS test set by 21.9 points absolute (from 86.9%), and by 10.7 points for a Hindi fine-tuned model, against 4.5 points for English; they call these artificially improved metrics. The same routine is used in MMS, Seamless and AssemblyAI Conformer-1 evaluations and ships in Hugging Face transformers 10. Ask which normalizer produced any Hindi number you are shown.
Here is the mechanism on R1's Devanagari fixture, where reference and hypothesis are both in Devanagari and one amount is wrong (dhai heard as do):
nfc-only हाँ मैं ढाई हज़ार दूँगा -> हाँ | मैं | ढाई | हज़ार | दूँगा 5 words
whisper-basic हाँ मैं ढाई हज़ार दूँगा -> ह | म | ढ | ई | हज | र | द | ग 8 fragments
fixture WER raw 50.0% whisper-basic 7.4% nfc-only 43.8% nfc+t-wer 6.2%
amount entity errors 1/3 under every policy
Whisper-basic and nfc+t-wer land about a point apart for opposite reasons. The T-WER mapping merges real spelling variants (haan with chandrabindu or anusvara, hazaar with or without the nukta) and keeps the one true error: 1 in 16 words. Whisper-basic deletes the vowel signs, inflates the denominator to 27 fragments, and reports 2 errors in 27. One is a measurement, the other an artifact.
Figure 2 shows the effect at corpus scale on R1's synthetic 1,200-utterance set from 180 speakers. Entity error rate, the share of gold amounts not recovered exactly after canonicalization, does not move.
Figure 2: Synthetic Tarazu data from the entity recipe (R1). The same hypotheses moved system B's corpus WER from 18.7% under raw to 8.9% under nfc+t-wer, a 9.8-point swing with no model change. Its amount entity error rate stayed at 5.9% (speaker-clustered 95% CI 4.1% to 7.7%) because amounts are canonicalized before they are compared.
Two checks catch most of this. Run NFC on both sides: the fixture scores 50.0% raw and 43.8% after NFC because one pair is the same text in two Unicode encodings. Then assert in CI that a frozen Devanagari fixture scores its expected WER, so a swapped-in mark-stripping normalizer fails the build.
How much does code-switching cost?
It depends on the model, its training data and the language pair, so no published penalty transfers to your calls. Comparisons run from a small change in either direction to more than double the monolingual error rate. Measure it on your own slices.
The ends of the range: with a model trained on pooled monolingual and code-switched speech, the penalty ran from roughly -10% to +15% relative (Shah et al., Microsoft, 2020) 11. Off-the-shelf Whisper-large-v3 had a character error rate more than double the monolingual rate (Yan et al., CS-FLEURS, Interspeech 2025) 12. CS-FLEURS covers 113 language pairs across 52 languages, mostly synthetic and read speech, and its error averaged three times higher for pairs written in two different scripts than for same-script pairs. For Hindi-English, a Microsoft bilingual model trained with 2,600 hours of Hinglish still scored 31.4% WER on code-mixed test utterances, against 15.1% on Hindi and 18.2% on English (Patil et al., IEEE SLT 2022) 13. Both of those papers compare different sentences in each condition, so content and transcription conventions ride along with the switching.
The widest public Hinglish spread comes from SwitchLingua. On its Hindi-English recordings, WER on the same audio ranged from 27.0% for Whisper-Large-v3 to 69.5% for Qwen2-Audio-7B-Instruct's raw output (65.7% once its extra preamble text was cleaned) across five open-source models 14. Mind the scope: the audio is speakers reading LLM-generated code-switched dialogue scripts on their own devices, not telephony or spontaneous call speech, and no commercial or India-specific STT was tested. It shows general models differ by more than a factor of two on Hinglish, and says nothing about vendorA-2026-10 on a Jio call.
For a defensible penalty, compare within speakers: take gold-v7 speakers with both low-density and high-density utterances, compute each slice's WER, and bootstrap the difference by resampling speakers (clustered standard errors). That holds voice, phone and accent constant.
How do you score code-switch density?
Tag each reference token with a language, count switch points between adjacent tagged tokens, bucket utterances into low, medium and high, and slice every STT and task metric by bucket. A corpus average dilutes the dense turns where failures like T08 happen.
Write three tagging rules down. Numerals and names are neutral. Shared loanwords (salary, EMI, payment, link) are neutral. Tags come from the reference, never the STT, since T08's STT tags are the failure.
| Utterance | Tagged tokens | Switch points | Switch-point fraction |
|---|
| T01: "haan main dhai hazaar de dunga bees tarikh ko, baaki next month" | 12 | 1 | 1/11 = 0.09 |
| T08: "haan, but meri salary 7 ko aati hai, toh after that I can pay, around 9th" | 13 | 3 | 3/12 = 0.25 |
| T08, if "salary" is tagged English | 14 | 5 | 5/13 = 0.38 |
The fraction is switch points over tagged tokens minus one. One tagging decision moves T08 from 0.25 to 0.38, which is why the loanword list belongs in the versioned policy.
Caller turns in collections are short, so bucket on switch count rather than the fraction. A workable start is 0 switches for low, 1 or 2 for medium, and 3 or more for high. Check that each bucket holds enough speakers for a clustered interval, and report the high bucket on every release. Density is a monitored slice at Tarazu, not one of the 12 gated segments of the agent release; the language-lock check below is an acceptance check on STT version changes instead. The entity error rate chapter applies the same slice to amounts and dates.
Should you use lenient or LLM-based scoring?
Use lenient scoring for spelling and keep it away from meaning. Crediting real orthographic variants makes Hindi WER honest; crediting near-misses hides wrong amounts. Keep entity exact match separate, and do not gate on an LLM-scored WER you cannot recompute next quarter.
OIWER's framework uses LLM-assisted generation of word variations 2. A proposed variant list for Tarazu will contain pairs that sound close and mean different things: do and dhai, nau and now, haan and hum. Exclude number words, negations and confirmations from every variant set, and have a bilingual reviewer prune the rest. A variant list that accepts "do" for "dhai" turns T01's Rs 500 error into a correct transcript.
An LLM that reads two transcripts and reports semantic errors is a judge, not a string metric. Its score depends on the model version and the prompt, so a rerun after the model is retired gives a different number. Use it to rank which utterances to listen to first. If you report it, pin the model and prompt, validate it against human labels as in audio judges, and keep it out of the gate.
What gate would have caught it?
A language-lock check on the high-density slice would have measured the failure before release, and the deterministic clarification-loop check would have flagged T08 on the call. Behind both sits a pinned scorer hash, so no WER is compared across scorer versions by accident.
The checks, in the order Tarazu runs them:
- Scorer hash on every number. Hash the policy file, the T-WER mapping and the scorer code. Every WER and dashboard tile carries the hash, and the comparison job refuses to diff numbers with different hashes. The 4-point drop would have failed this check the morning it appeared.
- Normalizer fixture in CI. The frozen Devanagari fixture must score its expected WER under the pinned policy, which catches a mark-stripping normalizer and a dropped NFC step.
- Language-lock rate on the high-density slice. An utterance is locked when, after a switch point, the STT tags every remaining token with one language while the reference switches back. T08 is locked: every STT tag after "but" is en, and the reference returns to Hindi at "meri". Compare a candidate STT version with production on the same utterances by paired bootstrap, and hold the change for sign-off when the interval for the difference lies entirely above zero.
- LOOP.CLARIFY on 100% of calls and on every simulated run. A third clarification on one slot is a deterministic trace event. Run code-switching callers such as P12 through the production audio path (voice simulation) and fail any run that loops.
- A regression case cut from T08. Its first turn, reference and audio go into the gold set with the high-density label.
Interval rule: Wilson only for a single binary proportion, such as the share of simulated P12 runs that loop. Utterance-level rates, corpus WER and the lock rate use a bootstrap clustered by speaker, and the release comparison uses the paired bootstrap (bootstrap and paired tests).
What goes in the decision memo?
Record the scoring policy as a dated decision with an owner and the events that force a re-baseline. Tarazu's memo:
- Decision: Normalization policy v1 for every STT number: NFC, combining marks kept, code-mix reference (Hindi in Roman, English in Latin, digits), T-WER mapping tarazu-translit-v1, a numeral accept-set ("7", "seven", "saat"), a filler list that drops hmm and umm but never haan, nahi or accha, and a scorer hash on every report.
- Context: WER fell 4 points overnight from a scorer change alone; T08's language lock is invisible to every current metric.
- Options considered: The harness default (unpinned, may strip matras); all-Devanagari references (fights the code-mix STT mode); uncurated variant credit (admits do for dhai).
- Evidence: On R1's synthetic 1,200-utterance set (180 speakers), the four policies moved system B's WER from 18.7% to 8.9%; amount entity error rate stayed at 5.9% (speaker-clustered 95% CI 4.1% to 7.7%). gold-v7 is re-scored under v1 before any comparison.
- Owner and cadence: STT evaluation lead; quarterly, and on any STT vendor, output-mode or harness change.
- Tripwires: Any scorer change re-baselines every number; new unmapped Devanagari tokens go to the mapping owner weekly; a lock-rate regression on the high-density slice holds an STT version change.
- Dated facts: Whisper's basic normalizer still maps marks to spaces (November 2026); Saaras five-mode output (September 2026); OIWER figures are from its arXiv preprint (v1, March 2026).
What to do this week
Start with the scorer, because every other number on this page depends on it. Metric targets and a day-1 starter live in the voice playbook.
- Read the code of the normalizer behind your STT numbers. If it calls Whisper's basic normalizer on Hindi, your Hindi WER is flattered.
- Write the policy file: NFC, marks kept, reference convention, numeral accept-set, filler list, loanword tags. Version it like a dataset (versioning and lineage), hash it, and print the hash beside every WER you report.
- Re-score last quarter's stored hypotheses under the new policy before comparing anything. Keep raw STT output in the trace; normalized text cannot be un-normalized.
- Tag gold-set references with languages, compute switch counts, and add the low, medium and high slices to the STT report.
- Listen to ten high-density production calls and count the language locks.
NOTE
What will date on this page. The openai/whisper BasicTextNormalizer behaviour (checked November 2026); Sarvam's output modes and its vendor-reported language-ID figures (Saaras V4 blog, 25 September 2026; docs accessed 20 November 2026); OIWER's preprint figures (arXiv v1, March 2026); SN-WER (ACL 2026 workshop) and IndicContextEval (Interspeech 2026) conventions; the SwitchLingua and CS-FLEURS model lists, which predate newer STT releases. Re-check each quarter, and re-run R1 whenever the scorer changes.