Call T08 went wrong in the first word: Hazel's speech-to-text decided the borrower was speaking English, wrote her Spanish as English-sounding noise, and asked for the payment date three times until she hung up. A week earlier Steelyard's STT word error rate had dropped 4 points overnight, and nothing about the model had changed; the scoring job had picked up a different normalizer.
Both trace back to one gap: nobody had written down what a correct transcript of a Spanish or Spanglish caller looks like. This chapter writes it down; the entity metric built on it is in entity error rate.
What happened on the call?
The borrower answered in Spanglish, the STT locked to English after her first word, and Hazel asked for the date three times before the borrower gave up at 71 seconds.
t_ms ch event / text
caller REF: "si, pero my paycheck comes el viernes, so after that I can pay, like the twenty-third"
gloss: "yes, but my paycheck comes on Friday, so after that I can pay, like the twenty-third"
caller STT (language locked to en after the first word): "see pedro my paycheck comes el bee air ness so after that I can pay like the twenty third"
agent TTS: "Sorry, could you tell me the date you can pay?"
caller REF: "el veintitres, the twenty-third, after payday"
gloss: "the twenty-third, the twenty-third, after payday"
caller STT: "el bent he tres the twenty third after payday"
agent TTS: "Just to confirm, is it the third or the twenty-third?"
caller REF: "ay, no. Twenty-three. Two, three." (third clarification; hangs up at 71 s)
gloss: "oh, no. Twenty-three. Two, three."
Diagnosis: a language-identification lock (STT.LID) turned "viernes" [Friday] into "bee air ness" and "veintitres" [twenty-three] into "bent he tres", and the agent broke the two-clarification rule (LOOP.CLARIFY). The shape is a published one. In June 2026 Marcus Chen wrote that a regression suite of 312 recorded production calls all passed, and then a caller who switched between English and another language mid-sentence caused a 3am page: speech recognition mis-segmented the audio and the agent looped on clarifications. The pattern appeared zero times in the 312 calls 1. A suite with no dense code-switching has no way to fail on it.
Why does normalization decide whether the number means anything?
WER is computed after a normalizer rewrites both the reference and the hypothesis, so swapping the normalizer moves WER with the model untouched. For Spanish and Spanglish it decides accents, punctuation, currency symbols, digit groups and number words, which makes it part of the metric and versioned with it.
Steelyard's 4-point drop was a normalizer swap reported as a model gain. This page scores T08's first turn under four policies:
| Policy | What it does before alignment |
|---|
| raw | Splits on whitespace; case, punctuation and hyphens count as errors |
| whisper-basic | Mirrors Whisper's BasicTextNormalizer: lowercase, NFKC, then every mark, symbol and punctuation character becomes a space |
| nfc+fold | Unicode NFC, lowercase, punctuation and symbols removed, digit groups joined, accents folded (the n with a tilde is kept) |
| nfc+fold+num | nfc+fold, then a numeral accept-set: number words, digits and ordinals become one value, and a currency word after a number is dropped |
Steelyard's gold reference keeps Spanish accents (first line). The second line is the logged STT; the third heard every word and wrote them in a formatted style, without accents and with digits.
reference sí, pero my paycheck comes el viernes, so after that I can pay, like the twenty-third
gloss yes, but my paycheck comes on Friday, so after that I can pay, like the twenty-third
logged see pedro my paycheck comes el bee air ness so after that I can pay like the twenty third
formatted si pero my paycheck comes el viernes so after that I can pay like the 23rd
| Hypothesis | raw | whisper-basic | nfc+fold | nfc+fold+num |
|---|
| STT as logged (locked to en) | 50.0% | 29.4% | 29.4% | 31.2% |
| Heard every word, formatted style | 25.0% | 17.6% | 11.8% | 0.0% |
Synthetic Steelyard data; scored for this page with the alignment code from the entity recipe.
In the logged row, raw finds 8 errors in 16 tokens: the three real misses plus "pay," with its comma and "twenty-third" with its hyphen. The next two policies split the hyphen, so 17 tokens and 5 edits: "see" for the word meaning yes, "pedro" for "pero" [but], and "bee air ness" for "viernes" (one substitution, two insertions). Numeral canonicalization makes "twenty third" one value on both sides, the denominator falls to 16, and WER rises to 31.2% with the same mistakes. Three words were misheard; the policy decides whether that reads as 29% or 50%.
The second row contains no recognition error at all, and it scores anywhere from 0.0% to 25.0%.
Which reference convention should the gold set use?
Pick one convention, write it into the policy file, and run every hypothesis through the same normalizer before scoring. Steelyard's convention: words as spoken, numbers as spoken words, Spanish with its accents, English in standard spelling, nothing translated.
The convention moves WER, never the amount check, since entities are compared as integer cents and ISO dates after canonicalization (entity error rate). STT products differ in what they emit by default (digits and currency symbols, words, missing accents), so ask every vendor which output style produced its numbers.
The table scores one utterance against four references, with one hypothesis that heard every word and wrote them in the formatted style above. Synthetic Steelyard data.
| Reference convention | whisper-basic | nfc+fold+num |
|---|
| A. Steelyard: accents kept, numbers as words | 17.6% | 0.0% |
| B. Accents stripped, numbers as words | 11.8% | 0.0% |
| C. Accents kept, numbers as digits | 6.2% | 0.0% |
| D. Spanish translated into English | 35.3% | 25.0% |
hypothesis si pero my paycheck comes el viernes so after that I can pay like the 23rd
A sí, pero my paycheck comes el viernes, so after that I can pay, like the twenty-third
B si, pero my paycheck comes el viernes, so after that I can pay, like the twenty-third
C sí, pero my paycheck comes el viernes, so after that I can pay, like the 23rd
D yes, but my paycheck comes on Friday, so after that I can pay, like the twenty-third
Under whisper-basic the same perfect hypothesis scores 6.2% to 35.3% depending on how someone typed the reference. Under nfc+fold+num, rows A to C agree at zero. Row D does not, and no normalizer can fix it: "yes", "but", "on" and "Friday" are four words the borrower never said. A translated reference measures translation. Build the gold set from native Spanish and Spanglish calls and never translate an English set; non-English RAG evals makes the same argument for text.
Which normalizer bugs should you check first?
Start with whatever normalizer produced the number you were handed. Whisper's basic normalizer keeps accents and leaves numbers alone, so "quince" against "15" and an accented yes against a plain one are errors unless you add rules; it also breaks digit groups into separate words. Then check Unicode normalization.
Whisper's BasicTextNormalizer, the one used for non-English text, lowercases and strips symbols and bracketed text but keeps diacritics by default (remove_diacritics=False) and does not standardize numbers; the number normalizer exists only in the English normalizer 2 3. The same routine is used in MMS, Seamless and AssemblyAI Conformer-1 evaluations and ships in Hugging Face transformers 4, so a published Spanish WER may already carry its choices. Here is what it does on a short Steelyard line, against the policies above:
reference sí, quince dólares hypothesis si 15 dolares
gloss yes, fifteen dollars
whisper-basic sí | quince | dólares vs si | 15 | dolares 3 of 3 wrong, WER 100%
nfc+fold si | quince | dolares vs si | 15 | dolares 1 of 3 wrong
nfc+fold+num si | 15 vs si | 15 0 errors
reference Sí, puedo pagar $1,500.00 el viernes
gloss Yes, I can pay $1,500.00 on Friday
whisper-basic sí | puedo | pagar | 1 | 500 | 00 | el | viernes 8 words, 4 errors against "si puedo pagar 1500 el viernes"
nfc+fold si | puedo | pagar | 1500.00 | el | viernes 6 words, 1 error
Commas and periods are punctuation to the basic normalizer, so one amount becomes three words. It also blanks out any combining mark, harmless for precomposed Spanish letters and destructive for scripts that write vowels as marks; the India edition shows that failure.
Two more checks. Run NFC on both sides: an accented letter typed as one code point or as a letter plus a combining accent gives two different strings, and under raw the pair scores as an error. And parse numbers with a locale instead of comparing strings. In Unicode CLDR 48 data, es-US, es-MX and es-419 use a period decimal and a comma group, like en, while Spain uses a comma decimal and a period group and does not group four-digit numbers, so 1.234 means about one in Mexico and one thousand two hundred thirty-four in Spain 5. The Real Academia Espanola accepts either decimal sign depending on country 6. A WER numeral step that deletes every period reads "1.500" as fifteen hundred, which matches a writer who groups thousands with a period and is wrong under the es-US and es-MX convention, where it means one and a half. The locale tag on the call decides which parse runs, and the entity chapter tests the amount itself.
Figure 1 shows how far the policy can move two systems on one frozen fixture of eight Steelyard utterances (English, Spanish and Spanglish, four amounts). System A writes digits and currency symbols and drops accents; it hears every word except one amount, "two fifty" written as $215. System B writes words with accents, gets every amount right, and makes two word errors ("sip" for "ZIP", "lo" [it] for "los" [the]).
reference yeah I can do two fifty on the twentieth, rest next month
A yeah I can do $215 on the 20th, rest next month
B yeah I can do two fifty on the twentieth rest next month
reference puedo pagar mil quinientos dólares el veinte
gloss I can pay one thousand five hundred dollars on the twentieth
A puedo pagar $1,500 el 20
B puedo pagar mil quinientos dólares el veinte
reference my ZIP is eight five seven one nine
A my ZIP is 85719
B my sip is eight five seven one nine
Figure 1: Synthetic Steelyard data, eight utterances scored for this page. Under raw, whisper-basic and nfc+fold, system B wins by 18 to 30 points. Under nfc+fold+num, system A wins, 1.6% to 3.2%, and its one remaining error is the amount it wrote as 215 dollars. Amount errors stay at A 1 of 4 and B 0 of 4 under every policy because amounts are canonicalized before they are compared.
The scorer can reorder vendors. It cannot make system A's $35.00 mistake go away, which is why WER never stands in for the amount check.
What changes when a language uses two scripts?
Pick a script in the reference convention too, because exact-match WER counts the same word written in two scripts as an error. English and Spanish share the Latin alphabet, so Steelyard faces only accents and numerals; a language written in two scripts adds the script choice, and a mapping file joins the metric.
Serbian (Cyrillic and Latin), Indian languages written in their own scripts and in Roman letters, and Arabic typed in Latin letters are the common cases. Transliterated WER (T-WER), from the MUCS 2021 challenge, counts a word as correct in either script 7, and Orthographically informed WER (OIWER) credits permitted spelling variants 8; both depend on that mapping file, so it is hashed with the policy. The India edition applies both metrics to its code-switched calls.
Is the language setting a test variable?
Yes. On a public Spanish-English code-switched telephone test, one model's WER moved by 12 points between being told to decode English and being told to decode Spanish, and by 13 points between forced English and automatic language detection. Test every language setting a vendor offers on your Spanglish slice before you accept a default.
From Ugan et al. (PIER, ICASSP 2025) 9, on a 986-utterance, 1.63-hour test set drawn from Fisher:
| whisper-large-v3 decoding language | WER on the Fisher Spanish-English test |
|---|
| Forced English | 42.58% |
| Forced Spanish | 30.57% |
| Automatic detection | 29.43% |
Steelyard runs automatic detection, a setting to measure, not assume. T08 shows its limit: automatic detection can still lock inside one utterance, which a per-utterance language label cannot see, so Steelyard measures locks directly (the gate below).
How much does code-switching cost?
There is no portable number. It depends on the model, its training data and the language pair; published results run from a small change either way to over double the monolingual error. Measure it within speakers on your calls and score switched words separately, since overall WER can improve while they get worse.
The Whisper paper reports 3.0% WER for large-v2 on FLEURS Spanish read speech 10. In the PIER paper, whisper-large-v3 scored 4.91% on Common Voice Spanish and 29.43% (19.17% CER) on the Fisher code-switched set 9. That gap mixes code-switching with a move from read speech to telephone conversation, so it is an upper bound on the penalty, not a measurement of it.
With a model trained on pooled monolingual and code-switched speech, the penalty ran from roughly -10% to +15% relative (Shah et al., Microsoft, 2020) 11. Off-the-shelf Whisper-large-v3 had a character error rate more than double the monolingual rate on CS-FLEURS (Yan et al., Interspeech 2025) 12, but that paper reports group averages, finds Whisper did better on its human-read code-switched test than on original FLEURS, and gives no Spanish-English figure, so "2x" is not a Spanish-English penalty. For Hindi-English, a bilingual model trained with 2,600 hours of code-mixed speech still scored 31.4% WER on code-mixed utterances, against 15.1% and 18.2% on the two monolingual sets (Patil et al., IEEE SLT 2022) 13. These compare different sentences per condition, so content rides along with the switching.
The PIER paper's sharper finding is about what overall WER hides. Point-of-interest error rate (PIER) scores only the words of interest, here the code-switched words. Fine-tuning whisper-large-v3 on monolingual data cut overall Fisher WER from 29.43% to 27.94%, while PIER on the switched words, with automatic language detection, rose from 36.41 to 50.78 9. A review of the first number alone ships the model that is worse at exactly the words T08 lost.
Figure 2: Published results, not Steelyard data: Ugan et al., PIER (ICASSP 2025), Tables III and IV, automatic language detection. Overall WER fell 1.49 points while the error on code-switched words rose 14.37 points.
Public data has limits that matter for a phone agent:
| Dataset | What it is | Limit for a collections agent |
|---|
| Bangor Miami corpus 14 | 56 recordings of informal conversation among 84 Miami speakers, 35 hours, 242,475 words, recorded 2008 to 2011 | Conversation, not calls; we cite no WER from it, since no primary source was found for the figure in circulation |
| Fisher Spanish, LDC2010S01 15 | About 163 hours of telephone speech: 819 conversations from 136 native speakers in the continental US and Puerto Rico, 8 kHz mu-law, two channels | The closest public match to US telephone Spanish; licensed through LDC, not free |
| CS-FLEURS spa-eng 12 | Code-switched sentences in a human-read test set and two synthetic TTS sets | Machine-generated text, unvalidated TTS, CC BY-NC 4.0: read or synthetic, non-commercial |
CAUTION
Vendor-reported. Treat it as a hypothesis to test on your own calls.
A June 2026 ServiceNow benchmark on the Hugging Face blog tested seven commercial and open STT models on 918 code-switched IT and HR records, 259 of them Spanish-English, with all audio produced by TTS 16. It is a vendor blog on synthetic audio, so use it as a pointer to an open harness, not as evidence about real callers.
For a defensible penalty, compare within speakers: take gold-v7 speakers with both low-density and high-density utterances, compute each slice's WER and PIER, and bootstrap the difference by resampling speakers (clustered standard errors). That holds voice, handset and accent constant.
How do you score code-switch density?
Tag each reference token with a language, count the switch points between adjacent tagged tokens, bucket each utterance as low, medium or high, and report every STT and task metric per bucket. A corpus average buries the dense turns where T08-type failures live, and at Steelyard 14% of accounts prefer Spanglish.
The slice is common in the US: in a Pew survey fielded in 2022 and published in 2023, 63% of US Latinos said they use Spanglish at least sometimes, and 24% said they can hold a Spanish conversation only a little or not at all 17, which is why language comes from the account preference and the call, never a surname.
Write three tagging rules into the policy. Numerals and names are neutral. Tags come from the reference, never from the STT, since T08's STT tags are the failure being measured. And decide whether a spoken number word is a numeral (neutral) or a word in its language, because the choice moves the score:
| Utterance | Tagged tokens | Switch points | Switch-point fraction |
|---|
| T01: "yeah I can do two fifty on the twentieth, rest next month" | 9 | 0 | 0/8 = 0.00 |
| T08 turn 1: "si, pero my paycheck comes el viernes, so after that I can pay, like the twenty-third" | 15 | 3 | 3/14 = 0.21 |
| T08 turn 2, number words neutral: "el veintitres, the twenty-third, after payday" | 4 | 1 | 1/3 = 0.33 |
| T08 turn 2, number words tagged by language | 6 | 1 | 1/5 = 0.20 |
The fraction is switch points over tagged tokens minus one. One rule about number words moves T08's second turn from 0.33 to 0.20, so the rule is versioned with everything else.
Caller turns in collections are short, so bucket on the switch count rather than the fraction: 0 switches is low, 1 or 2 medium, 3 or more high. T08's first turn is high. Check each bucket has enough speakers for a clustered interval, and report the high bucket every release. Density is a monitored slice at Steelyard, not one of the 12 gated segments; the lock check below gates STT version changes instead. The entity error rate chapter runs the same slice on amounts and dates.
Should you use lenient or LLM-based scoring?
Be lenient about spelling and strict about meaning. Folding accents and accepting "15" for "quince" make Spanish WER honest; crediting a close-sounding near-miss hides a wrong amount. Keep the entity exact match separate, and never gate on an LLM-scored WER you cannot recompute next quarter.
Accent folding is lenient for a reason: the word meaning if and the accented word meaning yes sound the same, so the accent is the STT's spelling guess, and WER should not charge for a distinction no microphone hears. Whether the borrower said yes is the entity layer's call, made against the read-back question.
Variant lists are where leniency goes wrong. A proposed list for Steelyard will contain pairs that sound close and mean different things: fifty and fifteen, "doce" and "dos" [twelve and two], "trece" and "tres" [thirteen and three]. Keep number words, negations and confirmations ("si", "no", "aja" [yes, no, uh-huh]) out of every variant set, and have a bilingual reviewer prune the rest. A list that accepted "fifteen" for "fifty" would turn T01's $35.00 error into a correct transcript.
An LLM that compares two transcripts for meaning is a judge, not a string metric: its score moves with the model version and prompt, so a rerun after retirement differs. Use it to rank which utterances to listen to first; if you report it, pin model and prompt, validate it against human labels (audio judges), and keep it out of the gate.
What gate would have caught it?
A language-lock check on the high-density slice measures the failure before release, PIER catches a model that trades switched words away, and the clarification-loop check flags T08 on the call. Under all three, a scorer hash stops any WER comparison across scorer versions.
The checks, in the order Steelyard runs them:
- Scorer hash on every number. Hash the policy file, the numeral accept-set, the tagging rules and the scorer code. Every WER and dashboard tile carries it, and the comparison job refuses to diff across hashes. The 4-point drop fails this check the morning it appears.
- Normalizer fixture in CI. A frozen Spanish and Spanglish fixture, including "$1,500.00", a decomposed accent and "quince" against "15", must score its expected WER under the pinned policy. A swapped-in basic normalizer, a dropped NFC step or a missing locale tag fails the build.
- Language-lock rate on the high-density slice. An utterance is locked when, after a switch point, the STT tags every remaining token with one language while the reference switches back. T08 is locked: every STT tag after the first word is en, and the reference returns to Spanish at "el viernes". Compare a candidate STT version with production on the same utterances by paired bootstrap; hold it for sign-off when the interval for the difference lies above zero.
- PIER on the Spanglish slice. Report switched-word error beside WER on every STT change; a WER gain with a PIER loss is a hold, not a win.
- LOOP.CLARIFY on 100% of calls and on every simulated run. A third clarification on one slot is a deterministic trace event. Run code-switching callers such as P12 through the production audio path (voice simulation) and fail any run that loops.
- A regression case cut from T08. Its first turn, reference and audio go into the gold set with the high-density label.
These are engineering checks, not legal duties. Steelyard collects its own debts in its own name, so the FDCPA and Regulation F largely do not bind it 18 19; it adopts their conduct rules as policy, and state law, the TCPA and the CFPB's ban on unfair, deceptive or abusive practices still apply.
The hash covers a file like this one, abbreviated. Everything that can move a WER without touching the model is in it, so a diff of two files explains a diff of two dashboards:
policy: stt-norm-v1
unicode: NFC
case: lower
punctuation: remove # the decimal point inside a number is kept
accents: fold # n with a tilde is kept
numerals:
locale_from: call.locale # en-US, es-US, es-ES; never guessed from the text
accept_set: [digits, number_words, ordinals]
drop_currency_word_after_number: true
fillers_dropped: [um, uh, eh]
never_dropped: [si, no, aja, uh-huh]
variant_sets: reviewed/variants-v1.tsv # no number words, negations or confirmations
density:
tags_from: reference
neutral: [numerals, names, number_words]
buckets: {low: 0, medium: [1, 2], high: 3+}
report_with: [wer, pier_switched_words, lock_rate]
Interval rule: Wilson only for a single binary proportion, such as the share of simulated P12 runs that loop. Utterance-level rates, corpus WER, PIER and the lock rate use a bootstrap clustered by speaker, and the release comparison uses the paired bootstrap (bootstrap and paired tests).
What goes in the decision memo?
Record the scoring policy as a dated decision with an owner and the events that force a re-baseline. Steelyard's memo:
- Decision: Normalization policy v1 for every STT number: NFC, lowercase, punctuation out; accented references, folded for WER (n with a tilde kept); locale-tagged numerals (en-US, es-US, es-ES) with an accept-set ("15", "fifteen", "quince"); fillers "um", "uh", "eh" dropped, never "si", "no", "aja" or "uh-huh"; density tagging rules; PIER beside WER; a scorer hash on every report.
- Context: WER fell 4 points overnight from a scorer change alone; T08's lock is invisible to WER and to a per-utterance language label.
- Options considered: The unpinned harness default (splits "$1,500.00" into three words, leaves "quince" unequal to "15"); accent-stripped references (no gain, since the scorer folds); uncurated variant credit (admits "fifteen" for "fifty").
- Evidence: On an 8-utterance synthetic fixture the policies moved system A from 32.9% to 1.6% and flipped the ranking against B; amount errors held at 1 of 4 and 0 of 4. gold-v7 (1,200 utterances, 180 speakers) is re-scored under v1, with speaker-clustered intervals, before any comparison.
- Owner and cadence: STT evaluation lead; quarterly, and on any STT vendor, output-style, language-setting or harness change.
- Tripwires: Any scorer change re-baselines every number; a PIER or lock-rate regression on the Spanglish or high-density slice holds an STT change even when WER improves.
- Dated facts: Whisper's basic normalizer keeps diacritics and does not standardize numbers (checked November 2026); PIER figures from ICASSP 2025; CLDR 48 separator data.
What to do this week
Start with the scorer, because every other number on this page depends on it. Metric targets and a day-1 starter live in the voice playbook.
- Read the normalizer code behind your STT numbers. Whisper's basic normalizer with no extra rules charges number formatting and accent spelling as recognition errors.
- Write the policy file: NFC, accent handling, reference convention, locale tags, numeral accept-set, filler list, density tagging rules. Version it like a dataset (versioning and lineage), hash it, and print the hash beside every WER you report.
- Re-score last quarter's stored hypotheses under the new policy before comparing anything. Keep raw STT output in the trace; normalized text cannot be un-normalized.
- Run your STT on the Spanglish slice under every language setting it offers, and add PIER on switched words to the STT report.
- Listen to ten high-density production calls and count the language locks.
NOTE
What will date on this page. The openai/whisper BasicTextNormalizer behavior (checked November 2026); the PIER results and model versions (ICASSP 2025, arXiv v2); the CS-FLEURS and ServiceNow model lists (Interspeech 2025; June 2026 blog), which predate newer STT releases; the CLDR 48 separator data; the Pew survey (fielded 2022, published September 2023). Re-check each quarter, and re-run the fixture whenever the scorer changes.