On call T01, Ryan Mitchell told Hazel "two fifty" and the speech-to-text (STT) output said "two fifteen". One word out of twelve was wrong, so the utterance WER was 8.3%, and the promise to pay went into the CRM $35.00 short. This recipe builds the scorer that catches that call: canonicalizers that turn English and Spanish amounts into integer cents and spoken dates into ISO dates, an entity error rate per entity type, a speaker-clustered bootstrap interval, and the WER scoring policies that the normalization chapter compares. All data is synthetic and generated inside the script, so your run prints the same numbers as this page.
Soniox makes the point with an illustration, not a measurement: if phone numbers are 10 of a transcript's 1,000 words, getting every one wrong costs 1 point of WER, so a transcript can be 97% right overall and 0% right on phone numbers 1. Cresta's four-layer speech-to-text evaluation gives business-critical tokens (phone numbers, emails, amounts, dates) their own entity layer next to lexical WER 2. A collections agent lives or dies on amounts, dates and a yes or a no, so that layer is where the release gate belongs.
The script and synthetic data for this recipe are also in the ai-evals-course repository on GitHub, in the folder voice-ai/labs/voice-entity-error-rate/. That copy also carries the India edition's block, which this page leaves out.
What you build
One standard-library Python file, voice_entity_error_rate.py, that you paste together from the six code blocks below and run once. It contains:
canon_amount(text, locale): number words in English and Spanish, the multipliers hundred, thousand and mil [thousand], the amount shorthand ("two fifty" is $250.00), currency words and the dollar sign, and digits read with the locale's separators, all mapped to integer cents.
extract(text, call_date, locale): every amount, date and yes or no in an utterance, canonicalized. Dates cover "the twentieth", "November twentieth", "next Friday", "el veinte de noviembre" [the twentieth of November], "el viernes" [Friday] and a numeric "12/03", read month first in en-US and day first in es-US.
wer(ref, hyp, policy) with substitution, deletion and insertion counts, a CER, and four scoring policies: raw, a mirror of Whisper's basic normalizer, NFC only, and NFC with accent folding. Numeral canonicalization can sit on top of any of them. switch_word_counts scores only the Spanish words of a Spanglish turn, in the style of the PIER metric.
- Two seeded synthetic sets in the shape of Steelyard's gold-v7 STT set: 400 utterances from 60 speakers and 1,200 utterances from 180 speakers, with English, Spanish-dominant and Spanglish speakers in the 64/18/14 account-preference mix. Each is scored for two STT systems with seeded error injection: number-word substitutions such as "fifty" heard as "fifteen", negation flips, dropped yeses, dropped accents, amounts written as digits, and Spanish words turned into English sounds when language identification sticks on English.
ratio_ci: a bootstrap that resamples speakers, not utterances, 1,000 times.
- An
--edition flag. The default, global, runs everything on this page.
The script asserts every number this page treats as fixed: the eight surface forms of $1,500.00, the "1.500" locale trap, the T01 WER of 1/12, T01's amount error of 1/1, that the canonicalizer recovers the generator's gold entities from all 1,600 reference transcripts, and the switched-word count on T08.
Step 1: Make a clean environment
There is nothing to install. A fresh virtual environment proves it:
python3.11 -m venv .venv
source .venv/bin/activate
python --version # Python 3.11.x
The code blocks in steps 2 to 7 go into one file, voice_entity_error_rate.py, in order.
Step 2: Copy the Steelyard constants and the lexicons
Every voice recipe in this cookbook starts with the same GLOBAL block, so the recipes agree on product codes, campaigns, the Customer Advocate and the entity forms without importing each other. Under it sit the word lists the canonicalizer needs. The scorer carries a version string, steelyard-scorer-v1, because changing a policy changes every WER you report.
"""Entity error rate for spoken amounts and dates (aievals.co recipe R1).
Standard library only. Python 3.11.
Run: python voice_entity_error_rate.py (global edition, the default)
python voice_entity_error_rate.py --edition india (India edition)
"""
from __future__ import annotations
import argparse
import datetime as dt
import random
import re
import unicodedata
from collections import Counter, defaultdict
from fractions import Fraction
EDITIONS: dict = {} # edition name -> (constants block, report function)
# ---- GLOBAL constants (copied from the global world bible, world-global-v1.1) ----
GLOBAL = {
"lender": "Steelyard Credit, Inc.",
"dataset": "steelyard-calls (synthetic, seed 20261121)",
"products": {"AUTO": "Auto loan", "PIL": "Personal installment loan, digital",
"POS": "Retail point-of-sale financing", "SBL": "Small business loan",
"SDL": "Credit-builder and small-dollar loan"},
"campaigns": {"AUTO-early-nov": "AUTO", "PIL-digital-nov": "PIL", "POS-short-nov": "POS",
"SBL-guar-nov": "SBL", "SDL-biwk-nov": "SDL"},
"calling_window_local": {"default": ("08:00", "21:00"), "SDL": ("09:00", "20:00")},
"customer_advocate": {"name": "Ms Dana Brooks", "email": "advocate@steelyard.example",
"phone": "(520) 555-0199",
"address": "1200 Example Avenue, Tucson, AZ 85701"},
"versions": {"stt": "vendorA-2026-10", "gold": "gold-v7", "taxonomy": "gl-fail-v4"},
"money_unit": "integer cents in tools and the CRM; dollars in speech",
"language_groups": ["English", "Spanish-dominant", "Spanglish"],
"amount_shorthand": {"two fifty": "250.00", "two fifteen": "215.00"},
"usd_1500_forms": [("fifteen hundred", "en-US"), ("one thousand five hundred", "en-US"),
("1,500", "en-US"), ("1500", "en-US"), ("$1,500.00", "en-US"),
("mil quinientos", "es-US"), ("mil quinientos dolares", "es-US"),
("1.500", "es-ES")],
"separators": {"en-US": (",", "."), "es-US": (",", "."), # (group, decimal), CLDR 48
"es-MX": (",", "."), "es-ES": (".", ",")},
"date_order": {"en-US": "mdy", "es-US": "dmy", "es-MX": "dmy", "es-ES": "dmy"},
}
SEED = 20261121
SCORER_VERSION = "steelyard-scorer-v1"
# ---- Lexicons (English and Spanish, accents folded) -------------------------
EN_NUM = {w: i for i, w in enumerate(
"zero one two three four five six seven eight nine ten eleven twelve thirteen "
"fourteen fifteen sixteen seventeen eighteen nineteen".split())}
EN_NUM.update({"twenty": 20, "thirty": 30, "forty": 40, "fifty": 50, "sixty": 60,
"seventy": 70, "eighty": 80, "ninety": 90})
ES_NUM = {w: i for i, w in enumerate(
"cero uno dos tres cuatro cinco seis siete ocho nueve diez once doce trece catorce "
"quince dieciseis diecisiete dieciocho diecinueve veinte veintiuno veintidos "
"veintitres veinticuatro veinticinco veintiseis veintisiete veintiocho "
"veintinueve".split())}
ES_NUM.update({"treinta": 30, "cuarenta": 40, "cincuenta": 50, "sesenta": 60,
"setenta": 70, "ochenta": 80, "noventa": 90, "cien": 100, "ciento": 100,
"doscientos": 200, "trescientos": 300, "cuatrocientos": 400,
"quinientos": 500, "seiscientos": 600, "setecientos": 700,
"ochocientos": 800, "novecientos": 900})
NUM = {**EN_NUM, **ES_NUM}
MULT = {"hundred": 100, "thousand": 1000, "mil": 1000, "million": 1_000_000,
"millon": 1_000_000, "millones": 1_000_000}
ORDINAL = {w: i + 1 for i, w in enumerate(
"first second third fourth fifth sixth seventh eighth ninth tenth eleventh "
"twelfth thirteenth fourteenth fifteenth sixteenth seventeenth eighteenth "
"nineteenth twentieth".split())}
ORDINAL.update({"thirtieth": 30, "primero": 1})
WEEKDAY = {d: i for i, d in enumerate("monday tuesday wednesday thursday friday saturday "
"sunday".split())}
WEEKDAY.update({d: i for i, d in enumerate("lunes martes miercoles jueves viernes sabado "
"domingo".split())})
MONTH = {m: i % 12 + 1 for i, m in enumerate(
"january february march april may june july august september october november "
"december enero febrero marzo abril mayo junio julio agosto septiembre octubre "
"noviembre diciembre".split())}
RELATIVE = {"today": 0, "tomorrow": 1, "hoy": 0, "mañana": 1, "manana": 1}
CURRENCY = {"dollars", "dollar", "bucks", "dolares", "dolar"}
YES, NO = {"yes", "yeah", "yep", "si", "okay", "ok", "correct"}, {"no", "nope"}
YES_PAIRS = (["that's", "right"], ["eso", "es"])
BACKCHANNEL = {"uh-huh", "mm-hm", "mhm", "aja", "hmm"}
The lexicons are plain ASCII because entity_tokens folds accents before any lookup, so "dolares" [dollars] and its accented spelling hit the same entry. The fold keeps the n with a tilde, which is a separate letter in Spanish. Two choices in this block are policy, not linguistics. A doubled "si si" [yes yes] and "uh-huh" are backchannels, not a yes: on T01, Ryan answered "uh-huh, okay" after Hazel stated the wrong amount, without registering the change, and a backchannel never confirms a changed amount. And YES_PAIRS accepts "that's right" and "eso es" [that's it] as confirmations only at the start of a turn.
Step 3: Canonicalize amounts and dates
Entity error rate is the share of gold entities that the hypothesis does not recover exactly, after both sides are canonicalized. It is exact match on canonical values, not edit distance on strings: "two fifteen" against "two fifty" is one wrong amount, whatever the character overlap.
# ---- Canonicalization ------------------------------------------------------
def fold(s: str) -> str:
"""Drop accents (acute, diaeresis); keep the n with a tilde, which is a different letter."""
return "".join(c if c in "ñÑ" else "".join(
x for x in unicodedata.normalize("NFD", c) if not unicodedata.combining(x))
for c in unicodedata.normalize("NFC", s))
def entity_tokens(text: str) -> list[str]:
toks = []
for t in fold(text.lower()).split():
t = t.strip(",.?!;:\"()")
parts = t.split("-")
if len(parts) > 1 and all(p in NUM or p in ORDINAL for p in parts):
toks += parts # "twenty-five", "twenty-third"
elif t:
toks.append(t)
return toks
def _numeric(t: str, locale: str = "en-US") -> Fraction | None:
"""Digits read with the locale's separators: '1.500' is 1500 in es-ES and 1.5 in es-MX."""
group, dec = GLOBAL["separators"][locale]
t, g, d = t.lstrip("$"), re.escape(group), re.escape(dec)
if re.fullmatch(rf"\d{{1,3}}(?:{g}\d{{3}})+(?:{d}\d+)?|\d+(?:{d}\d+)?", t):
return Fraction(t.replace(group, "").replace(dec, "."))
return None
def _is_num_at(toks: list[str], i: int, locale: str = "en-US") -> bool:
t = toks[i]
nxt, prv = (toks[i + 1] if i + 1 < len(toks) else ""), (toks[i - 1] if i else "")
if t == "a": # "a hundred twenty thousand"
return nxt in ("hundred", "thousand")
if t == "y": # "treinta y cinco"
return NUM.get(prv, 0) in range(20, 100, 10) and NUM.get(nxt, 10) < 10
return _numeric(t, locale) is not None or t in NUM or t in MULT
def _run_end(toks: list[str], i: int, locale: str = "en-US") -> int:
j = i
while j < len(toks) and _is_num_at(toks, j, locale):
j += 1
return j
def _day_ordinal(toks: list[str], i: int) -> tuple[int, int] | None:
"""'20th', 'twentieth', 'twenty third' or 'el primero' -> (day, index after it)."""
t, nxt = toks[i], (toks[i + 1] if i + 1 < len(toks) else "")
m = re.fullmatch(r"(\d{1,2})(st|nd|rd|th)", t)
if m:
return int(m.group(1)), i + 1
if t in ("twenty", "thirty") and ORDINAL.get(nxt, 99) <= 9:
return NUM[t] + ORDINAL[nxt], i + 2
return (ORDINAL[t], i + 1) if t in ORDINAL else None
def _fmt(v: Fraction) -> str:
return str(v.numerator) if v.denominator == 1 else str(float(v))
def number_value(tokens: list[str], locale: str = "en-US") -> Fraction | None:
"""Western place values. In an amount, 'X fifty' with X from one to nine is X hundred fifty."""
words = ["one" if t == "a" else t for t in tokens if t != "y"]
if (len(words) > 1 and 1 <= EN_NUM.get(words[0], 0) <= 9
and all(w in EN_NUM for w in words[1:])):
rest = number_value(words[1:])
if rest is not None and 10 <= rest <= 99:
return EN_NUM[words[0]] * 100 + rest # "two fifty" = 250, "four twenty-five" = 425
total, cur, seen = Fraction(0), Fraction(0), False
for t in words:
if t in MULT:
if MULT[t] == 100:
cur = (cur or 1) * 100
else:
total, cur = total + (cur or 1) * MULT[t], Fraction(0)
seen = True
continue
v = _numeric(t, locale)
if v is None:
v = NUM.get(t)
if v is None:
return None
cur, seen = cur + v, True
return total + cur if seen else None
def _cents(dollars: Fraction | None) -> int | None:
return None if dollars is None or (dollars * 100).denominator != 1 else int(dollars * 100)
def canon_amount(text: str, locale: str = "en-US") -> int | None:
"""Spoken or written amount -> integer cents, or None."""
toks = [t for t in entity_tokens(text) if t not in CURRENCY]
ok = toks and _run_end(toks, 0, locale) == len(toks)
return _cents(number_value(toks, locale)) if ok else None
def resolve_day(day, call_date: dt.date) -> dt.date | None:
if day is None or Fraction(day).denominator != 1 or not 1 <= day <= 31:
return None
y, m = call_date.year, call_date.month
for _ in range(3):
try:
d = dt.date(y, m, int(day))
if d >= call_date:
return d
except ValueError:
pass
y, m = (y + 1, 1) if m == 12 else (y, m + 1)
return None
def resolve_day_month(day, month: int, call_date: dt.date) -> dt.date | None:
try:
d = dt.date(call_date.year, month, int(day))
except (ValueError, TypeError):
return None
return d if d >= call_date else d.replace(year=d.year + 1)
def resolve_weekday(wd: int, call_date: dt.date) -> dt.date:
"""Policy: 'next Friday' and 'el viernes' are the first Friday strictly after the call date."""
return call_date + dt.timedelta(days=(wd - call_date.weekday()) % 7 or 7)
def extract(text: str, call_date: dt.date, locale: str = "en-US") -> list[tuple[str, object]]:
"""All amount, date and confirmation entities in one utterance, canonicalized."""
toks, out, i = entity_tokens(text), [], 0
if toks:
if toks[:2] == ["si", "si"] or toks[0] in BACKCHANNEL:
pass # a backchannel is never a yes
elif toks[0] in YES or toks[:2] in YES_PAIRS:
out.append(("confirm", "yes"))
elif toks[0] in NO:
out.append(("confirm", "no"))
while i < len(toks):
t, prv = toks[i], (toks[i - 1] if i else "")
if t in WEEKDAY:
out.append(("date", resolve_weekday(WEEKDAY[t], call_date)))
i += 1
elif t in RELATIVE:
out.append(("date", call_date + dt.timedelta(days=RELATIVE[t])))
i += 1
elif re.fullmatch(r"\d{1,2}/\d{1,2}", t): # date order follows the locale
a, b = map(int, t.split("/"))
m, d = (a, b) if GLOBAL["date_order"][locale] == "mdy" else (b, a)
out.append(("date", resolve_day_month(d, m, call_date) if 1 <= m <= 12 else None))
i += 1
elif t in MONTH and i + 1 < len(toks) and (
_day_ordinal(toks, i + 1) or _numeric(toks[i + 1]) is not None):
day, j = _day_ordinal(toks, i + 1) or (_numeric(toks[i + 1]), i + 2)
out.append(("date", resolve_day_month(day, MONTH[t], call_date)))
i = j
elif _day_ordinal(toks, i):
day, i = _day_ordinal(toks, i)
out.append(("date", resolve_day(day, call_date)))
elif _is_num_at(toks, i, locale):
j = _run_end(toks, i, locale)
run, after = toks[i:j], (toks[j] if j < len(toks) else "")
val = number_value(run, locale)
if after == "de" and j + 1 < len(toks) and toks[j + 1] in MONTH:
out.append(("date", resolve_day_month(val, MONTH[toks[j + 1]], call_date)))
j += 2 # "el veinte de noviembre"
elif prv == "el" and val is not None and val <= 31 and after not in CURRENCY:
out.append(("date", resolve_day(val, call_date))) # "el veinte"
elif val is not None and (val >= 100 or after in CURRENCY or run[0][0] == "$"):
out.append(("amount", _cents(val)))
i = j
else:
i += 1
return out
def entity_counts(gold: list, hyp: list, kind: str) -> tuple[int, int, int]:
"""Exact match after canonicalization. Returns (gold, missed, hallucinated)."""
g = Counter(v for k, v in gold if k == kind)
h = Counter(v for k, v in hyp if k == kind)
hit = sum((g & h).values())
return sum(g.values()), sum(g.values()) - hit, sum(h.values()) - hit
Four rules in this block decide numbers you will report, so write them into your scoring policy document:
- Western place values and one shorthand.
number_value multiplies inside a group at hundred and closes a group at thousand or mil. Spanish hundreds are words of their own that add, so "mil quinientos" [fifteen hundred] is 1,500 and "ciento veinte mil" [a hundred twenty thousand] is 120,000. In an amount, "X fifty" with X from one to nine means X hundred fifty, so "two fifty" is 25000 cents and "two fifteen" is 21500. The two differ by one sound, which is why Hazel reads every amount back.
- Digits follow the locale. In Unicode CLDR 48 data, es-US, es-MX and es-419 use a period decimal and a comma group, like en, while Spain uses a comma decimal and a period group and does not group four-digit numbers, so 1.234 means about one in Mexico and one thousand two hundred thirty-four in Spain 3. The Real Academia Espanola accepts either decimal sign depending on country 4.
_numeric therefore reads "1.500" as 150000 cents when the call is tagged es-ES and as 150 cents when it is tagged es-MX. Never compare amount strings; parse them with the call's locale.
- Dates resolve against the call date, and numeric dates follow the locale. "The twentieth" on a call dated 14 November is 2026-11-20; "the fifth" on the same call is 2026-12-05, because the 5th has passed. "Next Friday" and "el viernes" are the first Friday strictly after the call. In CLDR 48 the short date pattern is M/d/yy for en and d/M/y for es-US 3, so "12/03" is 3 December for an en-US call and 12 March for an es-US one. That is trace T06 from the other side: the English stack wrote month first and the Spanish voice read it day first (speech output chapter).
- A number after "el" is a day. "El veinte" is the twentieth, not an amount of 20, and "el veinte de noviembre" is a full date. A number with "dollars" or a currency word after it is an amount even when it is under 100.
entity_counts reports both directions. A missed entity is a gold amount the hypothesis does not contain; a hallucinated entity is a hypothesis amount the reference does not contain. A substitution like T01's is one of each. Gate on missed (that is the entity error rate) and watch hallucinated, because a hallucinated amount is what gets read back to a borrower.
Step 4: WER, CER, switched words and four scoring policies
# ---- Scoring policies (the normalizer is part of the metric) ---------------
def _join_digit_groups(s: str) -> str:
return re.sub(r"(?<=\d),(?=\d)", "", s)
def norm_raw(s: str) -> list[str]:
return s.split()
def norm_whisper_basic(s: str) -> list[str]:
"""Mirror of Whisper's BasicTextNormalizer: NFKC keeps precomposed accents, no number rules."""
s = s.lower()
s = re.sub(r"[<\[][^>\]]*[>\]]", "", s)
s = re.sub(r"\(([^)]+?)\)", "", s)
s = "".join(" " if unicodedata.category(c)[0] in "MSP" else c
for c in unicodedata.normalize("NFKC", s))
return s.split()
def norm_nfc(s: str) -> list[str]:
"""NFC, lowercase, punctuation and symbols to spaces, digit groups joined, accents kept."""
s = _join_digit_groups(unicodedata.normalize("NFC", s).lower())
s = re.sub(r"(?<=\d)\.(?=\d)", "\x00", s) # keep decimal points
s = "".join(" " if unicodedata.category(c)[0] in "PS" else c for c in s)
return s.replace("\x00", ".").split()
def norm_nfc_fold(s: str) -> list[str]:
return [fold(t) for t in norm_nfc(s)]
def with_numerals(tokens: list[str]) -> list[str]:
"""Numeral canonicalization on top of any policy: a number run or an ordinal becomes its
value, and a currency word right after a number is dropped."""
out, i = [], 0
while i < len(tokens):
j = _run_end(tokens, i)
if _day_ordinal(tokens, i):
day, i = _day_ordinal(tokens, i)
out.append(str(day))
elif j > i:
v = number_value(tokens[i:j])
out.append(_fmt(v) if v is not None else " ".join(tokens[i:j]))
i = j + (j < len(tokens) and fold(tokens[j]) in CURRENCY)
else:
out.append(tokens[i])
i += 1
return out
POLICIES = {"raw": norm_raw, "whisper-basic": norm_whisper_basic,
"nfc": norm_nfc, "nfc+fold": norm_nfc_fold}
# ---- WER, CER and switch-word error -----------------------------------------
def align(ref: list, hyp: list) -> tuple[int, int, int]:
"""Levenshtein alignment; returns (substitutions, deletions, insertions)."""
n, m = len(ref), len(hyp)
d = [[(0, 0, 0, 0)] * (m + 1) for _ in range(n + 1)] # (cost, S, D, I)
for i in range(1, n + 1):
d[i][0] = (i, 0, i, 0)
for j in range(1, m + 1):
d[0][j] = (j, 0, 0, j)
for i in range(1, n + 1):
for j in range(1, m + 1):
c, s, de, ins = d[i - 1][j - 1]
best = (c, s, de, ins) if ref[i - 1] == hyp[j - 1] else (c + 1, s + 1, de, ins)
c, s, de, ins = d[i - 1][j]
best = min(best, (c + 1, s, de + 1, ins))
c, s, de, ins = d[i][j - 1]
best = min(best, (c + 1, s, de, ins + 1))
d[i][j] = best
return d[n][m][1:]
def ref_errors(ref: list, hyp: list) -> list[bool]:
"""One flag per reference token: True if a minimum-edit alignment substitutes or deletes it."""
n, m = len(ref), len(hyp)
d = [[i + j if i * j == 0 else 0 for j in range(m + 1)] for i in range(n + 1)]
for i in range(1, n + 1):
for j in range(1, m + 1):
d[i][j] = min(d[i - 1][j - 1] + (ref[i - 1] != hyp[j - 1]),
d[i - 1][j] + 1, d[i][j - 1] + 1)
flags, i, j = [False] * n, n, m
while i > 0:
if j > 0 and d[i][j] == d[i - 1][j - 1] + (ref[i - 1] != hyp[j - 1]):
flags[i - 1], i, j = ref[i - 1] != hyp[j - 1], i - 1, j - 1
elif d[i][j] == d[i - 1][j] + 1:
flags[i - 1], i = True, i - 1
else:
j -= 1
return flags
def wer(ref: str, hyp: str, policy: str = "nfc+fold", numerals: bool = False):
r, h = POLICIES[policy](ref), POLICIES[policy](hyp)
if numerals:
r, h = with_numerals(r), with_numerals(h)
s, de, ins = align(r, h)
return {"S": s, "D": de, "I": ins, "N": len(r), "wer": (s + de + ins) / max(len(r), 1)}
def cer_counts(ref: str, hyp: str) -> tuple[int, int]:
r, h = " ".join(norm_nfc_fold(ref)), " ".join(norm_nfc_fold(hyp))
return sum(align(list(r), list(h))), len(r)
def switch_word_counts(words: list[tuple[str, str]], hyp: str, lang: str = "es"):
"""PIER-style: errors on the reference words tagged `lang` only (insertions not counted).
words = [(reference word, language tag)]. Returns (errors, tagged tokens)."""
ref, tags = [], []
for w, tag in words:
for t in norm_nfc_fold(w):
ref.append(t)
tags.append(tag)
flags = ref_errors(ref, norm_nfc_fold(hyp))
return sum(f for f, t in zip(flags, tags) if t == lang), tags.count(lang)
The policies differ only in what they do before the alignment:
| Policy | What it does to the text | Risk |
|---|
| raw | Splits on whitespace | Case, punctuation and hyphens count as errors |
| whisper-basic | Lowercases, then replaces every mark, symbol and punctuation character with a space after NFKC | Keeps precomposed accents and has no number rules, so "quince" [fifteen] against "15" is an error; splits "1,500" into two words |
| nfc | NFC, lowercase, punctuation and symbols to spaces, digit groups joined, accents kept | An accented and a plain spelling of one word still count as an error |
| nfc+fold | nfc, then accents folded (the n with a tilde kept) | "si" [if] and the accented "si" [yes] become one word; the confirmation check, not WER, has to keep them apart |
The whisper-basic function mirrors BasicTextNormalizer in the openai/whisper repository, which lowercases and strips symbols and bracketed text but keeps diacritics by default (remove_diacritics=False) and does not standardize numbers; the number normalizer exists only in the English normalizer 5. with_numerals is the numeral step on top: a number run or an ordinal becomes one value on both sides, and a currency word right after a number is dropped.
ref_errors walks one minimum-edit alignment back and flags each reference word that was substituted or deleted. switch_word_counts uses it to score only the words tagged with one language. Ugan et al. introduced point-of-interest error rate (PIER) for this reason: on a Spanish-English code-switched Fisher test, fine-tuning whisper-large-v3 on monolingual data cut overall WER from 29.43% to 27.94%, while the error rate on the code-switched words, with automatic language detection, rose from 36.41 to 50.78 6. The function here is a simplified version in that style: it counts substitutions and deletions on the Spanish words of Spanglish turns and ignores insertions.
Step 5: Generate the synthetic sets
# ---- Synthetic data: templates plus seeded error injection -----------------
AMOUNT_FORMS = { # cents -> spoken surface forms by language (WG7 canonical values)
25000: {"en": ["two fifty", "two hundred fifty"], "es": ["doscientos cincuenta dólares"],
"num": ["250", "$250"]},
21500: {"en": ["two fifteen", "two hundred fifteen"], "es": ["doscientos quince"],
"num": ["215"]},
42500: {"en": ["four twenty-five", "four hundred twenty-five"],
"es": ["cuatrocientos veinticinco"], "num": ["$425"]},
13000: {"en": ["one thirty", "a hundred thirty"], "es": ["ciento treinta dólares"],
"num": ["130"]},
35000: {"en": ["three fifty", "three hundred fifty"], "es": ["trescientos cincuenta"],
"num": ["350"]},
68000: {"en": ["six eighty", "six hundred eighty"], "es": ["seiscientos ochenta"],
"num": ["$680"]},
150000: {"en": ["fifteen hundred", "one thousand five hundred"],
"es": ["mil quinientos", "mil quinientos dólares"], "num": ["1,500", "$1,500.00"]},
12000000: {"en": ["a hundred twenty thousand"], "es": ["ciento veinte mil"],
"num": ["120,000"]},
}
EN_ORD = {v: k for k, v in ORDINAL.items() if k != "primero"}
for _d in range(21, 32):
EN_ORD.setdefault(_d, ("twenty-" if _d < 30 else "thirty-") + EN_ORD[_d % 10 or 10])
ES_DAY = {v: k for k, v in ES_NUM.items() if v <= 31}
ES_DAY.update({1: "primero", 16: "dieciséis", 22: "veintidós", 23: "veintitrés",
26: "veintiséis"})
WEEKDAYS = {"en": ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"],
"es": ["lunes", "martes", "miércoles", "jueves", "viernes", "sábado"]}
ES_MONTH = ("enero febrero marzo abril mayo junio julio agosto septiembre octubre "
"noviembre diciembre").split()
TEMPLATES = { # language group -> [(pieces [(text, language)], polarity of the opening word)]
"English": [
([("yes I can pay {A} on {D}", "en")], "yes"),
([("yeah I can do {A} on {D}, rest next month", "en")], "yes"),
([("right now I only have {A}, the rest {D}", "en")], None),
([("I'll send {A} by {D}", "en")], None),
([("I already paid {A} last week", "en")], None),
([("can you text me the link for {A}", "en")], None),
([("no no, I said {A}", "en")], "no"),
([("after payday, {D}", "en")], None),
([("yes", "en")], "yes"), ([("no", "en")], "no"),
([("yes that's right", "en")], "yes"), ([("no, that's wrong", "en")], "no")],
"Spanish-dominant": [
([("sí, puedo pagar {A} {D}", "es")], "yes"),
([("ahorita solo tengo {A}, el resto {D}", "es")], None),
([("le pago {A} {D}", "es")], None),
([("ya pagué {A} la semana pasada", "es")], None),
([("mándeme el enlace de {A}", "es")], None),
([("no, no, dije {A}", "es")], "no"),
([("sí", "es")], "yes"), ([("no", "es")], "no"),
([("sí, eso es", "es")], "yes"), ([("no, está mal", "es")], "no")],
"Spanglish": [
([("sí, pero", "es"), ("my paycheck comes", "en"), ("{D}", "es"),
("so after that I can pay", "en")], "yes"),
([("I can do", "en"), ("{A}", "en"), ("{D}", "es")], None),
([("puedo pagar", "es"), ("{A}", "es"), ("next week, on", "en"), ("{D}", "en")], None),
([("ya pagué", "es"), ("{A}", "en"), ("last week", "en")], None),
([("okay,", "en"), ("entonces", "es"), ("{A}", "es"), ("on", "en"), ("{D}", "en")],
"yes"),
([("no, no,", "es"), ("I said", "en"), ("{A}", "es")], "no")],
}
LOCALE = {"English": "en-US", "Spanish-dominant": "es-US", "Spanglish": "es-US"}
NUM_CONFUSION = {"fifty": "fifteen", "fifteen": "fifty", "thirty": "thirteen",
"eighty": "eighteen", "twenty-five": "twenty-nine", "cincuenta": "sesenta",
"doscientos": "trescientos", "trescientos": "doscientos", "quince": "trece",
"treinta": "trece", "quinientos": "cuatrocientos", "seiscientos": "setecientos",
"twentieth": "twelfth", "november": "december", "friday": "monday",
"veinte": "treinta", "noviembre": "diciembre", "viernes": "jueves"}
GEN_VARIANT = {"i'll": "all", "already": "all ready", "right": "write", "rest": "rust",
"send": "sent", "said": "set", "link": "lint", "text": "next", "comes": "come",
"paycheck": "pay check", "only": "lonely", "have": "half", "pagar": "pagan",
"tengo": "tango", "resto": "restó", "entonces": "tonces", "dije": "dice",
"ahorita": "ahora", "enlace": "enlaces", "semana": "semanas", "puedo": "pueda"}
ANGLO = {"sí": "see", "pero": "pedro", "viernes": "bee air ness", "ya": "yeah",
"pagué": "page", "puedo": "pueblo", "pagar": "pa car", "entonces": "and on says",
"dije": "the hay", "veinte": "bane tay", "doscientos": "those see and toes",
"cincuenta": "seen when ta", "quince": "keen say", "ciento": "see and toe",
"treinta": "train ta", "mil": "meal", "quinientos": "key and toes", "de": "day",
"dólares": "dollar is", "veintitrés": "bent he tres", "noviembre": "no be em bray"}
CALL_DATES = [dt.date(2026, 11, d) for d in range(9, 20) if dt.date(2026, 11, d).weekday() != 6]
SYSTEMS = { # per-token error probabilities, scaled by a per-speaker difficulty
"A": {"gen": 0.15, "amt": 0.012, "digit": 0.004, "date": 0.015, "neg": 0.03,
"yes": 0.04, "accent": 0.50, "numfmt": 0.30, "switch": 0.04},
"B": {"gen": 0.10, "amt": 0.050, "digit": 0.012, "date": 0.025, "neg": 0.07,
"yes": 0.07, "accent": 0.15, "numfmt": 0.05, "switch": 0.20},
}
def strip_accents(w: str) -> str:
return "".join(x for x in unicodedata.normalize("NFD", w) if not unicodedata.combining(x))
def render_date(rng: random.Random, call: dt.date, lang: str) -> tuple[str, dt.date]:
kind = rng.choices(["day", "month", "weekday", "other"], [5, 2, 2, 1])[0]
if kind == "weekday":
wd = rng.randrange(6)
day_name = WEEKDAYS[lang][wd]
return (f"next {day_name}" if lang == "en" else f"el {day_name}"), resolve_weekday(wd, call)
if kind == "other" and lang == "es":
return "mañana", call + dt.timedelta(days=1)
target = call + dt.timedelta(days=rng.randint(1, 25))
if lang == "en":
if kind == "other":
suffix = {1: "st", 2: "nd", 3: "rd"}.get(target.day % 10 * (target.day // 10 != 1), "th")
return f"the {target.day}{suffix}", target
if kind == "month":
return f"{target.strftime('%B')} {EN_ORD[target.day]}", target
return f"the {EN_ORD[target.day]}", target
if kind == "month":
return f"el {ES_DAY[target.day]} de {ES_MONTH[target.month - 1]}", target
return f"el {ES_DAY[target.day]}", target
def make_utterance(rng: random.Random, uid: str, speaker: str, group: str) -> dict:
pieces, pol = rng.choice(TEMPLATES[group])
call = rng.choice(CALL_DATES)
segs, gold, style = [], [("confirm", pol)] if pol else [], "none"
for chunk, lang in pieces:
for piece in re.split(r"(\{[AD]\})", chunk):
if piece == "{A}":
cents = rng.choice(list(AMOUNT_FORMS))
forms = AMOUNT_FORMS[cents][lang] + AMOUNT_FORMS[cents]["num"]
form = rng.choice(forms)
digits = f"${cents // 100:,}" # how a formatting STT writes it
segs += [(w, "amt", lang, digits) for w in form.split()]
gold.append(("amount", cents))
first = form.split()[0]
style = ("digits" if first[0] in "$0123456789" else "es-words" if lang == "es"
else "shorthand" if number_value(entity_tokens(form)) is not None
and "hundred" not in form and "thousand" not in form else "en-words")
elif piece == "{D}":
phrase, target = render_date(rng, call, lang)
segs += [(w, "date", lang, None) for w in phrase.split()]
gold.append(("date", target))
else:
for w in piece.split():
role = ("pol" if not segs and w.strip(",") in ("yes", "yeah", "no", "sí", "okay")
else "gen")
segs.append((w, role, lang, None))
ref = " ".join(w for w, *_ in segs)
return {"uid": uid, "speaker": speaker, "group": group, "locale": LOCALE[group],
"call": call, "ref": ref, "segs": segs, "gold": gold, "style": style,
"words": [(w, lang) for w, _, lang, _ in segs]}
def inject(rng: random.Random, utt: dict, p: dict, m: float) -> str:
out, amount_style = [], None
for w, role, lang, digits in utt["segs"]:
w = w.strip(",").lower()
if role == "amt" and amount_style is None: # decided once per amount
spoken = w[0] not in "$0123456789"
amount_style = "digits" if spoken and rng.random() < p["numfmt"] else "as-said"
if amount_style == "digits":
out.append(digits) # same amount, written as digits
if role == "amt" and amount_style == "digits":
continue
if utt["group"] == "Spanglish" and lang == "es" and rng.random() < p["switch"] * m:
out.append(ANGLO.get(w, w[:-1] if len(w) > 3 else w)) # language ID stuck on English
continue
if role in ("amt", "date"):
if w[0].isdigit() and rng.random() < p["digit"] * m and len(w) > 1:
k = rng.randrange(1, len(w))
if w[k].isdigit():
w = w[:k] + str((int(w[k]) + rng.randint(1, 9)) % 10) + w[k + 1:]
elif w in NUM_CONFUSION and rng.random() < p["date" if role == "date" else "amt"] * m:
w = NUM_CONFUSION[w]
elif role == "pol":
if w in ("yes", "yeah", "sí", "okay") and rng.random() < p["yes"] * m:
continue
if w == "no" and rng.random() < p["neg"] * m:
if rng.random() < 0.7:
continue
w = "yeah" if lang == "en" else "sí"
elif rng.random() < p["gen"] * m: # generic word error
w = GEN_VARIANT.get(w, w[:-1] if len(w) > 3 else w + "h")
if w != strip_accents(w) and rng.random() < p["accent"]:
w = strip_accents(w) # many STT outputs drop accents
out.append(w)
if rng.random() < p["gen"] * m * 0.2:
out.append("uh")
return " ".join(out)
def make_set(rng: random.Random, n_utts: int, n_speakers: int, tag: str) -> list[dict]:
groups, weights = GLOBAL["language_groups"], [64, 18, 14] # account preference mix
speakers = [(f"{tag}-spk{k:03d}", min(4.0, max(0.25, rng.lognormvariate(0, 0.6))),
rng.choices(groups, weights)[0]) for k in range(n_speakers)]
utts = []
for u in range(n_utts):
spk, diff, group = speakers[u % n_speakers]
utt = make_utterance(rng, f"{tag}-u{u:04d}", spk, group)
utt["hyp"] = {s: inject(rng, utt, p, diff) for s, p in SYSTEMS.items()}
utts.append(utt)
return utts
The templates are Steelyard caller turns in three language groups: promises with an amount and a date, already-paid claims, link requests, corrections ("no no, I said ..."), payday answers in the shape of T08, and bare confirmations. Spanish references keep their accents, as Steelyard's gold convention does. Each speaker gets one language group and one difficulty multiplier, drawn once and shared by both systems, which is what makes utterances from one speaker correlated. System A makes more generic word errors, drops more accents and writes 30% of spoken amounts as digits. System B makes fewer generic errors and more number-word substitutions, dropped confirmations and negation flips, and its language identification sticks on English more often in Spanglish turns, so Spanish words come out as English sounds ("pero" [but] as "pedro"). Those rates are invented to make the trade-off visible; the method is what transfers.
Step 6: Bootstrap by speaker and score each system
# ---- Speaker-clustered bootstrap -------------------------------------------
def ratio_ci(units: list[tuple[str, int, int]], rng: random.Random, n_boot: int = 1000,
cluster: bool = True) -> tuple[float, float, float]:
"""units = (cluster id, errors, opportunities). Percentile 95% interval of sum/sum."""
groups = defaultdict(lambda: [0, 0])
for i, (cid, e, n) in enumerate(units):
g = groups[cid if cluster else i]
g[0] += e
g[1] += n
keys = list(groups)
point = sum(e for e, _ in groups.values()) / sum(n for _, n in groups.values())
stats = []
for _ in range(n_boot):
e = n = 0
for _k in range(len(keys)):
ge, gn = groups[keys[rng.randrange(len(keys))]]
e, n = e + ge, n + gn
stats.append(e / n if n else 0.0)
stats.sort()
return point, stats[int(0.025 * n_boot)], stats[int(0.975 * n_boot) - 1]
def score(utts: list[dict], system: str) -> dict:
tot = Counter()
units = defaultdict(list)
wrong_amount_wer = [] # utterance WER of every wrong-amount utterance
for u in utts:
hyp, grp = u["hyp"][system], u["group"]
w = wer(u["ref"], hyp)
tot["edits"] += w["S"] + w["D"] + w["I"]
tot["words"] += w["N"]
tot[grp + "_edits"] += w["S"] + w["D"] + w["I"]
tot[grp + "_words"] += w["N"]
ce, cn = cer_counts(u["ref"], hyp)
tot["cedits"], tot["chars"] = tot["cedits"] + ce, tot["chars"] + cn
got = extract(hyp, u["call"], u["locale"])
for kind in ("amount", "date", "confirm"):
g, miss, hall = entity_counts(u["gold"], got, kind)
tot[kind + "_gold"] += g
tot[kind + "_miss"] += miss
tot[kind + "_hall"] += hall
if g:
units[kind].append((u["speaker"], miss, g))
if kind == "amount":
tot[grp + "_amount_gold"] += g
tot[grp + "_amount_miss"] += miss
if miss:
wrong_amount_wer.append(w["wer"])
gp = dict(u["gold"]).get("confirm")
hp = dict(got).get("confirm")
if gp and hp and gp != hp:
tot["flips"] += 1
if grp == "Spanglish":
e, n = switch_word_counts(u["words"], hyp, "es")
tot["switch_err"], tot["switch_n"] = tot["switch_err"] + e, tot["switch_n"] + n
e, n = switch_word_counts(u["words"], hyp, "en")
tot["other_err"], tot["other_n"] = tot["other_err"] + e, tot["other_n"] + n
return {"tot": tot, "units": units, "wrong_amount_wer": wrong_amount_wer}
ratio_ci treats the speaker as the unit. It sums errors and gold entities per speaker, resamples speakers with replacement, and recomputes the pooled ratio each time. Entity error rate is a ratio over utterances that share speakers, so a Wilson interval on the utterance count would assume independence the data does not have. Pass cluster=False to get the utterance-level interval for comparison. The clustered standard errors chapter covers the general case. score also keeps WER and amount errors per language group, and the switched-word counts for Spanglish turns.
Step 7: Run the report
# ---- The report ------------------------------------------------------------
def pct(x: float) -> str:
return f"{100 * x:.1f}%"
def run_global(world: dict) -> None:
print(world["lender"])
print(f"dataset {world['dataset']}; scorer policy nfc+fold ({SCORER_VERSION})")
# Step 1: canonicalization checks
print("\n== 1. Canonicalization ==")
for form, loc in world["usd_1500_forms"]:
cents = canon_amount(form, loc)
print(f" {form:<26} {loc:<6} -> {cents} cents")
assert cents == 150_000, form
mx = canon_amount("1.500", "es-MX")
print(f" {'1.500':<26} {'es-MX':<6} -> {mx} cents (period decimal: one dollar fifty)")
assert mx == 150
wg7 = {"two fifty": 25000, "two hundred fifty": 25000, "$250": 25000,
"doscientos cincuenta": 25000, "two fifteen": 21500, "four twenty-five": 42500,
"cuatrocientos veinticinco dólares": 42500, "a hundred twenty thousand": 12_000_000,
"ciento veinte mil": 12_000_000}
for form, want in wg7.items():
assert canon_amount(form) == want, (form, canon_amount(form))
print(f" {len(wg7)} more amount forms match; two fifteen -> {canon_amount('two fifteen')}")
call = dt.date(2026, 11, 14) # T01's call date, a Saturday
dates = [("the twentieth", "en-US", "2026-11-20"), ("November twentieth", "en-US", "2026-11-20"),
("el veinte de noviembre", "es-US", "2026-11-20"), ("next Friday", "en-US", "2026-11-20"),
("el viernes", "es-US", "2026-11-20"), ("the fifth", "en-US", "2026-12-05"),
("12/03", "en-US", "2026-12-03"), ("12/03", "es-US", "2027-03-12")]
for phrase, loc, want in dates:
got = [v for k, v in extract(phrase, call, loc) if k == "date"]
print(f" {phrase:<26} {loc:<6} -> {got[0].isoformat()} (call date {call})")
assert got[0].isoformat() == want, phrase
# Step 2: the T01 pair
print("\n== 2. T01: one substituted word ==")
ref = "yeah I can do two fifty on the twentieth, rest next month"
hyp = "yeah I can do two fifteen on the twentieth rest next month"
w = wer(ref, hyp)
g, miss, _ = entity_counts(extract(ref, call), extract(hyp, call), "amount")
print(f" WER {w['S'] + w['D'] + w['I']}/{w['N']} = {w['wer']:.3f} (S={w['S']} D={w['D']} I={w['I']})")
print(f" amount entity errors {miss}/{g}: gold {extract(ref, call)[1][1]} cents, "
f"STT {extract(hyp, call)[1][1]} cents")
print(f" raw policy (no punctuation strip): WER {wer(ref, hyp, 'raw')['wer']:.3f}")
assert (w["S"] + w["D"] + w["I"], w["N"]) == (1, 12) and round(w["wer"], 3) == 0.083
assert (miss, g) == (1, 1)
# Step 3: synthetic sets
rng = random.Random(SEED)
small = make_set(rng, 400, 60, "s")
gold7 = make_set(rng, 1200, 180, "g")
for s in (small, gold7): # the canonicalizer agrees with the generator
for u in s:
got = extract(u["ref"], u["call"], u["locale"])
assert sorted(got, key=repr) == sorted(u["gold"], key=repr), u["ref"]
print("\n== 3. Bake-off on the 1,200-utterance set (180 speakers) ==")
print(" extract(reference) == generator gold on all 1,600 utterances")
boot = random.Random(SEED + 1)
res = {s: score(gold7, s) for s in SYSTEMS}
print(" entity error rate = gold entities not recovered exactly / gold entities")
print(f" {'sys':<4}{'WER':>7}{'CER':>7} {'amount, 95% CI':<24}{'date, 95% CI':<24}{'flips':>5}")
for s, r in res.items():
t = r["tot"]
a = ratio_ci(r["units"]["amount"], boot)
d = ratio_ci(r["units"]["date"], boot)
print(f" {s:<4}{pct(t['edits'] / t['words']):>7}{pct(t['cedits'] / t['chars']):>7} "
f"{pct(a[0]) + ' [' + pct(a[1]) + ', ' + pct(a[2]) + ']':<24}"
f"{pct(d[0]) + ' [' + pct(d[1]) + ', ' + pct(d[2]) + ']':<24}{t['flips']:>5}")
r["amount"] = a[0]
r["wer"] = t["edits"] / t["words"]
assert res["B"]["wer"] < res["A"]["wer"] and res["B"]["amount"] > res["A"]["amount"]
for s, r in res.items():
t = r["tot"]
ws = r["wrong_amount_wer"]
print(f" {s}: amounts {t['amount_gold']} gold, {t['amount_miss']} missed, "
f"{t['amount_hall']} hallucinated; {len(ws)} utterances with a wrong amount, "
f"{sum(x < 0.10 for x in ws)} of them at WER under 10%, "
f"{sum(x < 0.20 for x in ws)} under 20%")
print(" by language group:")
print(f" {'group':<18}{'utts':>5}{'WER A':>8}{'WER B':>8}{'amount A':>10}{'amount B':>10}")
for grp in world["language_groups"]:
ta, tb = res["A"]["tot"], res["B"]["tot"]
n_utts = sum(u["group"] == grp for u in gold7)
print(f" {grp:<18}{n_utts:>5}{pct(ta[grp + '_edits'] / ta[grp + '_words']):>8}"
f"{pct(tb[grp + '_edits'] / tb[grp + '_words']):>8}"
f"{pct(ta[grp + '_amount_miss'] / ta[grp + '_amount_gold']):>10}"
f"{pct(tb[grp + '_amount_miss'] / tb[grp + '_amount_gold']):>10}")
print(" Spanglish, errors on the Spanish words only (PIER-style) and on the English words:")
for s, r in res.items():
t = r["tot"]
print(f" {s}: Spanish words {t['switch_err']}/{t['switch_n']} = "
f"{pct(t['switch_err'] / t['switch_n'])}; English words {t['other_err']}/"
f"{t['other_n']} = {pct(t['other_err'] / t['other_n'])}")
ta, tb = res["A"]["tot"], res["B"]["tot"]
assert tb["switch_err"] / tb["switch_n"] > ta["switch_err"] / ta["switch_n"]
print(" system B, amount entity error rate by surface form:")
by_style = defaultdict(lambda: [0, 0])
for u in gold7:
g, miss, _ = entity_counts(u["gold"], extract(u["hyp"]["B"], u["call"], u["locale"]),
"amount")
by_style[u["style"]][0] += miss
by_style[u["style"]][1] += g
for style in ("shorthand", "en-words", "es-words", "digits"):
e, n = by_style[style]
print(f" {style:<9} {e:>3}/{n:<4} = {pct(e / n)}")
# Step 4: interval width by set size
print("\n== 4. 95% interval width, system B amount entity error rate ==")
widths = {}
for name, s in (("400 utterances / 60 speakers", small), ("1,200 / 180", gold7)):
units = score(s, "B")["units"]["amount"]
p, lo, hi = ratio_ci(units, boot)
_, ulo, uhi = ratio_ci(units, boot, cluster=False)
widths[name] = (hi - lo, uhi - ulo)
print(f" {name:<29} {pct(p)} speaker-clustered [{pct(lo)}, {pct(hi)}] "
f"width {100 * (hi - lo):.1f} pts; utterance-level width {100 * (uhi - ulo):.1f} pts")
(w400, u400), (w1200, u1200) = widths.values()
assert w1200 < w400
# Step 5: the scorer policy moves WER, not entities
print("\n== 5. Same hypotheses, five scoring policies (1,200 set) ==")
print(f" {'policy':<22}{'WER A':>8}{'WER B':>8}{'amount B':>10}")
for pol in list(POLICIES) + ["nfc+fold+num"]:
row = []
for s in SYSTEMS:
e = n = 0
for u in gold7:
x = wer(u["ref"], u["hyp"][s], pol.replace("+num", ""), pol.endswith("+num"))
e, n = e + x["S"] + x["D"] + x["I"], n + x["N"]
row.append(e / n)
print(f" {pol:<22}{pct(row[0]):>8}{pct(row[1]):>8}{pct(res['B']['amount']):>10}")
print("\n T08 first turn (call date 2026-11-16)")
t08_call = dt.date(2026, 11, 16)
t08_ref = "sí, pero my paycheck comes el viernes, so after that I can pay, like the twenty-third"
t08 = {"STT as logged": "see pedro my paycheck comes el bee air ness so after that I can pay "
"like the twenty third",
"formatted style": "si pero my paycheck comes el viernes so after that I can pay "
"like the 23rd"}
for label, h in t08.items():
cells = " ".join(f"{p} {pct(wer(t08_ref, h, p)['wer'])}" for p in POLICIES)
print(f" {label:<16} {cells} nfc+fold+num {pct(wer(t08_ref, h, numerals=True)['wer'])}")
gold_t08, hyp_t08 = extract(t08_ref, t08_call, "es-US"), extract(t08["STT as logged"], t08_call)
for kind in ("date", "confirm"):
g, miss, _ = entity_counts(gold_t08, hyp_t08, kind)
print(f" STT as logged, {kind}: {g - miss}/{g} recovered")
t08_words = [(w, "es") for w in "sí, pero".split()] + [(w, "en") for w in "my paycheck comes".split()] \
+ [(w, "es") for w in "el viernes,".split()] \
+ [(w, "en") for w in "so after that I can pay, like the twenty-third".split()]
e_es, n_es = switch_word_counts(t08_words, t08["STT as logged"], "es")
e_en, n_en = switch_word_counts(t08_words, t08["STT as logged"], "en")
print(f" STT as logged, Spanish words {e_es}/{n_es} wrong, English words {e_en}/{n_en} wrong")
assert (e_es, n_es, e_en) == (3, 4, 0)
print("\n Locale fixture: Spanish references with accents, hypotheses in other styles")
fixture = [("sí, pagué mil quinientos dólares", "si pague 1,500 dolares"),
("quince dólares", "15 dolares"),
("el doce de marzo", "el 12 de marzo"),
("son doscientos cincuenta", "son doscientos quince")]
fx_wer = {}
for pol in list(POLICIES) + ["nfc+fold+num"]:
e = n = 0
for r, h in fixture:
x = wer(r, h, pol.replace("+num", ""), pol.endswith("+num"))
e, n = e + x["S"] + x["D"] + x["I"], n + x["N"]
fx_wer[pol] = e / n
g = miss = 0
for r, h in fixture:
a, b, _ = entity_counts(extract(r, call, "es-US"), extract(h, call, "es-US"), "amount")
g, miss = g + a, miss + b
print(" " + " ".join(f"{p} {pct(v)}" for p, v in fx_wer.items()))
quince = {p: wer("quince dólares", "15 dolares", p.replace("+num", ""), p.endswith("+num"))["wer"]
for p in ("whisper-basic", "nfc+fold+num")}
print(f" 'quince dólares' vs '15 dolares': whisper-basic {pct(quince['whisper-basic'])}, "
f"nfc+fold+num {pct(quince['nfc+fold+num'])}")
print(f" amount entity errors {miss}/{g} under every policy (entities are canonicalized first)")
assert quince["whisper-basic"] > 0 and quince["nfc+fold+num"] == 0
assert fx_wer["nfc+fold+num"] < fx_wer["whisper-basic"] and (miss, g) == (1, 3)
print("\nAll assertions passed.")
EDITIONS["global"] = (GLOBAL, run_global)
def main(argv: list[str] | None = None) -> None:
ap = argparse.ArgumentParser(description="Entity error rate for spoken amounts and dates.")
ap.add_argument("--edition", choices=["global", "india"], default="global",
help="which fictional lender's data to run (default: global)")
args = ap.parse_args(argv)
if args.edition not in EDITIONS:
ap.error("this copy has no India edition block; use the full file from the course repository")
world, run = EDITIONS[args.edition]
run(world)
if __name__ == "__main__":
main()
main looks the edition up in EDITIONS. The file you pasted from this page registers only the global edition, so --edition india stops with a usage error; the full file in the course repository adds the India block and runs it.
Run it:
python voice_entity_error_rate.py
Running it with Python 3.11 prints:
Steelyard Credit, Inc.
dataset steelyard-calls (synthetic, seed 20261121); scorer policy nfc+fold (steelyard-scorer-v1)
== 1. Canonicalization ==
fifteen hundred en-US -> 150000 cents
one thousand five hundred en-US -> 150000 cents
1,500 en-US -> 150000 cents
1500 en-US -> 150000 cents
$1,500.00 en-US -> 150000 cents
mil quinientos es-US -> 150000 cents
mil quinientos dolares es-US -> 150000 cents
1.500 es-ES -> 150000 cents
1.500 es-MX -> 150 cents (period decimal: one dollar fifty)
9 more amount forms match; two fifteen -> 21500
the twentieth en-US -> 2026-11-20 (call date 2026-11-14)
November twentieth en-US -> 2026-11-20 (call date 2026-11-14)
el veinte de noviembre es-US -> 2026-11-20 (call date 2026-11-14)
next Friday en-US -> 2026-11-20 (call date 2026-11-14)
el viernes es-US -> 2026-11-20 (call date 2026-11-14)
the fifth en-US -> 2026-12-05 (call date 2026-11-14)
12/03 en-US -> 2026-12-03 (call date 2026-11-14)
12/03 es-US -> 2027-03-12 (call date 2026-11-14)
== 2. T01: one substituted word ==
WER 1/12 = 0.083 (S=1 D=0 I=0)
amount entity errors 1/1: gold 25000 cents, STT 21500 cents
raw policy (no punctuation strip): WER 0.167
== 3. Bake-off on the 1,200-utterance set (180 speakers) ==
extract(reference) == generator gold on all 1,600 utterances
entity error rate = gold entities not recovered exactly / gold entities
sys WER CER amount, 95% CI date, 95% CI flips
A 15.6% 8.7% 1.0% [0.4%, 1.9%] 0.6% [0.0%, 1.5%] 5
B 10.8% 4.3% 6.1% [4.1%, 8.2%] 4.3% [2.4%, 6.6%] 6
A: amounts 765 gold, 8 missed, 6 hallucinated; 8 utterances with a wrong amount, 0 of them at WER under 10%, 1 under 20%
B: amounts 765 gold, 47 missed, 38 hallucinated; 47 utterances with a wrong amount, 1 of them at WER under 10%, 9 under 20%
by language group:
group utts WER A WER B amount A amount B
English 760 14.7% 8.5% 0.4% 2.4%
Spanish-dominant 229 14.9% 9.9% 0.8% 6.9%
Spanglish 211 18.5% 18.0% 2.9% 15.7%
Spanglish, errors on the Spanish words only (PIER-style) and on the English words:
A: Spanish words 120/658 = 18.2%; English words 177/1073 = 16.5%
B: Spanish words 143/658 = 21.7%; English words 99/1073 = 9.2%
system B, amount entity error rate by surface form:
shorthand 8/113 = 7.1%
en-words 5/194 = 2.6%
es-words 24/114 = 21.1%
digits 10/344 = 2.9%
== 4. 95% interval width, system B amount entity error rate ==
400 utterances / 60 speakers 5.4% speaker-clustered [2.0%, 9.1%] width 7.1 pts; utterance-level width 5.8 pts
1,200 / 180 6.1% speaker-clustered [4.1%, 8.2%] width 4.0 pts; utterance-level width 3.5 pts
== 5. Same hypotheses, five scoring policies (1,200 set) ==
policy WER A WER B amount B
raw 31.7% 27.0% 6.1%
whisper-basic 16.8% 11.2% 6.1%
nfc 17.0% 11.4% 6.1%
nfc+fold 15.6% 10.8% 6.1%
nfc+fold+num 13.1% 11.5% 6.1%
T08 first turn (call date 2026-11-16)
STT as logged raw 50.0% whisper-basic 29.4% nfc 29.4% nfc+fold 29.4% nfc+fold+num 31.2%
formatted style raw 25.0% whisper-basic 17.6% nfc 17.6% nfc+fold 11.8% nfc+fold+num 0.0%
STT as logged, date: 1/2 recovered
STT as logged, confirm: 0/1 recovered
STT as logged, Spanish words 3/4 wrong, English words 0/13 wrong
Locale fixture: Spanish references with accents, hypotheses in other styles
raw 64.3% whisper-basic 64.3% nfc 64.3% nfc+fold 35.7% nfc+fold+num 10.0%
'quince dólares' vs '15 dolares': whisper-basic 100.0%, nfc+fold+num 0.0%
amount entity errors 1/3 under every policy (entities are canonicalized first)
All assertions passed.
Step 8: Read the numbers
T01 is 8.3% WER and 100% amount error. One substitution in twelve words, and the only amount in the utterance is wrong: 25000 cents said, 21500 stored. Under the raw policy the same pair scores 16.7%, because the reference has a comma after "twentieth" and the STT output does not. The normalizer is part of the metric before any model is involved.
The locale tag decides what "1.500" means. All eight forms of $1,500.00 land on 150000 cents, the last one because it carries an es-ES tag. The same four characters under es-MX are 150 cents. A normalizer that strips every period, or one that never asks for the locale, gets one of the two wrong.
WER and amount accuracy rank the two systems in opposite orders. System B has the lower WER (10.8% against 15.6%) and about six times the amount entity error rate (6.1% against 1.0%); its date entity error rate is 4.3% against 0.6%. Pick by WER and you ship the system that writes more wrong promises. Do not read the separation of the two amount intervals as the test of a difference: for a release decision, bootstrap the paired difference by speaker on the same utterances, as in the bootstrap chapter.
Wrong amounts hide inside acceptable WER. Of B's 47 utterances with a wrong amount, 9 have WER under 20% and 1 is under 10%. Most utterances here are short, so a single substitution often costs more than 10 points of utterance WER; on longer turns like T01, more of them would sit under 10%.
Report the worst slice, not only the mean. B's amount entity error rate is 2.4% on English speakers, 6.9% on Spanish-dominant speakers and 15.7% on Spanglish speakers. By surface form it is 21.1% on Spanish number words, 7.1% on the shorthand ("two fifty"), 2.9% on digits and 2.6% on other English number words. The injector was written that way, so do not take the ordering as a finding about real systems. The habit is the point: slice by language group, surface form and entity type, and gate on the worst slice that matters to the campaign.
Overall WER hides the switched words. On Spanglish turns the two systems are within half a point of WER (18.5% for A, 18.0% for B). Split by language, B gets more English words right (9.2% wrong against 16.5%) and more Spanish words wrong (21.7% against 18.2%). That is the shape of the PIER result on a much smaller scale, and it is where T08's failure lives: a review of the first number alone does not see it.
Interval width depends on speakers. The 400-utterance set puts B at 5.4% with a speaker-clustered interval of [2.0%, 9.1%], 7.1 points wide; the 1,200-utterance set puts it at 6.1% with [4.1%, 8.2%], 4.0 points wide. A 400-utterance set alone could not tell you whether B's amount entity error rate is 2% or 9%. Here the clustered intervals are wider than the utterance-level ones (7.1 against 5.8 points, and 4.0 against 3.5), because each speaker contributes about 4 amounts (765 over 180 speakers) and one language group and one difficulty level are shared by all of them. The utterance-level interval is too narrow by exactly that much.
The scoring policy moves WER by 16 points and amount accuracy by zero. On the 1,200 set, B's WER is 27.0% raw, 11.2% under whisper-basic, 11.4% under nfc, 10.8% under nfc+fold and 11.5% with numerals, while the amount entity error rate stays at 6.1% in every row, because entities are canonicalized before they are compared. The policy also changes the gap between the systems: B's WER is 4.8 points below A's under nfc+fold and 1.6 points below with numerals, because numeral canonicalization forgives A's habit of writing spoken amounts as digits. Report which policy produced a number, every time.
T08's first turn shows the opposite case to T01. The logged STT output scores 29.4% WER under the three normalizing policies and 31.2% with numerals, against 50.0% raw. It recovers one of the two dates ("the twenty-third") and loses the other ("el viernes" came out as "el bee air ness"), and it loses the opening "si" [yes], heard as "see", which the confirmation check catches and WER does not single out. Three of its four Spanish words are wrong and none of its 13 English words is. The same turn written in a formatted style with no recognition error scores 25.0% raw, 17.6% under whisper-basic and nfc, 11.8% under nfc+fold and 0.0% with numerals. These are the numbers the normalization chapter reports.
The locale fixture shows writing style inflating WER. Three of its four pairs contain no recognition error, only dropped accents and digits for number words; the fourth hears "quince" [fifteen] for "cincuenta" [fifty]. WER is 64.3% raw, 35.7% under nfc+fold and 10.0% with numerals, where the only edit left is the real one. The pair "quince dolares" [fifteen dollars] against "15 dolares" scores 100% under whisper-basic and 0.0% with numerals. The wrong amount counts as 1 of 3 amounts under every policy.
What to do next
Run the scorer on your own pairs before you trust it: put each reference transcript, STT hypothesis, speaker id, call date and locale into the make_set shape, and check that extract(reference) matches your hand-labeled gold entities on every row. Every disagreement is either a canonicalizer gap or a labeling error, and both need fixing before the numbers mean anything.
Run it with --edition india for the India edition's data; the India chapter explains that output.
- Read the entity error rate chapter for the gate built on this scorer: amount entity error rate per gated segment, blocking when the upper bound exceeds the campaign threshold, alongside the C12 read-back check.
- Read the normalization chapter before you change a policy or the accent fold; any scorer change re-baselines every number reported under the old one.
- Push your own audio through the phone path first with the telephony degradation pipeline, then score the STT output here.
- Check the other direction, Hazel speaking the amount, with round-trip TTS numbers and dates, which canonicalizes round-trip transcripts the same way.