On 2026-11-11 Hazel's LLM wrote "Su proximo pago de $680.00 vence el 12/03." [Your next payment of $680.00 is due on 12/03.], meaning 3 December. The Spanish voice read the date day-first as "doce de marzo" [the twelfth of March], the borrower agreed, and the CRM stored 3 December (trace T06). The text was right for an English reader and the audio was wrong, so a transcript review passed the call.
This recipe builds the test that catches T06 before a borrower hears it. It turns CRM values into the words Hazel should say, sends each line through TTS and STT, parses the transcript back to a canonical value and exact-matches it against the CRM. How a numeric date is read depends on the locale the engine normalizes with, not on the text. In Unicode CLDR 48 the short date pattern is M/d/yy for English and d/M/y for es-US, so a date the English side of the stack formats month-first is read day-first by a Spanish voice 1. Engines document the same split: Microsoft's Azure SSML docs show 10-12-2016 read as October twelfth by default and December tenth with format="dmy" 2, and Cartesia's docs show 03/04/2026 read as March fourth when normalization is set to en-US, even with a British-accented voice 3. The concepts live in the TTS for agents chapter.
The 30 minutes: five to set up and run, fifteen to read the output against the decisions it gates, ten to plan where your own engines plug in.
The script and synthetic data for this recipe are also in the ai-evals-course repository on GitHub, in the folder voice-ai/labs/tts-round-trip-numerics/.
What you build
One standard-library script, tts-round-trip-numerics.py, with five parts:
- A verbalizer for English and Spanish. Dollar amounts in words ("four hundred twenty-five dollars", "mil quinientos dolares" [fifteen hundred dollars]); dates written out in each language's order ("December third", "tres de diciembre" [the third of December]); 10-digit phone numbers in 3-3-4 groups and ZIP codes digit by digit; the Customer Advocate email as "advocate at steelyard dot example" in English and "advocate arroba steelyard punto example" [advocate at steelyard dot example] in Spanish.
- 200 prompts from CRM templates. 80 amounts, 50 dates, 25 phone numbers, 15 ZIP codes, 15 Customer Advocate emails and 15 Steelyard addresses, in English and Spanish call languages (Spanish-dominant and Spanglish calls get the Spanish voice), from the seeded dataset "steelyard-calls (synthetic, seed 20261121)". Each prompt has a numeric form (what the LLM wrote before T06) and a written-out form (the rule since prompt collections-v17).
- A constructed round-trip fixture. A toy TTS normalizer and a toy 8 kHz STT, written in the script to reproduce six behaviors: an M/d date read day-first by the Spanish voice, a Spanish voice requested with the bare locale "es" reading the comma in "$1,500.00" as a decimal sign, a smaller model reading "$1,000,000.00" as "one thousand thousand dollars", the email's "@" and "." read by their symbol names, a bare five-digit ZIP read by the English voice as a cardinal number, and digits read by the Spanish voice in pairs, with a pair that starts with zero read as its second digit alone. No audio is synthesized, and the fixture is labeled constructed everywhere it prints.
- A parse-back scorer. It strips the template's carrier words, canonicalizes what is left (amount to integer cents, date to ISO, phone to ten digits, ZIP to five, email to a lowercased address, address to a house number, tokens and an exact ZIP) and exact-matches per entity type. Beside the round trip it reports the STT floor: the same STT on a human reading of the same written-out line.
- The acceptance gate. Spoken-entity exact match of at least 99.5% and zero wrong amounts on a 400-line amount suite, with the exact one-sided binomial upper bound on the error rate.
The fixture is a stand-in so the scorer and the gate have something to score. The numbers it produces describe the fixture, not any vendor's engine.
Step 1: set up a fresh environment
The script needs nothing outside the standard library, so the venv has no installs. Use a fresh one anyway, so nothing from your global site-packages leaks in:
python3.11 -m venv .venv-r5
.venv-r5/bin/python --version # Python 3.11.x
Step 2: save the script
Every recipe in this voice series starts with the same Steelyard constants block, so the fictional lender's products, campaigns, calling windows and Customer Advocate stay identical across recipes, and an --edition flag that defaults to global. The scorer's rules and the gate are the same in every edition; the lender, the languages, the templates, the fixture's failure modes and the seed change. Save this as tts-round-trip-numerics.py:
"""Round-trip TTS numbers and dates: verbalize, speak, transcribe, parse back, gate.
Dataset: steelyard-calls (synthetic, seed 20261121). Standard library only.
Python 3.11. No network. --edition india switches to the India edition's
fictional lender and data set (its block sits between the INDIA EDITION markers).
The round trip here runs on a CONSTRUCTED fixture: a toy TTS normalizer and a
toy 8 kHz STT written to reproduce documented or plausible behaviors (a
numeric date written month-first and read day-first by a Spanish voice, a
Spanish voice requested with the bare locale "es" reading a comma as a
decimal sign, a small model reading $1,000,000 as "one thousand thousand",
"@" and "." read by their symbol names, bare digits read as a cardinal).
No audio is synthesized. Swap `fixture_tts` and `fixture_stt` for your own
engines in `round_trip()` (see the adapter at the bottom).
Run:
python3.11 tts-round-trip-numerics.py # global edition
python3.11 tts-round-trip-numerics.py --edition india # India edition
"""
from __future__ import annotations
import argparse
import datetime as dt
import math
import random
import re
from dataclasses import dataclass
from typing import Callable
# ---- GLOBAL constants (copied from WORLD-GLOBAL WG1-WG7; do not edit here) ------
GLOBAL = {
"lender_note": "Synthetic data.",
"lender": "Steelyard Credit, Inc. (US consumer lender, head office Tucson AZ)",
"dataset": "steelyard-calls (synthetic, seed 20261121)",
"seed": 20261121,
"products": { # code: (ticket $ range, payment $ range)
"AUTO": ((8_000, 35_000), (245, 680)),
"PIL": ((1_000, 40_000), (95, 1_850)),
"POS": ((300, 6_000), (45, 520)),
"SBL": ((25_000, 250_000), (980, 7_200)),
"SDL": ((300, 3_000), (35, 140)),
},
"campaigns": {"AUTO-early-nov": "AUTO", "PIL-digital-nov": "PIL", "POS-short-nov": "POS",
"SBL-guar-nov": "SBL", "SDL-biwk-nov": "SDL"},
"windows": {"default": ("08:00", "21:00"), "SDL": ("09:00", "20:00")},
"languages": {"English": 64, "Spanish-dominant": 18, "Spanglish": 14, "other": 4},
"bot_languages": ("English", "Spanish-dominant", "Spanglish"), # other -> human
"customer_advocate": {"name": "Ms Dana Brooks", "email": "advocate@steelyard.example",
"phone": "(520) 555-0199",
"address": "1200 Example Avenue, Tucson, AZ 85701"},
"callback_number": "(520) 555-0100",
"money": "integer cents in tools and CRM; dollars in speech",
"dates": "ISO in tools; words in speech; en-US reads M/d, es-US reads d/M",
"versions": {"tts": "vendorB-v4@8k", "stt": "vendorA-2026-10", "gold": "gold-v7"},
"gold_v7_tts_lines": 500, "gold_v7_amount_lines": 400,
}
# -----------------------------------------------------------------------------------
CALL_DATE = dt.date(2026, 11, 20)
EN_UNITS = ("zero one two three four five six seven eight nine ten eleven twelve thirteen "
"fourteen fifteen sixteen seventeen eighteen nineteen").split()
EN_TENS = "_ _ twenty thirty forty fifty sixty seventy eighty ninety".split()
EN_NUM = {w: i for i, w in enumerate(EN_UNITS)}
EN_NUM.update({w: 10 * i for i, w in enumerate(EN_TENS) if i >= 2})
ORD_UNITS = ("_ first second third fourth fifth sixth seventh eighth ninth tenth eleventh "
"twelfth thirteenth fourteenth fifteenth sixteenth seventeenth eighteenth "
"nineteenth").split()
EN_ORD = {w: i for i, w in enumerate(ORD_UNITS) if i >= 1}
EN_ORD.update({"twentieth": 20, "thirtieth": 30})
MONTHS = ("january february march april may june july august september october november "
"december").split()
# Spanish number words, written without accents (ASCII), as the STT writes them here.
ES_UNITS = ("cero uno dos tres cuatro cinco seis siete ocho nueve diez once doce trece "
"catorce quince dieciseis diecisiete dieciocho diecinueve veinte veintiuno "
"veintidos veintitres veinticuatro veinticinco veintiseis veintisiete "
"veintiocho veintinueve").split()
ES_TENS = "_ _ veinte treinta cuarenta cincuenta sesenta setenta ochenta noventa".split()
ES_HUNDREDS = ("_ ciento doscientos trescientos cuatrocientos quinientos seiscientos "
"setecientos ochocientos novecientos").split()
ES_NUM = {w: i for i, w in enumerate(ES_UNITS)}
ES_NUM.update({w: 10 * i for i, w in enumerate(ES_TENS) if i >= 3})
ES_NUM.update({w: 100 * i for i, w in enumerate(ES_HUNDREDS) if i >= 1})
ES_NUM.update({"cien": 100, "un": 1, "veintiun": 21})
ES_MONTHS = ("enero febrero marzo abril mayo junio julio agosto septiembre octubre "
"noviembre diciembre").split()
WORD_NUM = {**EN_NUM, **ES_NUM}
MULT = {"hundred": 100, "thousand": 1_000, "mil": 1_000}
FILLER = {"dollars", "dollar", "dolares", "dolar", "and", "y"}
STATES = {"AZ": "Arizona", "CA": "California", "TX": "Texas", "FL": "Florida",
"IL": "Illinois", "MA": "Massachusetts"}
# ---- the US verbalizer: English and Spanish -----------------------------------------
def en_below_100(n: int) -> str:
if n < 20:
return EN_UNITS[n]
t, u = divmod(n, 10)
return EN_TENS[t] + ("" if u == 0 else " " + EN_UNITS[u])
def en_say_int(n: int) -> str:
"""Integer in Western grouping: thousand, hundred."""
def below_1000(k: int) -> str:
h, r = divmod(k, 100)
return " ".join(([f"{EN_UNITS[h]} hundred"] if h else []) + ([en_below_100(r)] if r else []))
th, r = divmod(n, 1_000)
parts = ([f"{below_1000(th)} thousand"] if th else []) + ([below_1000(r)] if r else [])
return " ".join(parts) or "zero"
def es_below_100(n: int) -> str:
if n < 30:
return ES_UNITS[n]
t, u = divmod(n, 10)
return ES_TENS[t] + ("" if u == 0 else " y " + ES_UNITS[u])
def es_say_int(n: int) -> str:
"""Spanish cardinal: 'mil quinientos', 'ciento veinte mil', 'veintiun mil'."""
def below_1000(k: int) -> str:
h, r = divmod(k, 100)
if h == 1 and r == 0:
return "cien"
return " ".join(([ES_HUNDREDS[h]] if h else []) + ([es_below_100(r)] if r else []))
th, r = divmod(n, 1_000)
parts = []
if th:
head = below_1000(th)
head = head[:-1] if head.endswith("uno") else head # uno -> un before mil
parts.append("mil" if th == 1 else f"{head} mil")
if r:
parts.append(below_1000(r))
return " ".join(parts) or "cero"
def say_amount(cents: int, lang: str) -> str:
dollars = cents // 100
return f"{es_say_int(dollars)} dolares" if lang == "es" else f"{en_say_int(dollars)} dollars"
def en_ordinal(n: int) -> str:
if n in EN_ORD.values() and n < 20 or n in (20, 30):
return {v: k for k, v in EN_ORD.items()}[n]
t, u = divmod(n, 10)
return f"{EN_TENS[t]} {ORD_UNITS[u]}"
def say_date(d: dt.date, lang: str) -> str:
if lang == "es": # day first: "tres de diciembre" [December third]
return f"{'primero' if d.day == 1 else es_say_int(d.day)} de {ES_MONTHS[d.month - 1]}"
return f"{MONTHS[d.month - 1].capitalize()} {en_ordinal(d.day)}"
def say_digits(digits: str, lang: str) -> str:
"""ZIP codes and other IDs digit by digit."""
words = ES_UNITS if lang == "es" else EN_UNITS
return " ".join(words[int(c)] for c in digits)
def say_phone(digits: str, lang: str) -> str:
"""10-digit NANP number in 3-3-4 groups, each digit spoken (the course's choice
for Spanish too: how a Spanish voice groups digits is undocumented, so measure it)."""
return ", ".join(say_digits(g, lang) for g in (digits[:3], digits[3:6], digits[6:]))
def say_house(n: int, lang: str) -> str:
if lang == "en" and n % 100 == 0 and 1_000 <= n < 10_000:
return f"{en_below_100(n // 100)} hundred" # "twelve hundred"
return es_say_int(n) if lang == "es" else en_say_int(n)
def say_email(addr: str, lang: str) -> str:
local, domain = addr.split("@")
at, dot = ("arroba", "punto") if lang == "es" else ("at", "dot")
return f"{local} {at} {f' {dot} '.join(domain.split('.'))}"
def usd(cents: int) -> str:
"""$1,500.00 style (what the LLM wrote before the words-for-numbers rule)."""
return f"${cents // 100:,}.{cents % 100:02d}"
# ---- the constructed fixture: a toy TTS normalizer and a toy 8 kHz STT ------------
def es_pairs(digits: str) -> str:
"""Toy Spanish reader for bare digits: pairs, and a pair that starts with zero
is read as its second digit alone (constructed)."""
out = []
for i in range(0, len(digits), 2):
p = digits[i:i + 2]
out.append(ES_UNITS[int(p[1])] if len(p) == 2 and p[0] == "0" else es_below_100(int(p)))
return ", ".join(out)
def fixture_tts(text: str, lang: str) -> str:
"""What a default normalizer says for numeric input (constructed). The English
voice uses en-US; the Spanish voice was requested with the bare locale "es",
so it reads numbers with Spain's conventions (comma decimal, day-first dates)."""
def email(m: re.Match) -> str:
return m.group(1) + " commercial at " + m.group(2).replace(".", " full stop ")
def address(m: re.Match) -> str:
house, street, state, zip_ = m.groups()
zip_words = es_pairs(zip_) if lang == "es" else say_digits(zip_, "en")
return f"{say_house(int(house), lang)} {street}, {STATES[state]}, {zip_words}"
def amount(m: re.Match) -> str:
raw = m.group(1)
if lang == "es" and "," in raw: # comma read as a decimal sign
whole, dec = raw.split(",", 1)
dec = dec.replace(",", "")
zeros = len(dec) - len(dec.lstrip("0"))
rest = [es_say_int(int(dec))] if dec.strip("0") else []
return f"{es_say_int(int(whole))} coma {' '.join(['cero'] * zeros + rest)} dolares"
return say_amount(int(raw.replace(",", "")) * 100, lang)
def date(m: re.Match) -> str:
a, b = int(m.group(1)), int(m.group(2)) # written M/d by the English stack
if lang == "en":
return f"{MONTHS[a - 1].capitalize()} {en_ordinal(b)}"
if 1 <= b <= 12: # es reads d/M: day a, month b
return f"{'primero' if a == 1 else es_say_int(a)} de {ES_MONTHS[b - 1]}"
return f"{es_say_int(a)} barra {es_say_int(b)}"
def phone(m: re.Match) -> str:
digits = "".join(m.groups())
return es_pairs(digits) if lang == "es" else say_phone(digits, "en")
def bare_zip(m: re.Match) -> str: # no say-as tag: en reads a cardinal
return es_pairs(m.group(0)) if lang == "es" else en_say_int(int(m.group(0)))
text = re.sub(r"([\w.]+)@([\w.]+\w)", email, text)
text = re.sub(r"\b(\d{3,5}) ([A-Z][\w ]+, [A-Z][\w ]+), ([A-Z]{2}) (\d{5})\b", address, text)
text = re.sub(r"\$([\d,]+\d)(?:\.\d\d)?", amount, text) # the toy drops ".00"
text = re.sub(r"\b(\d\d)/(\d\d)\b", date, text)
text = re.sub(r"\((\d{3})\) (\d{3})-(\d{4})", phone, text)
return re.sub(r"\b\d{5}\b", bare_zip, text)
def fixture_small_model(text: str) -> str:
"""A smaller TTS model's reading of a million (constructed after the documented
'one thousand thousand dollars' example): every comma group becomes 'thousand'."""
def amount(m: re.Match) -> str:
groups = m.group(1).split(",")
return " ".join([en_say_int(int(groups[0]))] + ["thousand"] * (len(groups) - 1)) + " dollars"
return re.sub(r"\$([\d,]+\d)(?:\.\d\d)?", amount, text)
def fixture_stt(spoken: str) -> str:
"""Toy STT on 8 kHz mu-law audio (constructed): lowercase, no punctuation."""
return " ".join(re.sub(r"[^\w\s@]", " ", spoken.lower().replace("-", " ")).split())
def phone_path(audio):
"""Plug point: resample to 8 kHz and G.711 mu-law encode before STT.
The fixture passes text, so this is the identity here."""
return audio
def round_trip(tts_input: str, lang: str, tts=fixture_tts, stt=fixture_stt) -> str:
return stt(phone_path(tts(tts_input, lang)))
HUMAN_CONFUSIONS = {"fifteen": "fifty", "fifty": "fifteen", "fourteen": "forty",
"thirteen": "thirty", "doce": "dos", "trece": "tres"}
def human_reference_stt(line: str, rng: random.Random) -> str:
"""Toy STT on a consented volunteer reading the written-out line (constructed):
the same STT, plus T01-style confusions on 6% of eligible lines."""
toks = fixture_stt(line).split()
hits = [i for i, t in enumerate(toks) if t in HUMAN_CONFUSIONS]
if hits and rng.random() < 0.06:
i = rng.choice(hits)
toks[i] = HUMAN_CONFUSIONS[toks[i]]
return " ".join(toks)
# ---- the parse-back scorer --------------------------------------------------------
def parse_number(toks: list[str]) -> float | None:
"""English or Spanish number words, or en-US digits, -> a value. None if unsure."""
toks = [t for t in toks if t not in FILLER]
if not toks:
return None
if len(toks) == 1 and re.fullmatch(r"\$?\d[\d,]*(\.\d+)?", toks[0]):
return float(toks[0].lstrip("$").replace(",", "")) # en-US separators
if len(toks) >= 2 and all(WORD_NUM.get(t, 99) <= 9 for t in toks):
return float("".join(str(WORD_NUM[t]) for t in toks)) # digit run
if "coma" in toks: # decimal comma
k = toks.index("coma")
whole, tail, zeros = parse_number(toks[:k]), toks[k + 1:], 0
while zeros < len(tail) and tail[zeros] == "cero":
zeros += 1
rest = parse_number(tail[zeros:]) if tail[zeros:] else None
dec = "0" * zeros + (str(int(rest)) if rest else "")
return float(f"{int(whole)}.{dec}") if whole is not None and dec else None
total, cur, last_mult = 0.0, 0.0, False
for t in toks:
if t in WORD_NUM:
cur += WORD_NUM[t]
last_mult = False
elif t in MULT:
if last_mult and cur == 0:
return None # "thousand thousand"
if MULT[t] == 100:
cur = (cur or 1) * 100
else:
total += (cur or 1) * MULT[t]
cur = 0.0
last_mult = True
else:
return None
return (total + cur) or None
def canon_amount(text: str) -> int | None:
"""Spoken or written amount -> integer cents. None if no amount is found."""
v = parse_number(text.lower().split())
return round(v * 100) if v else None
def digits_from_words(toks: list[str]) -> str:
out, i = [], 0
while i < len(toks):
t = toks[i]
nxt = toks[i + 1] if i + 1 < len(toks) else ""
if t in EN_TENS[2:] and EN_NUM.get(nxt, 99) < 10: # "twenty five"
out.append(str(EN_NUM[t] + EN_NUM[nxt]))
i += 2
continue
if t in ES_TENS[3:] and nxt == "y" and i + 2 < len(toks) and \
ES_NUM.get(toks[i + 2], 99) < 10: # "cincuenta y dos"
out.append(str(ES_NUM[t] + ES_NUM[toks[i + 2]]))
i += 3
continue
if WORD_NUM.get(t, 100) < 100:
out.append(str(WORD_NUM[t]))
elif t.isdigit():
out.append(t)
else:
return ""
i += 1
return "".join(out)
def canon_digits(text: str, n: int) -> str | None:
d = digits_from_words(text.lower().split())
return d if re.fullmatch(rf"\d{{{n}}}", d) else None
def word_day(toks: list[str]) -> int | None:
if toks and all(t in EN_ORD or t in WORD_NUM or t == "primero" for t in toks):
return sum(EN_ORD.get(t, WORD_NUM.get(t, 1)) for t in toks)
if len(toks) == 1 and re.fullmatch(r"\d{1,2}(st|nd|rd|th)?", toks[0]):
return int(re.match(r"\d+", toks[0]).group())
return None
def resolve(day: int, month: int) -> dt.date | None:
try:
d = dt.date(CALL_DATE.year, month, day)
except ValueError:
return None
return d if d >= CALL_DATE else d.replace(year=d.year + 1)
def canon_date(text: str) -> dt.date | None:
toks = [t for t in text.lower().split() if t not in ("the", "of", "on", "el", "de", "y")]
for sep in ("slash", "barra"): # explicit mdy policy: the stack writes M/d
if sep in toks:
k = toks.index(sep)
month, day = word_day(toks[:k]), word_day(toks[k + 1:])
return resolve(day, month) if day and month else None
months = [i for i, t in enumerate(toks) if t in MONTHS or t in ES_MONTHS]
if len(months) != 1:
return None
k = months[0]
month = (MONTHS.index(toks[k]) if toks[k] in MONTHS else ES_MONTHS.index(toks[k])) + 1
day = word_day(toks[:k] + toks[k + 1:])
return resolve(day, month) if day else None
def canon_email(text: str) -> str | None:
toks = text.lower().split()
toks = ["@" if t in ("at", "arroba") else "." if t in ("dot", "punto") else t for t in toks]
s = "".join(toks)
return s if re.fullmatch(r"[a-z0-9._]+@[a-z0-9]+(\.[a-z0-9]+)+", s) else None
def canon_address(text: str) -> tuple | None:
"""(house number, street and city tokens..., ZIP); the ZIP must match exactly."""
toks = text.lower().split()
k = 0
while k < len(toks) and (toks[k] in WORD_NUM or toks[k] in MULT):
k += 1
j = len(toks)
while j > k and (toks[j - 1] in WORD_NUM or toks[j - 1] == "y"):
j -= 1
house = parse_number(toks[:k]) if k else None
zip_ = canon_digits(" ".join(toks[j:]), 5)
return (int(house), *toks[k:j], zip_) if house and zip_ else None
CANON = {"amount": canon_amount, "date": canon_date, "phone": lambda s: canon_digits(s, 10),
"zip": lambda s: canon_digits(s, 5), "email": canon_email, "address": canon_address}
# ---- prompts from CRM templates (each Spanish template has its English gloss) -------
TEMPLATES = {
("amount", "en"): ["Your past-due amount is {}.", "Your payment is {}.",
"To pay off the loan in full you need to pay {}."],
("amount", "es"): ["Su saldo vencido es de {}.", # Your past-due balance is {}.
"Su pago es de {}.", # Your payment is {}.
"Para liquidar el prestamo necesita pagar {}."], # To pay off the loan...
("date", "en"): ["Your next payment is due on {}."],
("date", "es"): ["Su proximo pago vence el {}."], # Your next payment is due on {}.
("phone", "en"): ["Is {} still the best number to reach you?"],
("phone", "es"): ["Sigue siendo {} el mejor numero para llamarle?"], # Is {} still the best...
("zip", "en"): ["Thank you, ZIP code {} matches our records."],
("zip", "es"): ["Gracias, el codigo postal {} coincide con nuestros registros."], # same
("email", "en"): ["Our complaints email is {}."],
("email", "es"): ["Nuestro correo de quejas es {}."], # Our complaints email is {}.
("address", "en"): ["Our address is {}."],
("address", "es"): ["Nuestra direccion es {}."], # Our address is {}.
}
AREA_CODES = ("520", "602", "213", "415", "713", "512", "305", "312", "617")
ZIP_PREFIXES = ("857", "850", "900", "941", "770", "787", "331", "606", "021")
def make_amount(rng: random.Random, kind: int) -> int:
code = rng.choice(list(GLOBAL["products"]))
(t_lo, t_hi), (p_lo, p_hi) = GLOBAL["products"][code]
if kind == 0: # past due: 1 or 2 payments
d = rng.randint(p_lo, p_hi) * rng.choice((1, 2))
elif kind == 1: # payment
d = rng.randint(p_lo, p_hi)
else: # payoff, rounded to $500
d = max(t_lo, rng.randint(t_lo, t_hi) // 500 * 500)
return d * 100
def make_prompt(etype: str, rng: random.Random) -> dict:
group = rng.choices(GLOBAL["bot_languages"], weights=(64, 18, 14))[0]
lang = "en" if group == "English" else "es" # Spanish-dominant and Spanglish: es voice
k = rng.randrange(len(TEMPLATES[(etype, lang)]))
tpl = TEMPLATES[(etype, lang)][k]
adv = GLOBAL["customer_advocate"]
if etype == "amount":
gold = make_amount(rng, k)
numeric, words = usd(gold), say_amount(gold, lang)
elif etype == "date":
gold = CALL_DATE + dt.timedelta(days=rng.randint(1, 72))
numeric, words = gold.strftime("%m/%d"), say_date(gold, lang)
elif etype == "phone":
area, line = rng.choice(AREA_CODES), f"01{rng.randint(0, 99):02d}" # 555-01XX
gold = f"{area}555{line}"
numeric, words = f"({area}) 555-{line}", say_phone(gold, lang)
elif etype == "zip":
gold = rng.choice(ZIP_PREFIXES) + f"{rng.randint(0, 99):02d}"
numeric, words = gold, say_digits(gold, lang)
elif etype == "email":
gold = adv["email"]
numeric, words = gold, say_email(gold, lang)
else:
street, zip_ = adv["address"].rsplit(" ", 1) # "1200 Example ... AZ", "85701"
house, rest = street.split(" ", 1)
rest = re.sub(r", ([A-Z]{2})$", lambda m: ", " + STATES[m.group(1)], rest)
gold = (int(house), *fixture_stt(rest).split(), zip_)
numeric = adv["address"]
words = f"{say_house(int(house), lang)} {rest}, {say_digits(zip_, lang)}"
return {"type": etype, "lang": lang, "tpl": tpl, "gold": gold,
"numeric": tpl.format(numeric), "words": tpl.format(words)}
def extract(transcript: str, tpl: str) -> str | None:
"""Strip the template's carrier words; what is left is the spoken entity."""
pre, suf = (fixture_stt(s).split() for s in tpl.split("{}"))
toks = transcript.split()
if toks[:len(pre)] != pre or (suf and toks[len(toks) - len(suf):] != suf):
return None
return " ".join(toks[len(pre):len(toks) - len(suf)])
def score(p: dict, transcript: str) -> tuple[bool, object]:
span = extract(transcript, p["tpl"])
got = CANON[p["type"]](span) if span is not None else None
return got == p["gold"], got
def global_intro() -> None:
print(GLOBAL["lender_note"])
print(f"dataset: {GLOBAL['dataset']}; call date {CALL_DATE}; fixture: constructed\n")
# sanity checks on the canonicalizer (WORLD-GLOBAL WG7 forms 1-7; form 8 is R1's)
for form in ["fifteen hundred", "one thousand five hundred", "1,500", "1500",
"$1,500.00", "mil quinientos", "mil quinientos dolares"]:
assert canon_amount(form) == 150000, form
assert canon_amount("two hundred fifty dollars") == canon_amount("doscientos cincuenta") == 25000
assert canon_amount("ciento veinte mil") == canon_amount("one hundred twenty thousand") == 12000000
t06 = round_trip("Su proximo pago de $680.00 vence el 12/03.", "es") # trace T06
assert t06 == "su proximo pago de seiscientos ochenta dolares vence el doce de marzo", t06
heard = t06.split(" vence el ")[1]
assert canon_date(heard) == dt.date(2027, 3, 12)
bare_es = round_trip("$1,500.00", "es")
assert bare_es == "uno coma quinientos dolares" and canon_amount(bare_es) == 150, bare_es
small = fixture_stt(fixture_small_model("$1,000,000.00"))
assert small == "one thousand thousand dollars" and canon_amount(small) is None, small
print("WG7 surface forms 1-7 -> 150000 cents: ok")
print(f"T06 line '12/03' (gold 2026-12-03) heard as: '{heard}' -> 2027-03-12")
print(f"bare 'es' voice, '$1,500.00' heard as: '{bare_es}' -> 150 cents (gold 150000)")
print(f"small model, '$1,000,000.00' heard as: '{small}' -> None (rejected)\n")
def global_date_rows(prompts: list[dict], speak: Callable, score_fn: Callable) -> list[str]:
dates = [p for p in prompts if p["type"] == "date"]
rows = []
for label, sel in (("English", lambda p: p["lang"] == "en"),
("Spanish, day 1-12", lambda p: p["lang"] == "es" and p["gold"].day <= 12),
("Spanish, day 13-31", lambda p: p["lang"] == "es" and p["gold"].day > 12)):
grp = [p for p in dates if sel(p)]
ok = sum(score_fn(p, speak(p, p["numeric"]))[0] for p in grp)
same = sum(p["gold"].day == p["gold"].month for p in grp)
rows.append(f"date rows, numeric input, {label:<18}: {pct(ok, len(grp))}"
+ (f" (day == month in {same})" if "1-12" in label else ""))
return rows
def global_check(tot: list[int], results: dict) -> None:
"""Pin this edition's own printed numbers, so a changed fixture fails loudly."""
assert tot == [200, 140, 200, 199], tot
assert results["numeric"][0] == 69, results["numeric"]
# ---- the acceptance gate (the same in every edition) --------------------------------
def binom_cdf(k: int, n: int, p: float) -> float:
if p <= 0:
return 1.0
if p >= 1:
return 0.0 if k < n else 1.0
return sum(math.exp(math.lgamma(n + 1) - math.lgamma(i + 1) - math.lgamma(n - i + 1)
+ i * math.log(p) + (n - i) * math.log1p(-p)) for i in range(k + 1))
def upper_bound(k: int, n: int, conf: float = 0.95) -> float:
"""Exact one-sided (Clopper-Pearson) upper bound on the error rate."""
if k == 0:
return 1 - (1 - conf) ** (1 / n)
lo, hi = k / n, 1.0
for _ in range(100):
mid = (lo + hi) / 2
lo, hi = (mid, hi) if binom_cdf(k, n, mid) > 1 - conf else (lo, mid)
return hi
def gate(k: int, n: int, min_match: float = 0.995) -> str:
return "PASS" if k == 0 and (n - k) / n >= min_match else "FAIL"
def show(x: object) -> str:
return x.isoformat() if isinstance(x, dt.date) else repr(x)
def pct(k: int, n: int) -> str:
return f"{k:>3}/{n:<3} {100 * k / n:6.1f}%"
@dataclass
class Edition:
world: dict
intro: Callable[[], None]
counts: dict
make_prompt: Callable[[str, random.Random], dict]
speak: Callable[[dict, str], str] # (prompt, TTS input) -> heard transcript
floor: Callable[[dict, random.Random], str] # (prompt, rng) -> STT of a human reading
score: Callable[[dict, str], tuple]
date_rows: Callable[[list, Callable, Callable], list]
check: Callable[[list, dict], None] = lambda tot, results: None
EDITIONS = {"global": Edition(
world=GLOBAL, intro=global_intro,
counts={"amount": 80, "date": 50, "phone": 25, "zip": 15, "email": 15, "address": 15},
make_prompt=make_prompt, speak=lambda p, text: round_trip(text, p["lang"]),
floor=lambda p, rng: human_reference_stt(p["words"], rng), score=score,
date_rows=global_date_rows, check=global_check)}
# The India edition's block (its constants, verbalizer, fixture and scorer) goes here
# in the full file; it adds EDITIONS["india"]. Without it, --edition accepts only "global".
# ---- run ------------------------------------------------------------------------
def run(ed: Edition) -> None:
ed.intro()
world = ed.world
rng = random.Random(world["seed"])
prompts = [ed.make_prompt(t, rng) for t, n in ed.counts.items() for _ in range(n)]
hrng = random.Random(world["seed"] + 1)
rows, fails = {}, {}
for p in prompts:
ok_num, got = ed.score(p, ed.speak(p, p["numeric"]))
ok_words, _ = ed.score(p, ed.speak(p, p["words"]))
ok_floor, _ = ed.score(p, ed.floor(p, hrng))
r = rows.setdefault(p["type"], [0, 0, 0, 0])
r[0] += 1
r[1] += ok_num
r[2] += ok_words
r[3] += ok_floor
if not ok_num and p["type"] not in fails:
fails[p["type"]] = (p["numeric"], ed.speak(p, p["numeric"]), got, p["gold"])
print(f"{len(prompts)} prompts from CRM templates; exact match after the round trip")
print(f"{'entity':<8} {'n':>4} {'numeric input':>15} {'written-out input':>17} "
f"{'STT floor (human)':>17}")
for t, (n, a, b, c) in rows.items():
print(f"{t:<8} {n:>4} {pct(a, n):>15} {pct(b, n):>17} {pct(c, n):>17}")
tot = [sum(r[i] for r in rows.values()) for i in range(4)]
print(f"{'all':<8} {tot[0]:>4} {pct(tot[1], tot[0]):>15} {pct(tot[2], tot[0]):>17} "
f"{pct(tot[3], tot[0]):>17}")
for line in ed.date_rows(prompts, ed.speak, ed.score):
print(line)
print("\nfirst failure per entity type, numeric input:")
for t, (src, heard, got, gold) in fails.items():
print(f" {t:<7} TTS input: {src}")
print(f" heard: {heard}")
print(f" parsed {show(got)}, expected {show(gold)}")
# gate on a 400-line amount suite (gold-v7 has 400 amount lines)
arng = random.Random(world["seed"] + 2)
suite = [ed.make_prompt("amount", arng) for _ in range(world["gold_v7_amount_lines"])]
print(f"\nacceptance gate: {len(suite)} amount lines, rule = exact match >= 99.5% and "
"zero wrong amounts")
print(f"{'input':<12} {'wrong':>7} {'exact match':>11} {'95% upper bound (one-sided)':>27} gate")
results = {}
for style in ("words", "numeric"):
k = sum(not ed.score(p, ed.speak(p, p[style]))[0] for p in suite)
n = len(suite)
ub = upper_bound(k, n)
results[style] = (k, n, ub)
name = "written-out" if style == "words" else "numeric"
print(f"{name:<12} {k:>3}/{n:<3} {100 * (n - k) / n:10.2f}% {ub:>27.4f} {gate(k, n)}")
k, n, ub = results["words"]
print(f"0 of {n}: exact bound {ub:.5f} ({100 * ub:.3f}%); rule of three 3/{n} = {3 / n:.4f}")
# [C] numbers shared by every edition (W17 / WG17): 0 of 400 -> 0.746%, about 0.75%
assert k == 0 and n == 400
assert round(ub, 4) == 0.0075 and round(100 * ub, 3) == 0.746
assert abs(ub - 0.0075) < 0.0002 and 3 / n == 0.0075
assert gate(*results["words"][:2]) == "PASS" and gate(*results["numeric"][:2]) == "FAIL"
ed.check(tot, results)
print("all assertions passed")
def main() -> None:
ap = argparse.ArgumentParser(description="Round-trip TTS numbers and dates.")
ap.add_argument("--edition", choices=sorted(EDITIONS), default="global",
help="which fictional world to run (default: global)")
run(EDITIONS[ap.parse_args().edition])
# ---- adapter: plug in your own engines -----------------------------------------
class YourEngines:
"""Replace the fixture with real engines. Keep the TTS version pinned and
rerun the suite on every TTS or STT update.
def tts(text: str, lang: str) -> bytes: # request 8 kHz mu-law, as production
... # does; set the full locale (es-US)
def stt(audio: bytes) -> str: # the production STT, same settings
...
heard = round_trip(line, lang, tts=tts, stt=stt)
"""
if __name__ == "__main__":
main()
Step 3: run it
.venv-r5/bin/python tts-round-trip-numerics.py
Running it with Python 3.11 prints:
Synthetic data.
dataset: steelyard-calls (synthetic, seed 20261121); call date 2026-11-20; fixture: constructed
WG7 surface forms 1-7 -> 150000 cents: ok
T06 line '12/03' (gold 2026-12-03) heard as: 'doce de marzo' -> 2027-03-12
bare 'es' voice, '$1,500.00' heard as: 'uno coma quinientos dolares' -> 150 cents (gold 150000)
small model, '$1,000,000.00' heard as: 'one thousand thousand dollars' -> None (rejected)
200 prompts from CRM templates; exact match after the round trip
entity n numeric input written-out input STT floor (human)
amount 80 61/80 76.2% 80/80 100.0% 79/80 98.8%
date 50 44/50 88.0% 50/50 100.0% 50/50 100.0%
phone 25 16/25 64.0% 25/25 100.0% 25/25 100.0%
zip 15 4/15 26.7% 15/15 100.0% 15/15 100.0%
email 15 0/15 0.0% 15/15 100.0% 15/15 100.0%
address 15 15/15 100.0% 15/15 100.0% 15/15 100.0%
all 200 140/200 70.0% 200/200 100.0% 199/200 99.5%
date rows, numeric input, English : 36/36 100.0%
date rows, numeric input, Spanish, day 1-12 : 0/6 0.0% (day == month in 0)
date rows, numeric input, Spanish, day 13-31: 8/8 100.0%
first failure per entity type, numeric input:
amount TTS input: Para liquidar el prestamo necesita pagar $31,500.00.
heard: para liquidar el prestamo necesita pagar treinta y uno coma quinientos dolares
parsed 3150, expected 3150000
date TTS input: Su proximo pago vence el 01/12.
heard: su proximo pago vence el primero de diciembre
parsed 2026-12-01, expected 2027-01-12
phone TTS input: Sigue siendo (713) 555-0197 el mejor numero para llamarle?
heard: sigue siendo setenta y uno treinta y cinco cincuenta y cinco uno noventa y siete el mejor numero para llamarle
parsed None, expected '7135550197'
zip TTS input: Thank you, ZIP code 78763 matches our records.
heard: thank you zip code seventy eight thousand seven hundred sixty three matches our records
parsed None, expected '78763'
email TTS input: Nuestro correo de quejas es advocate@steelyard.example.
heard: nuestro correo de quejas es advocate commercial at steelyard full stop example
parsed None, expected 'advocate@steelyard.example'
acceptance gate: 400 amount lines, rule = exact match >= 99.5% and zero wrong amounts
input wrong exact match 95% upper bound (one-sided) gate
written-out 0/400 100.00% 0.0075 PASS
numeric 69/400 82.75% 0.2066 FAIL
0 of 400: exact bound 0.00746 (0.746%); rule of three 3/400 = 0.0075
all assertions passed
The script asserts on the gate numbers and on its own table totals before it prints "all assertions passed", so a changed parser or fixture that moves them fails loudly.
The three lines under the forms check are the fixture's documented behaviors, one call each. The T06 line comes back as "doce de marzo" [the twelfth of March], which resolves to 12 March 2027. The bare "es" line comes back as "uno coma quinientos dolares" [one point five hundred dollars], which canonicalizes to 150 cents instead of 150000: in CLDR 48, Spain uses a comma decimal and a period group while es-US uses a period decimal like English 1, and Cartesia documents that a bare "es" uses es-ES conventions 3. The small-model line copies an example from ElevenLabs' best-practices guide, where Eleven Multilingual v2 reads "$1,000,000" as "one million dollars" and the smaller Eleven Flash v2.5 reads it as "one thousand thousand dollars" 4. The parser rejects that phrase instead of guessing, so it counts as a miss.
Written-out input survives the round trip on all 200 prompts. Numeric input drops to 140 of 200 (70.0%), and each entity type fails for its own reason:
- Amounts, 61 of 80. Every miss is a Spanish line of $1,000 or more: 19 lines. English lines pass 51 of 51 and Spanish lines under $1,000 pass 10 of 10, because without a separator the bare "es" reading and the en-US reading agree. The first miss is "Para liquidar el prestamo necesita pagar $31,500.00." [To pay off the loan you need to pay $31,500.00.], heard as "treinta y uno coma quinientos dolares" [thirty-one point five hundred dollars] and parsed as $31.50. That is why a suite of AUTO, POS and SDL payments alone looks clean.
- Dates, 44 of 50. The split rows show the format effect. English lines pass 36 of 36: the en-US voice reads M/d the way the stack wrote it. Spanish lines with a day from 13 to 31 pass 8 of 8: no month 13 exists, so the toy reader falls back to "once barra veinticinco" [eleven slash twenty-five] and the scorer's explicit month-first policy recovers the date. Spanish lines with a day from 1 to 12 pass 0 of 6. "01/12", meaning 12 January, becomes "primero de diciembre" [the first of December], six weeks early. In production a wrong date in a valid month is worse than a garbled one, because nobody notices it. Setting the full es-US locale does not fix this row: es-US is day-first too in CLDR 48 1, which is how T06 happened on an es-US voice. Words in the call language fix it.
- Phone numbers, 16 of 25. English lines pass 16 of 16. All 9 Spanish lines fail: the toy reads digits in pairs, a pair that starts with zero loses its zero, and every 555-01XX number puts 01 in a pair. We found no vendor documentation that says how a Spanish voice groups the digits of a phone number, so treat the pairing as engine behavior to measure on your own voice, not as a rule.
- ZIP codes, 4 of 15. English lines fail 10 of 10: with no tag, the toy reads "78763" as "seventy-eight thousand seven hundred sixty-three". Amazon's Polly docs draw the same line for phone numbers: a formatted 202-555-1212 is recognized without a tag, while bare digits such as 2025551212 need one 5. Spanish lines pass 4 of 5; the miss is 77069, whose pair "06" is read as "seis" [six].
- Email, 0 of 15. "commercial at" and "full stop" are the Unicode names of the two symbols. No caller writes those down as an address. Benchmarks find the same weak spot: in EmergentTTS-Eval, gpt-4o-audio-preview with the Ballad voice and strong prompting won 88.84% of judge-scored comparisons against a gpt-4o-mini-tts baseline on emotional dialogue, but only 40.40% on complex pronunciation: emails, phone numbers, URLs, addresses and similar strings 6.
- Address, 15 of 15, by luck. The English voice reads a ZIP that follows a state code digit by digit, and 85701 pairs as 85-70-1 for the Spanish voice, with no pair that starts with zero. A Massachusetts address, whose ZIP starts with 0, would come back one digit short from the Spanish voice. One passing address line says nothing about the next.
Do not read the 70.0% as a property of numeric input in general. It is what this fixture does to this mix. On your engines the failing classes will differ, and that is the reason to run the table, not guess it.
Step 5: read the STT floor
The STT floor column runs the same STT on a constructed human reading of each written-out line, with T01-style confusions ("fifty" heard as "fifteen", "doce" [twelve] heard as "dos" [two]) on 6% of eligible lines. It comes out at 199 of 200, below the written-out round trip at 200 of 200. The one miss is a payoff line where "thirteen thousand dollars" came back as "thirty thousand dollars".
Read the two columns together. When the round trip sits at or above the floor, as here, the STT is the weaker link and a round-trip miss needs a listen before anyone blames the TTS. When the round trip falls below the floor, the TTS is the suspect: the STT handled a person saying the same words. Never gate on the best score over several STT engines or transcript variants. That rewards a vendor for your scorer's flexibility. Gate on one pinned STT, and report its floor beside the result.
Step 6: read the gate
The gate runs on a separate 400-line amount suite, the size of gold-v7's amount set. Written-out input gets 0 of 400 wrong. Zero errors is not a zero error rate: the exact one-sided 95% upper bound for 0 of 400 is 1 - 0.05^(1/400) = 0.00746, about 0.75%, close to the rule-of-three shortcut 3/400 = 0.0075 7. That is the strongest claim 400 clean lines support. To bound the rate below 0.1% you would need 2,995 clean lines, the smallest n with 0.999^n at or below 0.05.
Numeric input gets 69 of 400 wrong (82.75% exact match) and fails the gate. All 69 are Spanish lines of $1,000 or more, out of 134 Spanish lines in the suite. Its upper bound, 0.2066, is the Clopper-Pearson bound for k = 69, found by bisection on the binomial CDF in upper_bound(); for k = 0 the same bound has the closed form above.
The script checks parts 1 and 2 of the four-part gate in the TTS for agents chapter, both on the 400 amount lines. In production, part 1's 99.5% bar runs on the full 500-line gold-v7 suite, every entity type included. The other two need people and versions: pairwise preference not worse than the incumbent voice at 8 kHz, and a pinned TTS version with the suite rerun on every update. The version pin matters because normalization ships with the model: the ElevenLabs example above is two models from one vendor reading the same string two ways, and the same guide lists phone numbers, currencies, dates and times as common failure cases and recommends writing them out in words, for example by instructing the LLM that generates the text 4. Hazel's words-for-numbers rule is that instruction; this gate checks that it still holds after every TTS update.
Step 7: plug in your own TTS and STT
round_trip() takes tts and stt as arguments. Replace the fixture with two functions:
tts(text, lang) -> bytes: call your production TTS with the production voice, version and normalization settings, and ask for 8 kHz mu-law, as the production stream does. Pass the full locale (en-US or es-US), never a bare "es".
stt(audio) -> str: your production STT in the same mode the agent uses (English-Spanish with automatic language detection, if that is what production runs).
Then fill in phone_path(). Use the phone-path degradation recipe to band-limit, encode and add loss, so the STT hears what the carrier delivers, not the 24 kHz demo. The telephony chapter explains why the band and the transcoding matter more than mu-law itself.
Keep the parser strict when you switch. If your STT writes "$4,250" or "Dec 5", extend canon_amount() and canon_date() with those exact forms, tagged with the locale they belong to, and add a fixture line for each. Do not loosen the scorer until a failing line passes. The normalization chapter covers how to version those rules, including Spanish numerals and code-switched turns.
The full file in the course repository also carries the India edition's block, which the copy above leaves out. Run it with --edition india for the India edition's data (its own lender, languages, amount formats and seed); the India chapter explains that output. The gate lines for written-out input and the 0 of 400 bound are identical in both editions; the table, the failure examples and the numeric gate row differ because the world does.
What to do next
- Point the 200 prompts at your real CRM templates. Oversample Spanish amounts of $1,000 and more, numeric dates with a day from 1 to 12, phone numbers and ZIP codes with a zero, and every string Steelyard's policy makes Hazel say aloud: here, the Customer Advocate's email, phone and address.
- Run the written-out and numeric columns on every TTS candidate in a vendor bake-off, at 8 kHz and in each locale you ship, before anyone listens for naturalness.
- Score the STT side of the same entities with the entity error rate recipe; the entity error rate chapter explains why WER hides them.
- Put the gate in CI next to the pinned TTS version string, so a version bump without a suite run fails the build.