On 2026-11-11 Hazel's LLM wrote "Su proximo pago de $680.00 vence el 12/03." [Your next payment of $680.00 is due on 12/03.], meaning 3 December; the Spanish text-to-speech (TTS) voice read it as "doce de marzo" [the twelfth of March], the borrower agreed, and the CRM stored 3 December (trace T06). Every log line was right for a reader of en-US dates, and the borrower heard the wrong month. This chapter is about testing what the caller hears: spoken amounts, dates and digits first, naturalness second, and both on phone-line audio.
What happened on the call?
The English side of the stack formatted the due date month-first as "12/03", the es-US voice read it day-first, and the borrower heard 12 March while the CRM held 3 December. Prompt collections-v16 still allowed digits in TTS input.
Excerpt from trace T06 (P10, Javier Ortiz, a Spanish-dominant PIL borrower; gloss: lines are English meanings, not part of the trace):
t_ms ch event / text
agent LLM output (the English stack formats the date as en-US M/d): "Su proximo pago de $680.00 vence el 12/03."
gloss: "Your next payment of $680.00 is due on 12/03."
agent TTS (es-US voice; es-US reads short dates day-first) audio: "...vence el doce de marzo."
gloss: "...is due on the twelfth of March."
caller REF: "Esta bien, pago entonces."
gloss: "Okay, I'll pay then."
agent tool record_promise_to_pay(ptp_date=2026-12-03, ...) (CRM holds 3 Dec; borrower heard 12 March)
The transcript view shows nothing wrong: read the English way, "12/03" is 3 December, the tool call says 2026-12-03, and Javier sounds agreeable. The fault exists only in the audio. In Unicode CLDR 48 the short date pattern is M/d/yy for English but d/M/y for es-US, so a date formatted month-first and passed unchanged to a Spanish voice is read day-first 1.
Diagnosis: two locales met in one string, the voice's locale chose the reading, and no test compared the spoken date with the stored one.
Why does entity correctness beat naturalness for an agent?
An agent's speech carries commitments: an amount, a due date, a phone number. A wrong one creates a broken promise or a compliance failure; a slightly less natural voice costs a little goodwill. The two qualities do not travel together, so test entities first and rank naturalness only among voices that pass.
EmergentTTS-Eval shows the split in numbers. gpt-4o-audio-preview with the Ballad voice and strong prompting won 88.84% of judge-scored comparisons against a gpt-4o-mini-tts baseline on emotional dialogue, but only 40.40% on complex pronunciation: emails, phone numbers, URLs, addresses and similar strings 2. The system that led on what a demo shows off won fewer than half its comparisons on what a collections call depends on.
For Steelyard the ordering is a matter of policy, not taste. C12 (read-back before a promise is written), C13 (the amount Hazel states equals the get_dues result) and C06 (the Customer Advocate's name and phone on every right-party call) are all spoken entities, owned by the compliance chapter. (Steelyard collects its own debts in its own name, so the FDCPA largely does not bind it 3; it adopts the FDCPA and Regulation F conduct rules as policy, and state law, the TCPA and the CFPB's ban on unfair, deceptive or abusive practices still apply.) No MOS gain buys back one wrong amount on a recorded call. The voice playbook sets MOS targets for TTS in general; for an agent, MOS and preference scores decide between candidates only after the entity gate passes, and never compensate for a failure on it.
How do you build a TTS test set?
Turn the TTS input texts in production traces into templates, fill them with synthetic values from the CRM's real distributions, and oversample the cases where the engine must choose: amounts of a thousand dollars or more, numeric dates with a day from 1 to 12, digit strings, emails and addresses, in both languages.
Templates, not raw turns. The tts.text field in a trace is personal data under the trace rules in chapter 1, so real lines never go into a test set. Group production turns by template ("Your past-due amount is {amount}", "Su pago de {amount} vence el {date}" [Your payment of {amount} is due on {date}]), keep each template's frequency, and fill the slots with synthetic values from the per-product distributions.
Oversample by entity class, not by frequency. Most Steelyard payments are below a thousand dollars: AUTO runs $245 to $680 a month, POS $45 to $520, SDL $35 to $140 every two weeks. Below a thousand there is no separator to misread, so a frequency-weighted set mostly tests lines where nothing is at stake. Oversample:
- Currency. Amounts of a thousand and above (SBL payments of $980 to $7,200, payoff amounts, the guarantor's $120,000 loan), and the eight surface forms of $1,500.00 from the entity error rate chapter, including "1.500", fifteen hundred in Spain and one and a half in Mexico. "Two fifty" heard as "two fifteen" is T01 in reverse, on the speaking side.
- Both languages' number words. A Spanglish turn can put "cuatrocientos veinticinco dolares" [four hundred twenty-five dollars] or the English words in the same slot, so every amount template gets both.
- Dates. Spoken forms ("December third", "el tres de diciembre" [the third of December]) and numeric dates with the day from 1 to 12, rendered by both the en-US and the es-US voice.
- Digits. Loan tails ("four four one seven", "cuatro cuatro uno siete" [four four one seven]), the Customer Advocate number (520) 555-0199 and the callback number (520) 555-0100, 12-digit confirmation numbers in groups, and ZIP codes.
- Names. Names read by the other language's voice ("Sofia Ramirez" in an English turn), titles ("Mr. Mitchell") and the Customer Advocate, Dana Brooks.
- Email and address. "advocate@steelyard.example" and "1200 Example Avenue, Tucson, AZ 85701", which Hazel offers aloud on dispute, complaint and hardship paths (C06), the complex-pronunciation class where EmergentTTS-Eval's expressive system won only 40.40%.
- Required notices. Every string policy makes Hazel say aloud, such as the AI disclosure in the opener and the debt-collection notice; a notice garbled in the audio fails even when the text check passes.
Steelyard's gold-v7 holds 500 TTS lines covering every class above, 400 of them amounts; the 400 is sized for the gate below.
How do you score a spoken entity?
Send the line through the production TTS, the phone path and one pinned STT, canonicalize the transcript (amount to integer cents, date to ISO), and exact-match each entity against the CRM value. Beside every result, report the STT floor: the same STT on a human reading the same line.
The loop:
flowchart LR
A["CRM value<br/>25000 cents"] --> B["TTS input text<br/>two hundred fifty dollars"]
B --> C["TTS<br/>pinned version, voice, locale"]
C --> D["Phone path<br/>8 kHz mu-law, codec chain"]
D --> E["STT<br/>pinned, English-Spanish mode"]
E --> F["Canonicalize<br/>shared rules, locale-aware"]
F --> G{"Exact match<br/>with CRM value?"}
G -->|yes| H["pass"]
G -->|no| I["fail: listen to the clip"]
J["Human reading of<br/>the same line"] --> D
Four rules keep the score meaningful.
Report the floor. The round trip joins two systems: if the STT mishears "fifty" as "fifteen" on a person's voice, the TTS gets blamed for the listener. So every run also passes a consented human recording of the same line, in the same language, through the same phone path and STT. If the synthesized line scores as well as the human reading, the STT is the limiting link and each miss needs a listen; if it scores worse, the TTS is the suspect. Reuse the entity error rate chapter's canonicalization rules, locale tag on every amount included; do not fork them.
Exact match per entity, not CER per line. A line can be 95% right by characters and carry a wrong amount. Score each entity right or wrong and report by entity type and by locale; a pooled pass rate hides the class that fails.
Count severe misreadings on their own line. Neural text normalizers reach high overall accuracy and still make errors that give a listener the wrong message, and later work used finite-state grammars to block misreadings such as "3 cm" read as "three kilometers" 4 5. A factor-of-a-thousand amount error is one token in a token-accuracy score, so report wrong amounts, dates and digit strings as their own counts.
Never gate on the best of several readings. One tempting scorer generates several spoken-form variants of the reference and several transcription candidates, computes CER for every pair, and keeps the minimum. A caller hears one rendering once. Min-over-variants scoring helps diagnose a failure; as a gate, it rewards the vendor for your scorer's flexibility. Gate on one pinned STT and one canonical value.
STT normalization rules are owned by the normalization chapter. Here they matter as a source of false failures: if the canonicalizer cannot parse "mil quinientos dolares" [fifteen hundred dollars], the TTS looks broken when it is not.
Why do dates get read differently?
The engine's normalization locale or date-format setting decides how a numeric date is spoken, whatever the voice's accent, and the default follows the locale, not the writer's intent: an es-US voice reads 12/03 day-first. Amount separators work the same way, and a model update can change either reading.
Every row but the last comes from the cited documentation or locale data; the last is the incident.
| Engine or data, setting | Written | Heard | Basis |
|---|
| Azure, en-US, no format | 10-12-2016 | October twelfth | Docs 6 |
| Azure, format="dmy" | 10-12-2016 | December tenth | Docs 6 |
| Cartesia, locale en-GB, normalization en-US | 03/04/2026 | March fourth | Docs 7 |
| CLDR 48 short date for es-US (d/M/y) | 03/04/2026 | can be read as 3 April | Locale data 1 |
| Polly, interpret-as date, yyyymmdd | ????0922 | September 22nd | Docs 8 |
| ElevenLabs Multilingual v2 | $1,000,000 | one million dollars | Docs 9 |
| ElevenLabs Flash v2.5 | $1,000,000 | one thousand thousand dollars | Docs 9 |
| Google, say-as currency, language="en-US" | $42.01 | forty two dollars and one cent | Docs 10 |
| Steelyard vendorB, es-US voice, digits in input | 12/03 (meant 3 December) | doce de marzo [twelfth of March] | Incident T06 |
Locale defaults. The voice and the normalizer are separate settings. Azure supports its interpret-as values other than characters and spell-out for all Spanish locales, but its date, telephone and currency examples are English only, so Spanish read-outs have to be tested per locale 6. Polly says you need to indicate the format for interpret-as date, except for yyyymmdd 8. Cartesia's Spanish locales are es-ES, es-MX and es-US, and a bare "es" uses Spain's conventions 7. Send an explicit format and a full locale on every request, and test numeric dates whose day runs from 1 to 12, where the wrong reading is a valid date nobody notices.
Separators and amounts. In CLDR 48, es-US, es-MX and es-419 use a period decimal and a comma group like English, while Spain uses a comma decimal and a period group and does not group four-digit numbers 1. Google's say-as currency falls back to the current locale when the language attribute is omitted 10. ElevenLabs lists phone numbers, currencies, dates and times as common failure cases 9.
Phone numbers. Polly reads 7- and 10-digit numbers in es-ES, es-MX and es-US, needs the tag for bare digits, and says the grouping logic is language-specific 8. Google's zero-as-zero telephone style works only in English locales 10. None of the vendor documentation reviewed here states how a Spanish voice groups phone digits, so measure it.
Normalization inside the model. ElevenLabs' apply_text_normalization takes auto (the default), on or off, and its language_code is not supported for multilingual_v2 models 11; the Flash and Multilingual rows above show one string read differently by model choice alone, so the reading of numbers and dates ships with the model version.
CAUTION
Vendor-reported. Treat it as a hypothesis to test on your own calls.
Cartesia's changelog for Sonic 3.6 (generally available 27 August 2026) says dates, times and phone numbers now read correctly in every supported language, with currency, numbers and units coming soon 12. Currency, the class that matters most here, is not yet covered, and the gate runs either way.
Version drift. Steelyard pins the TTS version string (vendorB-v4@8k) and the locale in every trace, and its policy treats any model, voice, locale or normalization change as a new TTS for the gate, whether or not the changelog mentions numbers.
Send amounts and dates as words in the call language, such as "December third" or "el tres de diciembre" [the third of December]. Words take the normalizer out of the path for the entities that matter, but the LLM now verbalizes them, so the amount check must parse the words back to numbers.
What each option moves where:
| Input | Who verbalizes | What can go wrong | What the check needs |
|---|
| Digits, engine default ("$680.00", "12/03") | TTS normalizer | Field order, swapped separators, version drift | Round trip only; text checks pass while audio fails |
| Digits with an explicit format (SSML say-as or a locale parameter) | TTS normalizer, constrained | Engines that ignore or lack the setting; no standard currency type; settings lost in a vendor swap | Round trip, plus a config check per request |
| Words in the call language ("el tres de diciembre") | LLM | The LLM writes the wrong words for the right number | Parse TTS input text to cents and ISO, compare with the tool result, then the round trip |
The middle row is weaker than it looks. The W3C say-as document is a Working Group Note, not a Recommendation; it makes the date format optional and defines no currency type, so currency handling is vendor-specific 13. Some voices ignore the markup: for Chirp 3: HD voices, es-US among them, Google supports SSML only on synchronous requests, not streaming, so a streaming agent has to send text that is already normalized 14. ElevenLabs recommends writing phone numbers, currencies and dates out in words, for example by instructing the LLM 9.
Steelyard chose words after T06, as a rule in prompt collections-v17 (live 2026-11-13). With words, the logged text is the spoken text, so the deterministic C13 check on the TTS input text covers the content of the audio, and the round trip only has to catch pronunciation. With digits, C13 can pass on "$680.00" while the voice says something else.
The cost is why this is a decision. The LLM can now write "two hundred fifteen dollars" for $250.00, T01's confusion made on the speaking side, and no TTS test sees it. So C13 parses the words back to integer cents with the shared canonicalizer and compares them with the tool result on 100% of turns, in both languages. Digit strings meant to be written down (phone numbers, confirmation numbers, loan tails) are written as single-digit words ("four four one seven", "cuatro cuatro uno siete" [four four one seven]), because a borrower copies them rather than computing with them.
Worked example: ten Steelyard lines. Scored by spoken-entity exact match, numeric input passes 5 of 10 and words pass 10 of 10, and every numeric failure falls in a class that a frequency-weighted test set undersamples. The round-trip transcripts are constructed, as in the R5 recipe's fixture, from documented and plausible engine behavior, not measured on a vendor.
The fixture sends Spanish lines with a bare "es" locale (Spain's conventions, so a comma is a decimal sign), reads numeric dates on Spanish lines day-first as CLDR es-US does, and reads "@" and "." by their symbol names. Its Spanish phone reading in pairs, with the STT dropping one "cero" [zero], is an assumption, since no vendor documents Spanish grouping. Brackets give canonical values.
| # | Entity | CRM value | Numeric input | Heard (constructed) | Match | Words input | Match |
|---|
| 1 | Amount, English | $425.00 | $425 | four hundred twenty-five dollars | yes | four hundred twenty-five dollars | yes |
| 2 | Amount, Spanish | $130.00 | $130 | ciento treinta dolares [$130.00] | yes | ciento treinta dolares | yes |
| 3 | Amount, Spanish | $680.00 | $680 | seiscientos ochenta dolares [$680.00] | yes | seiscientos ochenta dolares | yes |
| 4 | Amount, English | $250.00 | $250 | two hundred fifty dollars | yes | two hundred fifty dollars | yes |
| 5 | Amount, Spanish | $1,500.00 | $1,500 | uno coma cinco dolares [$1.50] | no | mil quinientos dolares | yes |
| 6 | Amount, Spanish | $120,000.00 | $120,000 | ciento veinte dolares [$120.00] | no | ciento veinte mil dolares | yes |
| 7 | Date, Spanish | 2026-12-03 | 12/03 | doce de marzo [2027-03-12] | no | el tres de diciembre | yes |
| 8 | Date, Spanish | 2026-11-25 | 11/25 | once barra veinticinco [11/25, mdy policy: 2026-11-25] | yes | el veinticinco de noviembre | yes |
| 9 | Phone, Spanish | 5205550199 | (520) 555-0199 | pairs, one "cero" lost [520555199] | no | cinco dos cero, cinco cinco cinco, cero uno nueve nueve | yes |
| 10 | Email, English | advocate@steelyard.example | advocate@steelyard.example | advocate at sign steelyard period example | no | advocate at steelyard dot example | yes |
Glosses for the Spanish words: line 2 "one hundred thirty dollars", line 3 "six hundred eighty dollars", line 5 "fifteen hundred dollars" (heard: "one point five dollars"), line 6 "one hundred twenty thousand dollars" (heard: "one hundred twenty dollars"), line 7 "the third of December" (heard: "twelfth of March"), line 8 "the twenty-fifth of November" (heard: "eleven slash twenty-five"), line 9 the digits one by one.
What the table shows:
- Amounts look clean until a Spanish line crosses a thousand. Lines 1 to 4 are typical payments and pass in both languages; lines 5 and 6 are off by a factor of a thousand, because the comma meant a group in the writer's locale and a decimal in the reader's.
- The worst date error is a plausible date. Line 8 survives because no month 25 exists. Line 7 becomes a valid date about three months after the real one.
- Words remove the normalizer from every failure here. That moves the remaining risk to the LLM's verbalization, which C13 checks on the text.
An English-trained MOS predictor sees none of this. Lines 5, 6, 7 and 10 can be fluent, and a naturalness score has no input that says "doce de marzo" [twelfth of March] is the wrong month. That is a property of the metric, not a measured score.
The R5 recipe, round-trip TTS numbers and dates, runs this scorer on 200 prompts and the 400-line gate.
Can you trust UTMOS for Spanish?
Not until it is checked against native listeners. UTMOS and UTMOSv2 were trained on English MOS data, and one multilingual study found UTMOSv2's absolute Spearman correlation with human MOS reached 0.7 in only 3 of 10 languages, with 0.26 on Hindi. Until validated, use it only as a drift signal inside one voice.
UTMOS is a learned predictor of mean opinion score (MOS): it returns the score a listening panel would likely give a clip. Its training data is English: BVCC for UTMOS, plus the English-only Blizzard Challenge 2008-2011 sets and SOMOS for UTMOSv2 15 16. The OpenBibleTTS figure is a system-level correlation over six conditions 17. Six conditions is a small sample and read Bible speech is far from a Spanglish call on an 8 kHz line, but neither caveat helps the predictor: nobody has shown it works for Hazel's audio in English, Spanish or Spanglish.
How Steelyard uses it, if at all:
- Not as a gate. No release decision rests on a UTMOS number for any of Hazel's voices on phone-path audio.
- As a drift alarm inside one voice. A sudden drop for the same voice, version, locale and lines flags a change worth a listen. That relative use needs stability, not calibration.
- Validated before anything more. Ranking candidate voices with it requires a correlation study against Steelyard's own native-listener ratings, per language group (English, Spanish-dominant, Spanglish), on phone-path audio. A predictor is a judge, and the judges chapter says how to validate one.
How should humans rate voices for Spanish and Spanglish callers?
Use native listeners for each language group with exposure to the accents in the book, include older listeners if a book skews older, play phone-path audio, and prefer pairwise preference or comparative MOS over absolute scores. Plan for 20 to 40 listeners as a starting point for stable ranks among a few voices.
Who listens. Three pools match the language groups: native English speakers, Spanish-dominant listeners, and listeners who code-switch themselves, because a Spanglish turn that sounds fine to a monolingual listener can sound wrong to people who talk that way. Take each listener's language from what they report, never from a surname. Because SDL-biwk-nov serves older borrowers such as P13 (Ramon Diaz, 64, Spanish-dominant, hard of hearing, asks for numbers to be repeated), that campaign's voice and speaking rate get a share of listeners over 60.
What they judge. Absolute 5-point MOS asks each listener to place one clip on a private scale. For choosing between an incumbent and a candidate, pairwise preference (A or B, same line, randomized order) or comparative MOS (CMOS, a graded A-versus-B difference) asks the question you actually have. The pairwise and Elo technique page covers position bias and ties. Keep naturalness questions apart from intelligibility: ask "which voice would you rather hear on this call?" separately from "write down the amount you heard".
The exception to binary. This site's rubric design advice prefers binary pass or fail judgments over rating scales. Naturalness is the exception: it is a graded perception, protocols such as ITU-T P.808 for crowdsourced MOS are built on rating scales 18, and "natural enough" hides the size of a preference. Entity correctness stays binary.
How many listeners. The closest evidence is a MUSHRA study with 492 native-speaker listeners in Hindi and Tamil. Ranks taken from a subset of the panel matched the full Hindi panel's ranks (Spearman above 0.90) only with at least 20 listeners, and more than 30 listeners on 30 utterances always gave above 0.95; the no-reference variant MUSHRA-NMR passed 0.95 with 20 utterances or 40 listeners 19. Those figures are rank agreement with the paper's own panel over three TTS systems plus an anchor and a reference. Treat 20 to 40 listeners per language pool as a starting point for stable ranks among a handful of systems, not a rule for pairwise, CMOS, English or Spanish tests.
How to report it. Listeners and utterances are both clustered sources of variance, so put intervals on preference shares and mean scores with a bootstrap that resamples listeners (and utterances), never a Wilson interval on pooled votes. Report agreement between listeners as well, with the methods in inter-rater agreement; a preference that half the panel reverses is a split panel, not a 50% tie. Report each language pool on its own: a candidate that wins in English and loses in Spanish has not won.
Why test through the phone path?
Because the caller hears 8 kHz, band-limited, transcoded audio, not the 24 kHz demo. Run the round trip and the listening panel on phone-path audio, and request the format the carrier receives where the engine offers it.
Hazel's TTS requests 8 kHz mu-law output directly (vendorB-v4@8k), with the locale set per call to en-US or es-US, and the tests use the same format and the same locale tags. Building the rest of the path, codec chains included, belongs to the telephony audio chapter. No TTS result counts unless it went through that path.
What gate would have caught it?
The TTS acceptance gate: spoken-entity exact match of at least 99.5% on the numerics suite after the phone path, in each locale; zero wrong amounts in 400 amount lines; pairwise preference not worse than the incumbent at 8 kHz; and a pinned TTS version, rerun on every update. T06 fails the first part.
The four parts:
- Spoken-entity exact match >= 99.5% on the 500-line numerics suite after TTS, phone path, STT and canonicalization, reported per entity type and per locale next to the STT floor.
- Zero wrong amounts in the 400 amount lines. Zero errors is not a zero error rate. With 0 of 400 wrong, the exact one-sided 95% upper bound on the true amount error rate is 1 - 0.05^(1/400) = 0.00746, about 0.75%, close to the rule-of-three shortcut 3/400 = 0.0075 20. That is the strongest claim 400 clean lines support.
- Pairwise preference not worse than the incumbent, on phone-path clips, judged by the panel above, with a listener-clustered bootstrap interval per language pool. The non-inferiority margin is written into the memo before the panel runs, so it cannot be chosen after the result.
- A pinned version. The TTS version, voice, locale and normalization settings are recorded in every trace, and any change to them reruns parts 1 to 3.
Know what the second part cannot prove. The 0.75% bound sits above the 0.5% the 99.5% line implies: a clean suite shows amount errors below about 1 in 134, not 1 in 200. Bounding the amount error below 0.5% with zero failures needs 598 clean lines (0.995^598 = 0.0499); below 0.1% needs 2,995. Steelyard accepts the 0.75% bound for TTS because C13 checks every amount on the text side on 100% of turns and the read-back under C12 gives the borrower a chance to correct the number; a team without those backstops should size the suite to its own risk. The 400 lines are also split across two locales, so the bound on the Spanish amount lines alone is wider.
Run on prompt collections-v16, part 1 would have failed on a cluster of es-US date lines. The words rule in v17 has to pass the same gate; it is not assumed to work.
What goes in the decision memo?
The memo turns the gate into a decision about one change: whether Steelyard may swap or upgrade its TTS.
Decision. Adopt TTS acceptance gate v1 for every TTS change, including the pending vendor evaluation. vendorB-v4@8k stays pinned, with en-US and es-US set explicitly per call, until a candidate passes all four parts in both locales. Words in the call language remain the input format for amounts and dates.
Context. T06 (2026-11-11): a month-first date read day-first by the es-US voice; the text was right, the audio wrong, and transcript review passed it.
Options considered. (a) Listening panel alone: misses entity errors. (b) Digits with SSML say-as or a locale parameter: depends on each engine honoring it, no standard currency type, and some streaming voices ignore SSML. (c) Words plus the round-trip gate: chosen; its LLM verbalization risk is covered by C13 on 100% of turns.
Evidence. gold-v7 (500 TTS lines, 400 amounts); 0 of 400 amounts wrong bounds the rate below about 0.75% (one-sided 95%); constructed 10-line example, numeric 5 of 10, words 10 of 10; UTMOSv2 Spearman 0.26 on Hindi in one study and no validation for Spanish, so no UTMOS gate.
Owner and cadence. Staff engineer, voice platform: suite, gate, version and locale pinning. Collections QA lead: listening panels per language pool. Compliance lead: C06 strings and required notices in the suite. Rerun on every TTS version, voice, locale or setting change and before each release.
Tripwires. Any wrong amount in a gate run. A vendor model or normalization change. An STT version change (the floor moves). A new spoken string required by policy, or a new locale such as es-ES, with no suite lines.
Dated facts. Azure, Polly, Google, ElevenLabs and Cartesia normalization behavior (docs, checked 2026-11-20); CLDR 48 locale data; EmergentTTS-Eval (2025); OpenBibleTTS (June 2026); MUSHRA listener counts (v3, May 2025).
What to do this week
Pull every TTS template your agent uses, fill each with synthetic values that cross a thousand dollars, use days 1 to 12 and contain zeros, and run them through your production TTS at 8 kHz and one pinned STT, once per locale. Score by entity type and locale, with a human-read floor beside it.
- Log the normalization locale and date-format settings each TTS request sends, and flag any request tagged with a bare language code such as "es".
- Add every string a policy or regulation makes the agent say aloud to the numerics suite, in each language it is said in.
- Decide the input format for amounts and dates, and if it is words, add the words-to-cents parse to the C13 check for both languages.
- Pin the TTS version string and locale, and make a version change without a suite run fail CI.
- Book native-listener pairwise panels per language pool on phone-path clips before the next voice change, and write the non-inferiority margin down first.
- Stop reporting UTMOS for a Spanish voice as a quality number until you have checked it against your own panel.
NOTE
What will date on this page. Azure, Polly and Google say-as behavior, Google's synchronous-only SSML for Chirp 3: HD voices, ElevenLabs normalization and its Flash v2.5 versus Multilingual v2 example, and Cartesia's normalization locales (documentation, checked 2026-11-20). Cartesia's Sonic 3.6 changelog, where currency, numbers and units are "coming soon" (August 2026). CLDR 48 date and number patterns. EmergentTTS-Eval's 88.84% and 40.40% (NeurIPS 2025 Datasets and Benchmarks). UTMOSv2 on Hindi (OpenBibleTTS preprint, June 2026; a Spanish or Spanglish validation would change the picture). MUSHRA listener counts (arXiv v3, 27 May 2025, TMLR).