On 2026-11-11 Asha's LLM wrote "Your next EMI of Rs 6,800 is due on 05/12.", the text-to-speech (TTS) engine read the date as "May twelfth", the borrower agreed, and the CRM stored 5 December (trace T06). Every log line was correct and the borrower heard the wrong month. This chapter is about testing what the caller hears: spoken amounts, dates and digits first, naturalness second, and both on phone-line audio.
What happened on the call?
The LLM wrote the due date as "05/12", the TTS engine read it in en-US order as May twelfth, and the CRM stored 5 December. Prompt collections-v16 allowed digits in TTS input, and the English voice ran with the engine's default en-US normalization.
Excerpt from trace T06:
t_ms ch event / text
agent LLM output: "Your next EMI of Rs 6,800 is due on 05/12."
agent TTS (en-IN voice, normalization default en-US) audio: "...due on May twelfth."
caller REF: "Okay, fine, I'll pay then."
agent tool record_promise_to_pay(ptp_date=2026-12-05, ...) (CRM holds 5 Dec; borrower heard 12 May)
The transcript view shows nothing wrong: an Indian reader takes "05/12" as 5 December, the tool call says 2026-12-05, and the borrower sounds agreeable. The fault exists only in the audio.
Diagnosis: the engine's normalization locale chose the reading, a transcript-only review passed the call, and no test compared the spoken date with the stored one.
Why does entity correctness beat naturalness for an agent?
An agent's speech carries commitments: an amount, a due date, a phone number. A wrong one creates a broken promise or a compliance failure; a slightly less natural voice costs a little goodwill. The two qualities do not travel together, so test entities first and rank naturalness only among voices that pass.
EmergentTTS-Eval makes the split concrete. gpt-4o-audio-preview with the Ballad voice and strong prompting won 88.84% of judge-scored comparisons against a gpt-4o-mini-tts baseline on emotional dialogue, but only 40.40% on complex pronunciation: emails, phone numbers, URLs, addresses and similar strings 1. The same system was far ahead on the dimension a demo shows off and won fewer than half its comparisons on the one a collections call depends on.
For Tarazu the ordering is not taste. C12 (read-back), C13 (amount stated equals the get_dues result) and C06 (grievance officer's name and phone on every right-party call) are all spoken entities, owned by the compliance chapter. No MOS gain buys back one wrong amount on a recorded call. The voice playbook sets MOS targets for TTS in general; for an agent, MOS and preference scores decide between candidates only after the entity gate passes, and never compensate for a failure on it.
How do you build a TTS test set?
Turn the TTS input texts in production traces into templates, fill them with values from the CRM's real distributions, and oversample the cases where the engine must choose: amounts of a lakh and above, dates with a day from 1 to 12, digit strings with zeros, emails and addresses.
Templates, not raw turns. The tts.text field in a trace is PII under the trace rules in chapter 1, so do not copy real lines into a test set. Group production turns by template ("Aapka overdue {amount} hai"), keep each template's frequency, and fill the slots with synthetic values from the CRM's per-product distributions. Templates keep the set honest about what Asha says; synthetic values keep it shareable.
Oversample by entity class, not by frequency. Tarazu's EMIs run from Rs 1,100 to Rs 72,000, all below a lakh, where Indian and Western digit grouping agree. A frequency-weighted set is mostly lines where grouping is never at stake. Oversample:
- Currency. Amounts in lakh and crore (SBL tickets up to Rs 25,00,000, foreclosure amounts, the guarantor's Rs 12,00,000 loan), and the colloquial fractions when Asha uses them: sava (1.25), dedh (1.5), dhai (2.5), paune X (X minus 0.25), saade X (X plus 0.5). "Dhai hazaar" heard as "do hazaar" is T01 in reverse.
- Hindi and English number words. A Hinglish line can carry "chaar hazaar do sau pachaas" or "four thousand two hundred fifty", so the set holds both for every amount template.
- Dates. Spoken forms ("paanch December", "fifth December") and, for the input-format comparison below, numeric dates with the day from 1 to 12.
- Digits. Loan tails ("chaar chaar ek saat"), the phone number 020-4000-0000 in its spoken groups, 12-digit UTRs read in groups, and pincodes with zeros.
- Names. Borrower names with the honorific ("Rahul Verma ji", "Sunita ji") and the grievance officer, Ms Neha Kulkarni.
- Email and address. "grievance@tarazu.example" and "Tarazu House, Baner Road, Pune 411045". On dispute, complaint and hardship paths Asha offers to read them aloud (C06), and they are exactly the complex-pronunciation class where EmergentTTS-Eval found the strong expressive system weakest.
Tarazu's gold-v7 holds 500 TTS lines, 400 of them amounts, plus dates, loan and phone digits, pincodes, the grievance email and the Tarazu House address. The 400 is sized for the gate below.
How do you score a spoken entity?
Send the line through the production TTS, the phone path and one pinned STT, canonicalize the transcript (amount to paise, date to ISO), and exact-match each entity against the CRM value. Beside every result, report the STT floor: the same STT on a human reading the same line.
The loop:
flowchart LR
A["CRM value<br/>250000 paise"] --> B["TTS input text<br/>dhai hazaar rupaye"]
B --> C["TTS<br/>pinned version and voice"]
C --> D["Phone path<br/>8 kHz mu-law, codec chain"]
D --> E["STT<br/>pinned, code-mix mode"]
E --> F["Canonicalize<br/>W7 rules"]
F --> G{"Exact match<br/>with CRM value?"}
G -->|yes| H["pass"]
G -->|no| I["fail: listen to the clip"]
J["Human reading of<br/>the same line"] --> D
Three rules keep the score meaningful.
Report the floor. The round trip conflates two systems: if the STT mishears "dhai" as "do" on a person's voice, the TTS gets blamed for the listener. So every run also passes a consented human recording of the same written-out line through the same phone path and STT. At or above the floor, the STT is the limiting link and each miss needs a listen; below the floor, the TTS is the suspect. Reuse the canonicalization rules the entity error rate chapter defines for the STT side; do not fork them.
Exact match per entity, not CER per line. A line can be 95% right by characters and carry a wrong amount. Score each entity right or wrong and report by entity type; a pooled pass rate hides the class that fails.
Never gate on the best of several readings. One tempting scorer generates several spoken-form variants of the reference and several transcription candidates, computes CER for every pair, and keeps the minimum. A caller hears one rendering once. Min-over-variants scoring helps diagnose a failure; as a gate, it rewards the vendor for your scorer's flexibility. Gate on one pinned STT and one canonical value.
STT normalization rules are owned by the normalization chapter. Here they matter as a source of false failures: if the canonicalizer cannot parse "paune do lakh", the TTS looks broken when it is not.
Why do dates get read differently?
The engine's normalization locale or date-format setting decides how a numeric date is spoken, whatever the voice's accent: an en-IN voice over en-US normalization reads 05/12 as May twelfth. Some models carry normalization inside them, so a model update can change how numbers and dates are read.
The first five rows are what the cited documentation says; the last row is the incident.
| Engine and setting | Written | Heard | Basis |
|---|
| Azure, default | 10-12-2016 | October twelfth | Docs 2 |
| Azure, format="dmy" | 10-12-2016 | December tenth | Docs 2 |
| Cartesia, normalization en-US, British voice | 03/04/2026 | March fourth | Docs 3 |
| Sarvam bulbul:v2, enable_preprocessing default false | Rs 1,50,000 or 05/12 | Numeric normalization off; the raw reading is not documented | Docs 4 |
| Sarvam bulbul:v3 or bulbul:v4-flash | Rs 1,50,000 or 05/12 | Normalization always on; the reading comes with the model version | Docs 4 |
| Tarazu vendorB, en-IN voice, normalization default en-US | 05/12 | May twelfth | Incident T06 |
Locale defaults. The voice and the normalizer are separate settings. Picking an Indian voice does not pick Indian date order. Fix the configuration with an explicit format or locale on every request, and test with numeric dates whose day runs from 1 to 12, where the wrong reading is a valid date nobody notices.
Normalization inside the model. Sarvam's API exposes an enable_preprocessing flag (normalization of English words and numeric entities such as numbers and dates) only on the legacy bulbul:v2, where it defaults to false; on bulbul:v3 and bulbul:v4-flash preprocessing is always on and the flag is not supported 4. You cannot test that normalizer on its own: the reading of numbers and dates comes with the model version, and a model change can change it.
Version drift. Tarazu pins the TTS version string (vendorB-v4@8k) in every trace, and its policy treats any model, voice or normalization change as a new TTS for the gate, whether or not the changelog mentions numbers.
Send amounts and dates as words in the call language: "chaar hazaar do sau pachaas rupaye", "paanch December". That takes the normalizer out of the path for the entities that matter, but makes the LLM responsible for verbalizing them, so the compliance check must parse the words back.
Three options, with what each moves where:
| Input | Who verbalizes | What can go wrong | What the check needs |
|---|
| Digits, engine default ("Rs 6,800", "05/12") | TTS normalizer | Locale order, Indian grouping read as digits, version drift | Round trip only; text checks pass while audio fails |
| Digits with an explicit format (SSML say-as or a locale parameter) | TTS normalizer, constrained | Engines that ignore or lack the setting; settings lost in a vendor swap | Round trip, plus a config check per request |
| Words in the call language ("paanch December") | LLM | The LLM writes the wrong words for the right number | Parse TTS input text to paise and ISO, compare with the tool result, then the round trip |
Tarazu chose words after T06, as a rule in prompt collections-v17. The deciding argument: with words, the logged text is the spoken text, so the deterministic C13 check on the TTS input text covers the audio's content and the round trip only has to catch pronunciation. With digits, C13 can pass on "Rs 6,800" while the voice says something else.
The cost is why this is a decision. The LLM can now write "chaar hazaar do sau paanch" for Rs 4,250, an error no TTS test sees. So C13 parses the words back to integer paise with the shared canonicalizer and compares them with the tool result on 100% of turns. Digit strings meant to be written down (phone numbers, UTRs, loan tails) stay digits spoken one by one ("chaar chaar ek saat"), because a borrower copies them rather than computing with them.
Worked example: ten Tarazu lines. Scored by spoken-entity exact match, numeric input passes 5 of 10 and words pass 10 of 10, and every numeric failure falls in a class that a frequency-weighted test set undersamples. The round-trip transcripts are constructed, as in the R5 recipe's fixture, from documented and plausible engine behaviour; nothing here was measured on a vendor.
The fixture reads numeric dates in en-US order, reads Indian grouping above a lakh digit by digit and loses a zero on the line, reads pincodes in pairs with the STT dropping "oh", and reads "@" and "." by their symbol names.
| # | Entity | CRM value | Numeric input | Heard (constructed) | Match | Words input | Match |
|---|
| 1 | Amount | Rs 4,250 | Rs 4,250 | rupees four thousand two hundred fifty | yes | chaar hazaar do sau pachaas rupaye | yes |
| 2 | Amount | Rs 6,800 | Rs 6,800 | rupees six thousand eight hundred | yes | chhah hazaar aath sau rupaye | yes |
| 3 | Amount | Rs 1,300 | Rs 1,300 | rupees one thousand three hundred | yes | ek hazaar teen sau rupaye | yes |
| 4 | Amount | Rs 2,500 | Rs 2,500 | rupees two thousand five hundred | yes | dhai hazaar rupaye | yes |
| 5 | Amount | Rs 1,50,000 | Rs 1,50,000 | rupees one five zero zero zero | no (Rs 15,000) | dedh lakh rupaye | yes |
| 6 | Amount | Rs 12,00,000 | Rs 12,00,000 | rupees one two zero zero zero zero | no (Rs 1,20,000) | baarah lakh rupaye | yes |
| 7 | Date | 2026-12-05 | 05/12 | May twelfth | no (2027-05-12) | paanch December | yes |
| 8 | Date | 2026-11-25 | 25/11 | twenty five slash eleven | yes (dmy policy) | pachchees November | yes |
| 9 | Pincode | 411007 | 411007 | forty one ten seven | no (41107) | chaar ek ek zero zero saat | yes |
| 10 | Email | grievance@tarazu.example | grievance@tarazu.example | grievance commercial at tarazu full stop example | no | grievance at tarazu dot example | yes |
What the table shows:
- Amounts look clean until they cross a lakh. Lines 1 to 4 are typical EMIs and pass; lines 5 and 6 are off by a factor of ten.
- The worst date error is a plausible date. Line 8 survives because no month 25 exists. Line 7 becomes a valid date about five months after the real one.
- The email fails in EmergentTTS-Eval's weak class. No borrower writes "commercial at" down as an address.
- Words remove the normalizer from every failure here. That moves the remaining risk to the LLM's verbalization, which C13 checks on the text.
An English-trained MOS predictor sees none of this. Lines 5, 6, 7 and 10 can be fluent, and a naturalness score has no input that says "May twelfth" is the wrong month. That is a property of the metric, not a measured score; measured scores get added here only when Tarazu has publishable synthesized audio for these lines.
The R5 recipe, round-trip TTS numbers and dates, runs this scorer on 200 prompts and the 400-line gate.
Can you trust UTMOS for Hindi?
Not without checking it against native listeners. UTMOS and UTMOSv2 were trained on English MOS data, and in one 2026 study UTMOSv2 reached |Spearman| >= 0.7 with human MOS in only 3 of 10 languages, and 0.26 on Hindi. Use it, at most, as a drift signal inside one voice.
UTMOS is a learned predictor of mean opinion score (MOS): give it a clip and it returns the score a listening panel would likely give. Its training data is English: BVCC for UTMOS, plus the English-only Blizzard Challenge 2008-2011 sets and SOMOS for UTMOSv2 5 6. The OpenBibleTTS Hindi figure is a system-level correlation over six conditions 7. Six conditions is a small sample and read Bible speech is far from Hinglish on an 8 kHz line, but neither caveat helps the predictor: nobody has shown it works for Asha's audio.
How Tarazu uses it, if at all:
- Not as a gate. No release decision rests on a UTMOS number for Hindi or Hinglish.
- As a drift alarm inside one voice. A sudden drop for the same voice, version and lines flags a change worth a listen. That relative use needs stability, not calibration.
- Validated before anything more. Ranking candidate voices with it requires a correlation study against Tarazu's own native-listener ratings, per language group, on phone-path audio. A predictor is a judge, and the judges chapter says how to validate one.
How should humans rate voices for Hinglish callers?
Use native listeners with exposure to the callers' accents, include older listeners if the book skews older, play phone-path audio, and prefer pairwise preference or comparative MOS over absolute scores. Plan for 20 to 40 listeners as a starting point for stable ranks among a few voices.
Who listens. The panel should hear the voice the way Tarazu's borrowers do. That means native Hindi and Hinglish speakers with exposure to the regional accents in the book, and, because MFI-fort-nov serves older borrowers such as P13 (64, hard of hearing, asks for numbers to be repeated), a share of listeners over 60 for that campaign's voice and speaking rate.
What they judge. Absolute 5-point MOS asks each listener to place one clip on a privately calibrated scale. For choosing between an incumbent and a candidate, pairwise preference (A or B, same line, randomized order) or comparative MOS (CMOS, a graded A-versus-B difference) asks the question you actually have. The pairwise and Elo technique page covers position bias and ties. Keep naturalness questions apart from intelligibility: ask "which voice would you rather hear on this call?" separately from "write down the amount you heard".
The exception to binary. This site's rubric design advice prefers binary pass or fail judgments over rating scales. Naturalness is the exception: it is a graded perception, protocols such as ITU-T P.808 for crowdsourced MOS are built on rating scales 8, and "natural enough" hides the size of a preference. Entity correctness stays binary.
How many listeners. The listener-count evidence closest to Tarazu's case comes from a MUSHRA study with 492 native-speaker listeners in Hindi and Tamil. There, ranks taken from a subset of the panel matched the full Hindi panel's ranks (Spearman above 0.90) only with at least 20 listeners, and more than 30 listeners on 30 utterances always gave above 0.95; the no-reference variant MUSHRA-NMR passed 0.95 with 20 utterances or 40 listeners 9. Those figures are rank agreement with the paper's own panel over three TTS systems plus an anchor and a reference. Treat 20 to 40 listeners as a starting point for stable ranks among a handful of systems, not a rule for pairwise, CMOS or Hinglish tests.
How to report it. Listeners and utterances are both clustered sources of variance, so put intervals on preference shares and mean scores with a bootstrap that resamples listeners (and utterances), never a Wilson interval on pooled votes. Report agreement between listeners as well, with the methods in inter-rater agreement; a preference that half the panel reverses is a split panel, not a 50% tie.
Why test through the phone path?
Because the caller hears 8 kHz, band-limited, transcoded audio, not the 24 kHz demo. Run the round trip and the listening panel on phone-path audio, and request the codec the carrier receives where the engine offers it.
Asha's TTS requests 8 kHz mu-law output directly (vendorB-v4@8k), and the tests use the same format; Sarvam's API, for one, can return 8000 Hz audio in mulaw or alaw, so you can test the audio the carrier receives 4. Building the rest of the path, and why the band matters more than mu-law companding, belong to the telephony audio chapter. No TTS result counts unless it went through that path.
What gate would have caught it?
The TTS acceptance gate: spoken-entity exact match of at least 99.5% on the numerics suite after the phone path, zero wrong amounts in 400 amount lines, pairwise preference not worse than the incumbent at 8 kHz, and a pinned TTS version rerun on every update. T06 fails the first part.
The four parts:
- Spoken-entity exact match >= 99.5% on the 500-line numerics suite after TTS, phone path, STT and canonicalization, reported per entity type next to the STT floor.
- Zero wrong amounts in the 400 amount lines. Zero errors is not a zero error rate. With 0 of 400 wrong, the exact one-sided 95% upper bound on the true amount error rate is 1 - 0.05^(1/400) = 0.00746, about 0.75%, close to the rule-of-three shortcut 3/400 = 0.0075 10. That is the strongest claim 400 clean lines support.
- Pairwise preference not worse than the incumbent, on phone-path clips, judged by the panel above, with a listener-clustered bootstrap interval. The non-inferiority margin is written into the memo before the panel runs, so it cannot be chosen after the result.
- A pinned version. The TTS version, voice and normalization settings are recorded in every trace, and any change to them reruns parts 1 to 3.
Know what the second part cannot prove. The 0.75% bound sits above the 0.5% the 99.5% line implies: a clean suite shows amount errors below about 1 in 134, not 1 in 200. Bounding the amount error below 0.5% with zero failures needs 598 clean lines (0.995^598 = 0.0499); below 0.1% needs 2,995. Tarazu accepts the 0.75% bound for TTS because C13 checks every amount on the text side on 100% of turns and the read-back under C12 gives the borrower a chance to correct the number; a team without those backstops should size the suite to its own risk.
Against T06, part 1 fails on the first run, with the en-US default showing up as a cluster of date failures. The words rule is then tested by the same gate, not assumed to work.
What goes in the decision memo?
The memo turns the gate into a decision about one change: whether Tarazu may swap or upgrade its TTS.
Decision. Adopt TTS acceptance gate v1 for every TTS change, including the pending vendor evaluation. vendorB-v4@8k stays pinned until a candidate passes all four parts. Words in the call language remain the input format for amounts and dates.
Context. T06 (2026-11-11): a numeric date read in en-US order, text correct, audio wrong, CRM and borrower disagree. Transcript review passed it.
Options considered. (a) Listening panel alone: misses entity errors. (b) Digits with an explicit locale: depends on each engine honouring it. (c) Words plus the round-trip gate: chosen; its LLM verbalization risk is covered by C13 on 100% of turns.
Evidence. gold-v7 (500 TTS lines, 400 amounts); 0 of 400 amounts wrong bounds the rate below about 0.75% (one-sided 95%); constructed 10-line example, numeric 5 of 10, words 10 of 10; UTMOSv2 Spearman 0.26 on Hindi in one study, so no UTMOS gate.
Owner and cadence. Staff engineer, voice platform: suite, gate, version pinning. Collections QA lead: listening panel. Compliance lead: C06 strings in the suite. Rerun on every TTS version, voice or setting change and before each release.
Tripwires. Any wrong amount in a gate run. A vendor model or normalization change. An STT version change (the floor moves). A new spoken string required by policy with no suite lines.
Dated facts. Azure and Cartesia date behaviour and Sarvam preprocessing (docs, checked 2026-11-20); EmergentTTS-Eval (2025); OpenBibleTTS (June 2026); MUSHRA listener counts (v3, May 2025).
What to do this week
Pull every TTS template Asha uses, fill each with synthetic values that cross a lakh, use days 1 to 12 and contain zeros, and run them through your production TTS at 8 kHz and one pinned STT. Score by entity type, with a human-read floor beside it.
- Log the normalization locale and date-format settings each TTS request sends, per language.
- Add every string a regulation makes the agent say aloud to the numerics suite.
- Decide the input format for amounts and dates, and if it is words, add the words-to-paise parse to the C13 check.
- Pin the TTS version string and make a version change without a suite run fail CI.
- Book a native-listener pairwise panel on phone-path clips before the next voice change, and write the non-inferiority margin down first.
- Stop reporting UTMOS for Hindi voices as a quality number until you have checked it against your own panel.
NOTE
What will date on this page. Azure SSML date-format behaviour and Cartesia's en-US normalization example (documentation, checked 2026-11-20). Sarvam's enable_preprocessing flag and always-on preprocessing on bulbul:v3 and bulbul:v4-flash, and its 8000 Hz mulaw and alaw output (API docs, fetched 2026-11-20; Sarvam ships model versions often). EmergentTTS-Eval's 88.84% and 40.40% (NeurIPS 2025 Datasets and Benchmarks). UTMOSv2 on Hindi (OpenBibleTTS preprint, June 2026; a later version or a Hinglish study may change the picture). MUSHRA listener counts (arXiv v3, 27 May 2025, TMLR).