On trace T08, Neha Gupta (persona P12) said her salary comes on the 7th and she could pay after that, STT locked onto English halfway through her sentence, and Asha asked for the date three times until she hung up at 71 seconds. The regression suite had passed, because none of its scenarios switched language mid-clause on the audio path.
What happened, and why did the regression suite pass?
STT language identification locked onto English after "but", the date came out as "now three", and Asha looped. The suite passed because no scenario in it produced a caller who switches language every clause through the real 8 kHz audio path.
T08, 2026-11-16, persona P12, flow F01:
t_ms ch event / text
caller REF: "haan, but meri salary 7 ko aati hai, toh after that I can pay, around 9th"
caller STT (language locked to en after "but"): "hum but mary salary seven ko RT hai to after that I can pay around ninth"
agent TTS: "Sorry, could you please tell me the date you can pay?"
caller REF: "nau tarikh, 9th, after salary"
caller STT: "now three, ninth after salary"
agent TTS: "Just to confirm, is it the third or the ninth?"
caller REF: "arre yaar, nine. N-I-N-E." (third clarification; hangs up at 71 s)
Diagnosis: an STT language-identification failure (STT.LID) that turned into a clarification loop (LOOP.CLARIFY), on a pattern the test set never contained.
The same shape has been published. One engineer described a regression suite of 312 recorded production calls that all passed; then a caller who switched between English and Hindi mid-sentence caused a 3am page, because STT mis-segmented the audio and the agent looped on clarifications. The pattern appeared zero times in the 312 calls 1.
A simulated caller earns its cost only if it produces the pattern that hurt you, on the path where it hurt you. It is also a second model inside your eval, and its mistakes land in the agent's pass rate unless you measure them.
Why is replaying recorded calls not enough?
A replay plays fixed caller turns. Once the new agent says something different, the recorded caller answers a question nobody asked, no one reacts to the agent's new reply, and the test holds only patterns that already happened.
Replays still earn their place for the stages that run before the agent decides anything: STT on real phone audio, the turn detector on real pauses, AMD on real call openings. The entity error rate and turn-taking chapters use them that way.
They break as soon as the conversation branches:
- Fixed caller turns. A new prompt asks for a closed read-back where the old one asked an open question. The replay feeds the recorded date to a yes/no prompt, and the result measures how the agent handles nonsense, not the change you made.
- The new reply goes nowhere. If the candidate says something harmful or confusing, no caller reacts. T02's "already paid" claim arrives because a caller reacts to the dues statement while it plays; a replay does not know when the amount is spoken.
- Only seen patterns. A rare caller behaviour has to occur in last month's sample before a replay can test it, and a new campaign has no recordings at all.
The fix is a caller that reacts: a model that holds a persona and a goal, listens to the agent, and answers what was actually said. That closes the loop, and it means the caller can now be wrong.
What does a synthetic caller need?
A scenario card that states who the caller is, what they know, how they sound, when they interrupt and what end state counts as success, plus a path that delivers their audio to the agent the way a phone call does. Without the production audio path, the audio conditions on the card are labels with nothing behind them.
Every persona in gold-v7 has a card with these fields: persona text for the simulator, audio conditions (SNR, codec chain, loss profile), an interruption script that is content-triggered rather than timer-based, the gold CRM end state, required utterances, forbidden utterances, the language mix, and a code-switch density label (low, medium, high). P12's card, abbreviated:
card: P12-F01
persona: >
Neha Gupta, PL borrower on PL-digital-nov. Switches language every clause.
Salary arrives on the 7th; will pay after that, around the 9th.
Accepts one clarification politely; on a second, repeats the date in
both languages; on a third, hangs up.
facts: {salary_day: 7, preferred_pay_day: 9, identity: dob from accounts}
language_mix: hinglish
code_switch_density: high
audio:
codec_chain: [amr-nb, g711-pcma, g711-pcmu]
condition: one cell of the telephony condition matrix (carrier, loss, SNR)
interruptions: [] # P12 tests language, not barge-in
required_caller_utterances:
- a clause-level switch inside the date answer ("salary 7 ko ... after that")
gold_end_state:
ptps: [{ptp_date: ">= the 8th", readback_confirmed: true}]
clarifications_on_date_slot: "<= 1"
forbidden_agent_utterances: [any dues before verify_identity]
Interruptions are where most simulators are weakest. P02's trigger is tied to content, not to time:
interruptions:
- when: agent TTS text contains the overdue amount
say: "arre maine already pay kar diya, teen tarikh ko, PhonePe se"
Three properties of the caller matter more than its voice quality:
- It is goal-directed and stateful. It holds facts constant (P02's payment date, P12's salary day) and ends the call when its persona would.
- It reacts to content. It cuts in when the amount is spoken and hangs up on the third clarification.
- Its audio goes through the production path. Synthesized or consented recorded speech is degraded through the card's codec chain, loss and noise, then delivered over SIP to the media stream Asha sees on a real call; testing on telephony audio builds that chain. A simulator that hands the agent text, or clean 24 kHz audio over a WebSocket, skips the stage that broke on T08.
Simulator fidelity is how closely the simulated caller's behaviour and audio match the card and the real callers it stands for. You measure it; the next four sections show how.
flowchart LR
A[Scenario store: gold-v7 cards] --> B[Persona caller: LLM + facts + interruption script]
B --> C[Speech: TTS or consented clips]
C --> D[Impairments: codec chain, loss, noise]
D --> E[SIP trunk to production media stream]
E --> F[Asha: VAD, STT, turn detector, LLM, tools, TTS]
F --> G[CRM sandbox]
F --> H[Agent audio back to the caller]
H --> B
G --> I[Validators: call end, behaviour, speech]
I -->|reject| B
I -->|accept| J[Checks and results store]
The return leg is the point: the caller hears Asha's TTS, transcribes it and reacts. A simulator that reads her text cannot be confused by her audio the way a borrower can.
Where do simulators flatter the agent?
Almost always in the same direction: simulated callers are more patient, more consistent and easier to hear than real ones, and their accents and interruptions are often settings rather than behaviour.
Patience, memory and a direct line to the agent's words. The tau-Voice authors (278 English tasks; voice agents at 31-51% in clean audio against 85% for a text model) say their simulated user "is more patient than real users, with perfect memory and instantaneous tool calls", that it reads the agent's transcript directly instead of transcribing its speech, and that accent results from TTS personas are indicative rather than definitive 2.
Synthetic speech is easier to recognize than real speech. In VoiceBench, all six voice assistants scored higher on synthetic speech than on real recorded speech for the same CommonEval instructions: 4.04 real against 4.22 synthetic for a Whisper-plus-LLM pipeline, 2.15 against 3.21 for VITA. On Indian and Philippine English, the drop was larger on real accented speech than on synthetic accents, which the authors attribute to TTS accents sounding closer to standard English 3. Take the direction, not the size: no telephony audio, no Hindi, no matched accent pairs. It is still enough to rule out a TTS "Indian English" voice as a stand-in for P13's Kannada-accented Hindi.
CAUTION
Vendor-reported. Treat it as a hypothesis to test on your own calls. The vendor behaviours below come from each vendor's own documentation, read on 2026-11-20; none of these pages carries a date.
Accent is a voice ID, not an instruction. In Cekura, the simulated caller's accent comes only from the voice ID: the accent shown on a personality is a read-only label derived from the voice, and the docs say "Evaluator instructions cannot override a personality setting". Asking for a strong accent in the instructions does nothing, and the call still returns a result 4. Its Transcription Accuracy metric in simulation scores the testing agent's speech-to-text, not yours 5. Our reading: check which side of the call a metric measures before trusting it.
Interruptions on a timer. Coval's persona settings set proactive interruptions on a timer: None, Low, Medium or High means never, about every 90, 45 or 30 seconds, not tied to what the agent is saying. A caller who cuts in when the amount is read out has to be scripted by hand 6. A timer-based interruption tests the barge-in detector, not recovery, and T02 is a recovery failure.
Code-mixing as one language. LiveKit's simulation docs say mixed speech, "such as Hindi mixed with English, runs in the non-English language" 7. A caller rendered in one language never switches mid-clause, so that suite cannot reproduce T08 at any seed count. Its outbound gaps are in the outbound section below.
Clean pipelines under-represent disfluency. ServiceNow's EVA-Bench authors note that their clean STT-LLM-TTS simulator under-represents real callers' disfluencies (v1 adds hesitations and emotional variation) and that by default it does not systematically generate interruptions 8. T05's 720 ms pause before "paanch tarikh" is exactly the disfluency a clean simulator smooths away.
Every one of these gaps makes the agent look better, so treat a simulated pass rate as an upper estimate until the sim-vs-real check says otherwise.
How do you validate the simulator itself?
Run automated validators on every simulated call before scoring the agent, reject calls where the simulated caller broke its card, and regenerate them with a new seed. Validators read only the caller's side.
ServiceNow's EVA-Bench runs each simulated call through validators for a valid call end, user behavioural fidelity and user speech fidelity, and regenerates any call that fails. Across four systems, 12% of trials were regenerated because of user simulator error, almost all of it behavioural drift 8. Tarazu uses the same three, adapted to its cards:
| Validator | What it checks | Rejects when | Tarazu example |
|---|
| Valid call end | The call reached a terminal event the flow allows | No terminal event, or the caller hung up early for a reason outside its card | P12 hangs up after the first clarification, though her card allows one |
| Behavioural fidelity | The caller did what the card requires and nothing it forbids; facts held constant | A required behaviour is missing, a fact changed, or an interruption fired without its content trigger | P02 never claims "already paid"; P12 states her salary day as the 5th |
| Speech fidelity | The caller's audio and language match the card | Language mix off card, unintelligible TTS, audio missing the card's codec chain | P12 answers every clause in English; caller audio delivered at 16 kHz with no codec chain |
| Interruption trigger (Tarazu addition) | Every caller interruption follows its card's content trigger | An interruption with no trigger in the preceding agent text | A barge-in 40 s into a P02 call, before the amount was spoken |
Validators never look at whether Asha passed; one that reads the agent's outcome can reject the agent's failures and inflate the pass rate.
Test the validators too: run them on a few dozen hand-labelled simulated calls and count false rejections. A validator that rejects good calls inflates regeneration cost and quietly biases the suite toward calls the simulator finds easy.
Run the cheapest, most structural check first and let the first failure decide the reason. Cap regeneration at two retries; a scenario that fails three times is dropped from that night's score and listed by id. Dropping it silently shrinks the denominator to scenarios the simulator handles well.
Worked example: 20 runs of P12 (synthetic Tarazu data). One night, P12's card runs 20 times with different seeds through the audio path. The validators reject 3 runs on the first attempt:
- one truncated call (valid call end),
- one where the simulated Neha answered every clause in English (speech fidelity, language mix off card),
- one where she gave the 9th with no salary condition (behavioural fidelity, required utterance missing).
All three pass validation on the retry. Of the 20 valid runs, 12 meet the gold end state: a PTP on the 8th or later, confirmed after a closed read-back, with at most one clarification. Per-trial pass rate 12/20 = 0.60, Wilson 95% interval 0.39 to 0.78.
The rejected originals matter. The English-only run and the run without the salary condition both passed, because neither contained the clause-level switch that breaks language ID; the truncated run failed. Their replacements went 1 pass and 2 fails, since two of them did switch mid-clause and the agent looped. Without validators the count is 13/20 = 0.65.
Carry that into pass^k at k = 4, the chance that all four trials of the scenario succeed. With the unbiased estimator, validated runs give C(12,4)/C(20,4) = 495/4,845 = 0.102; unvalidated runs give C(13,4)/C(20,4) = 715/4,845 = 0.148. The naive power 0.60^4 = 0.130 overstates even the validated figure; the estimators are in reliability statistics. Two drifted simulated calls moved a reliability number by 4.5 points, both in the agent's favour.
The caller-simulator validator recipe runs the same loop on a synthetic night of 120 planned runs: it rejects 25 on the first attempt (20.8%) and needs 152 simulator calls to keep 118, a 26.7% cost overhead.
How do you audit failed simulated calls?
For each failed simulated run, two raters listen and attribute the first critical error to the agent or to the simulator. Report raw agreement, Cohen's kappa, and the simulator's share of failures with a Wilson interval; the simulator share is how much of your failure count is not about the agent.
Validators catch drift you anticipated; the audit catches the rest. The unit is the first critical error, the earliest event after which the call could not reach its gold end state. If that event is the caller's (an invented date, an interruption with nothing to say), the failure belongs to the simulator.
A published reference: in tau-Voice, two raters audited 91 failed runs from two selected cohorts, agreed on 84% of them (76 of 91), and found that the simulated user caused the first critical error in 21% of one cohort's failures and 10% of the other's 2. Two cohorts, English only; keep the order of magnitude, one failure in ten to one in five, as your prior.
Report kappa next to agreement. Most failures are agent failures, so two raters who mostly say "agent" agree often by chance, and raw agreement overstates how much they agree beyond it 9. In the recipe's synthetic audit of 42 failed runs, agreement is 0.810 but kappa is 0.475, and the simulator share after adjudication is 7/42 = 16.7% (Wilson 95% interval 8.3% to 30.6%). All seven were timer-based interruptions. The audit's output is the next validator to write, here the interruption-trigger check.
Tarazu audits 50 failed simulated runs per release with two raters. At Rs 62.50 per 150 s audited call that is about Rs 6,250 a release, if simulated calls run as long as real ones. The coding method is in error analysis on real calls; this audit adds one binary code, agent or simulator.
How do you check sim-vs-real calibration?
Take 30 consented, redacted real calls, build a simulated twin of each, run both through the same checks, and compare pass rates check by check with paired tests. The mean overestimate is the correction to apply to simulated pass rates until the next calibration.
Validators check the simulator against its card; calibration checks the card against reality. A twin is a simulated call built from a real call's persona, facts, language mix and audio condition. Run the same checks on both: task success from the CRM diff (task success and outbound call outcomes owns the scorer), grievance details stated (C06), no clarification loop, no talk-over, amount stated correctly.
The pairs share a scenario, so treat them as paired data. McNemar's exact test uses only the discordant pairs: twins that passed in simulation and failed for real, and the reverse 10. The mean overestimate gets a paired bootstrap interval. Checking two independent Wilson intervals for overlap throws the pairing away.
In the recipe's synthetic set, the simulated pass rate is higher on every check and the mean overestimate across five checks is 8.7 points (paired bootstrap 95% interval 2.0 to 16.0). Task success shows the largest gap, 0.83 simulated against 0.63 real, and with 30 twins even that gives McNemar p = 0.070. Thirty twins show that the simulator flatters the agent; they cannot price each check.
Tarazu runs it quarterly, per language, as calibration, not a gate. Two readings change behaviour:
- A consistent overestimate on one check means the cards are too easy on that dimension. Fix the card before raising the agent's bar.
- A simulated pass rate below reality is also a finding. An over-harsh card wastes time on failures no borrower causes.
Use only calls consented for this use, redacted, and handled per safety, privacy and compliance checks.
Which scenarios run in audio, and how often?
Text scenarios run on every commit because they are cheap and catch logic and policy regressions. Audio runs nightly and before every release, because only audio exercises STT, the turn detector, barge-in and TTS. At Tarazu's prices, the commit suite in audio would cost more than the whole eval budget.
Tarazu's cadence policy: 600 text scenarios at k = 2 on every commit, 240 audio scenarios at k = 4 nightly (20 per gated segment, 12 gated segments), and the audio suite again before release. LiveKit's simulation docs recommend the same split, "Run text simulations on every commit, and save audio runs for a nightly or pre-release pass" 7, and its testing docs say to reserve audio runs for turn-taking and speech-specific issues 11. That is vendor advice, not evidence; the cost arithmetic below is the evidence.
Using Tarazu's invented cost model (Rs 0.50 per text scenario run, Rs 6 per audio call, about 10 commits a day over 22 working days):
| Run | Volume | Per run | Monthly |
|---|
| Text on every commit | 600 x 2 = 1,200 runs | Rs 600 | Rs 1,32,000 (220 commits) |
| Audio nightly | 240 x 4 = 960 calls | Rs 5,760 | Rs 1,72,800 (30 nights) |
| Commit suite in audio instead (not done) | 1,200 calls | Rs 7,200 | Rs 15,84,000 (220 commits) |
The last row alone is over the Rs 10 lakh monthly cap. The load and cost chapter prices the whole plan.
Text mode hands the agent the caller's words, so it tests tool order, policy checks such as C04 and C08, read-back logic and the clarification-loop rule, and is blind to T08, T02 and T05. Audio is required wherever the failure starts before the transcript: code-switching (P12), noise and barge-in (P02), hesitation pauses (T05), hard-of-hearing callers (P13) and every machine path. The rule: if the failure code is STT., TURN., LAT., OUT.AMD or ENT. on the spoken side, the regression case runs in audio.
The nightly 240 follow the gated segments, with fixed slots for incident regressions. Which grid cells to fill follows dimensional sampling: every pair of factor levels once, plus the three-way cells behind past incidents. Synthetic vs human data covers when synthetic examples help at all.
Which outbound paths need their own harness?
Voicemail, IVR and operator announcements, transfers to a human, and family members who answer. The vendor simulators read for this page are built around an inbound caller, so an outbound agent's first seconds, AMD, voicemail messages and transfers go untested unless you build a callee harness.
Asha dials out. The callee says "hello" at once, or a carrier plays an announcement, or voicemail greets for 9 seconds before the beep, and AMD decides by 2.5 s. The simulator has to be the callee.
LiveKit's docs say phone tests are "Inbound calls only": the simulated user is always the caller, mocks do not apply and tool calls are not evaluated. An outbound agent is simulated by skipping the dial step, with the simulated user already in the room. In audio mode that user speaks first only after about 10 seconds of silence, so it cannot answer the moment the call connects; it plays no voicemail greetings or phone menus, and transfers to a person are not simulated 7. The agent's logic gets tested; the outbound phone path does not. Coval runs outbound tests by POSTing to your trigger endpoint so your own system dials its simulated customer 12. That is closer to the real path, and you still supply the Indian machine openings.
Tarazu's callee harness is a SIP endpoint the production dialer calls like any number, with one mode per path:
| Path | Personas and flows | What the harness plays | What the checks score |
|---|
| Human answers at once | P01, P12 and most borrowers | "Hello?" as soon as the call connects | AMD = human by 2.5 s; opener has AI disclosure and recording notice (C03) |
| Voicemail | P08, F07 | English and Hindi greeting, beep after 9 s | Neutral message with the grievance officer's name and phone; no amount, product, DPD or "overdue" (C04, C06); mark_contact(voicemail) |
| Operator announcement or IVR | P09, F08 | Recorded Indian carrier announcements; "press 1" menus | Hang up within 4 s; mark_contact(ivr); no opener |
| Family member | P05, F05 | "Woh office gaye hain, kya kaam hai?" | No dues disclosed; mark_contact(third_party) |
| Transfer | P04, P07, P11 | A fake human queue that answers and records the handover | transfer_to_human lands on the right queue with the reason; no PTP request after hardship (C09) |
The machine modes need real Indian openings, not US-style greetings: T07 happened because AMD tuned on US voicemail labelled an operator announcement as human. Record or license them and version them with the gold set. The AMD confusion matrix and its costs belong to task success and outbound call outcomes; the harness puts the openings on the line.
What gate would have caught it?
An audio suite of code-switching personas on the production path, scored with the deterministic clarification-loop check and pass^k over seeds, and a rule that a production failure cut into a scenario must fail on the unfixed agent before the fix ships. T08 fails all three.
gate: caller-simulation
suite:
scenarios: P12 card plus high code-switch density variants in every Hinglish gated segment
path: production audio (codec chain from the card, SIP, production media stream)
k: 4 seeds per scenario, nightly and before release
validators: [valid_call_end, behavioural_fidelity, speech_fidelity, interruption_trigger]
regeneration: up to 2 retries; still invalid -> dropped and listed by id
block_if:
- any run with more than two clarifications on one slot (LOOP.CLARIFY) # deterministic
- pass^4 for a code-switch scenario below the signed bar # chapter 13 estimator
- candidate worse than incumbent on the same scenarios and seeds # McNemar or paired bootstrap
regression_cases:
- T08 cut as a scenario; must FAIL on asha-3.2.1 before the fix is accepted
report_always: [first_attempt_rejection_rate, rejection_reasons, dropped_ids,
simulator_share_from_last_audit, last_sim_vs_real_overestimate]
Interval rule: a per-trial pass rate is a single proportion, so Wilson; candidate against incumbent on the same scenarios and seeds is paired, so McNemar or a paired bootstrap. Intervals for pass^k and seed counts are in reliability statistics; pass^k was introduced in tau-bench as the chance that all k trials of a task succeed 13.
Teams skip the fail-first rule. If the T08 scenario passes on the agent that produced T08, the simulator cannot reproduce the failure: its caller is not switching mid-clause, or its audio skips the codec chain, or its STT sees clean speech. Per-turn and per-session agent scoring reuses the multi-turn conversation recipe, and the tau-bench recipe shows the same closed loop in text.
What goes in the decision memo?
Tarazu's memo builds its own outbound callee harness, scenario cards and validators, keeps text scenarios on a cheap framework, and sets the validator thresholds that pause a scenario family. The evidence section carries the recipe's rejection rate, simulator share and sim-vs-real overestimate, each with its interval.
Decision. Build Tarazu's own outbound callee harness, scenario cards and validators; keep text scenarios on a framework or vendor that runs them cheaply on every commit. Validator thresholds: two retries, then drop and list by id; a scenario family whose first-attempt rejection rate doubles between releases is paused until its card or caller is fixed.
Context. T08: a code-switching borrower looped on three clarifications after STT locked to English; the regression suite had no caller who switched mid-clause on the audio path.
Options considered. (1) Buy a vendor simulator for audio: documented gaps on outbound dialing, voicemail, IVR, transfers, mixed Hindi-English and content-triggered interruptions. (2) Build everything: duplicates text tooling. (3) Build the audio path and validators, reuse text tooling: chosen.
Evidence. Recipe synthetic run, seed 20261120: 20.8% first-attempt rejection; simulator share 7/42 (Wilson 8.3% to 30.6%); sim-vs-real mean overestimate +8.7 points (paired bootstrap 2.0 to 16.0) on 30 twins. P12 worked example: validated pass^4 0.102 against 0.148 unvalidated. Gold set gold-v7.
Owner and cadence. QA lead owns cards and validators; eval platform staff engineer owns the gate. Nightly audio, quarterly calibration, two-rater audit of 50 failed runs per release.
Tripwires. First-attempt rejection rate rises between releases; simulator share's lower bound rises; a cut incident passes before its fix; a new check overestimates in calibration.
Dated facts. LiveKit, Coval and Cekura docs read 2026-11-20; tau-Voice March 2026; EVA-Bench v3 September 2026.
What to do this week
Start with fail-first: cut your recent incidents into scenario cards and check that each one fails on the agent that produced it. A simulator that cannot reproduce your own incidents is not ready to gate anything.
- Cut your last three production incidents into scenario cards and run each against the agent version that produced it. Any that pass expose a simulator gap.
- List every simulator setting you rely on (accent, interruptions, language) and check in the docs whether it is behaviour or a label.
- Add a valid-call-end and a language-mix validator to your audio runs, log the first-attempt rejection rate, and list dropped runs.
- Pick 30 consented, redacted real calls, build their twins, and compute the per-check gap with McNemar.
- Have two people attribute the first critical error on 20 failed simulated runs, and report agreement and kappa together.
NOTE
What will date on this page. LiveKit's simulation and testing docs, Coval's personas and outbound pages and Cekura's personality and accent-testing pages carry no dates and were read 2026-11-20; the quoted limits are the most likely facts to change. tau-Voice figures are from v1 of the paper (March 2026); later leaderboard runs use newer versions and are not comparable. VoiceBench figures are from v3 (December 2024). EVA-Bench figures are from v3 (September 2026). The 312-call incident is from a June 2026 post. Recipe numbers are tied to seed 20261120.