Valeria Cruz (persona P12, trace T08) told Hazel "si, pero my paycheck comes el viernes" [yes, but my paycheck comes on Friday], Hazel's speech-to-text (STT) decided after her first word that the call was in English, and Hazel asked for the payment date three times before Valeria hung up at 71 seconds. Every scenario in the regression suite had passed the night before, and not one of them switched language inside a clause on the phone audio path.
What happened, and why did the regression suite pass?
STT language identification locked onto English after "si", the Spanish words became English-sounding noise, and Hazel looped on clarifications. The suite passed because none of its callers switched language every clause through the real 8 kHz path, so the failing stage never ran.
T08, 2026-11-16 (a Monday, so "el viernes" [on Friday] is 2026-11-20), persona P12, flow F01:
t_ms ch event / text
caller REF: "si, pero my paycheck comes el viernes, so after that I can pay, like the twenty-third"
gloss: "yes, but my paycheck comes on Friday, so after that I can pay, like the twenty-third"
caller STT (language locked to en after the first word): "see pedro my paycheck comes el bee air ness so after that I can pay like the twenty third"
agent TTS: "Sorry, could you tell me the date you can pay?"
caller REF: "el veintitres, the twenty-third, after payday"
gloss: "the twenty-third, the twenty-third, after payday"
caller STT: "el bent he tres the twenty third after payday"
agent TTS: "Just to confirm, is it the third or the twenty-third?"
caller REF: "ay, no. Twenty-three. Two, three." (third clarification; hangs up at 71 s)
gloss: "oh, no. Twenty-three. Two, three."
Diagnosis: a language-identification failure in STT (STT.LID) that became a clarification loop (LOOP.CLARIFY), on a caller pattern the test set did not contain.
T08 has a published twin. An engineer wrote up a regression suite of 312 recorded production calls that all passed, followed by a 3am page when a caller switched between English and another language mid-sentence: STT mis-segmented the audio and the agent looped on clarifications. The pattern had appeared zero times in the 312 calls 1.
A simulated caller pays for itself only when it produces the pattern that hurt you, on the path where it hurt you. It is also a second model inside the eval, and every mistake it makes is booked against the agent unless you measure it separately.
Why is replaying recorded calls not enough?
A recorded call has fixed caller turns. The moment the candidate agent says something new, the recording answers a question that was never asked, nothing reacts to the new reply, and the suite can only contain behavior that has already happened in production.
Replays still serve the stages before Hazel decides anything (STT on real phone audio, the turn detector on real pauses, AMD on real openings), which is how the entity error rate and turn-taking chapters use them.
They fail as soon as the dialogue branches:
- The caller's turns are frozen. Suppose a new prompt asks a closed read-back ("So that's November twenty-third. Is that right?") where the old one asked an open question. The replay answers the yes/no prompt with a recorded date, and the run measures how Hazel copes with a non-answer, not the change you shipped.
- The agent's new words reach no one. If the candidate says something confusing or out of policy, the recording does not react. Sofia Ramirez's "ya lo pague" [I already paid it] on T02 is a reaction to the amount being read out; a replay has no idea when in the new audio the amount is spoken.
- Only behavior you have already seen. A rare caller habit must show up in last month's sample before a replay can test it, and a campaign that has not launched has no recordings.
What fixes all three is a caller that listens: a model that holds a persona and a goal, hears what Hazel actually said, and answers that. That closes the loop, and makes the caller something that can be wrong.
What does a synthetic caller need?
Two things: a scenario card that fixes who the caller is, what they know, how they sound, when they interrupt and which end state counts as a pass; and a delivery path that brings the caller's audio to the agent the way a phone call does. Without that path, audio conditions are only labels.
Every persona in gold-v7 has a card with the same fields: persona text for the simulator (in English or Spanish), audio conditions (SNR, codec chain, loss profile), an interruption script triggered by content rather than by a timer, the gold CRM end state, required caller utterances, forbidden agent utterances, the language mix and a code-switch density label (low, medium, high). The language mix comes from the account preference and the call itself, never from a surname; in a 2022 Pew survey published in 2023, 63% of US Latinos said they use Spanglish at least sometimes 2. P12's card, shortened:
card: P12-F01
persona: >
Valeria Cruz, PIL borrower on PIL-digital-nov. Switches between Spanish
and English every clause. Her paycheck comes on Friday; she will pay
after that, around the twenty-third. Accepts one clarification politely;
on a second, repeats the date in both languages; on a third, hangs up.
facts: {call_date: 2026-11-16, payday: 2026-11-20, preferred_pay_day: 2026-11-23,
identity: dob from accounts}
language_mix: spanglish
code_switch_density: high
audio:
codec_chain: [amr-nb, g711-pcmu]
condition: one cell of the telephony condition matrix (carrier, codec chain, loss, SNR)
interruptions: [] # P12 tests language, not barge-in
required_caller_utterances:
- a clause-level switch inside the date answer ("pero my paycheck comes el viernes")
# gloss: "but my paycheck comes on Friday"
gold_end_state:
ptps: [{ptp_date: ">= 2026-11-21", readback_confirmed: true}]
clarifications_on_date_slot: "<= 1"
forbidden_agent_utterances: [any dues before verify_identity]
Interruptions are where simulators are usually weakest. P02 is Spanish-dominant, so her card's persona text is written in Spanish, and her interruption fires on what Hazel says, not on the clock:
card: P02-F02 (excerpt)
persona_language: es
interruptions:
- when: agent TTS text contains the past-due amount ("ciento treinta dolares")
# gloss: "one hundred thirty dollars"
say: "oiga, ya lo pague, el tres, con mi tarjeta"
# gloss: "listen, I already paid it, on the third, with my card"
Three properties of the caller count for more than how natural its voice sounds:
- It has a goal and keeps state. It holds its facts fixed (P02's payment on 3 November, P12's Friday paycheck) and ends the call when its persona would.
- It reacts to content. It cuts in when the amount is spoken and hangs up on the third clarification.
- Its audio travels the production path. Synthesized speech in an en-US or es-US voice, or consented recorded clips, is degraded through the card's codec chain, loss and noise, then sent over SIP into the same media stream Hazel receives on a live call; testing on telephony audio builds that chain. A simulator that hands Hazel text, or clean 24 kHz audio over a WebSocket, bypasses the stage that broke on T08.
The STT language setting belongs on the card as well, because it changes what a code-switched caller sounds like to the agent. On the Fisher Spanish-English telephone test, forcing whisper-large-v3 to decode as English gave 42.58% WER, against 29.43% with automatic language detection 3. The normalization and code-switching chapter owns that measurement; the simulator runs each Spanglish card under the production setting.
Simulator fidelity is how closely the simulated caller's behavior and audio match its card and the real borrowers behind it. The next four sections measure it.
flowchart LR
A[Scenario store: gold-v7 cards] --> B[Persona caller: LLM + facts + interruption script]
B --> C[Speech: en-US or es-US TTS, or consented clips]
C --> D[Impairments: codec chain, loss, noise]
D --> E[SIP trunk to production media stream]
E --> F[Hazel: VAD, STT, turn detector, LLM, tools, TTS]
F --> G[CRM sandbox]
F --> H[Agent audio back to the caller]
H --> B
G --> I[Validators: call end, behavior, speech, trigger]
I -->|reject| B
I -->|accept| J[Checks and results store]
The return leg matters most: the caller hears Hazel's audio, transcribes it and responds. A caller that reads her text cannot be confused by her audio the way Valeria or Sofia can.
Where do simulators flatter the agent?
Nearly always in one direction: simulated callers are more patient, more consistent and easier to understand than real borrowers, and their accents and interruptions are often settings on a form rather than behavior on the line.
Patience, perfect memory and a direct feed of the agent's words. The tau-Voice authors (278 English tasks; voice agents at 31-51% in clean audio against 85% for a text model) say their simulated user "is more patient than real users, with perfect memory and instantaneous tool calls". They also note that it reads the agent's transcript instead of transcribing the agent's speech, and that accent results from TTS personas are indicative, not definitive 4.
Synthetic speech is easier to recognize. In VoiceBench, every one of six voice assistants scored higher on synthetic speech than on real recorded speech for the same CommonEval instructions: 4.04 real against 4.22 synthetic for a Whisper-plus-LLM pipeline, and 2.15 against 3.21 for VITA. On the low-resource English accents in its 11-accent set, real accented speech dropped further than synthetic accents did, which the authors put down to TTS accents sounding closer to standard English 5. Keep the direction and drop the size: no telephone audio, no Spanish, no matched accent pairs. The direction is enough reason not to let a stock TTS accent stand in for Luis Herrera (P07), whose English is Spanish-accented, or for Ramon Diaz (P13), who is 64 and hard of hearing.
CAUTION
Vendor-reported. Treat it as a hypothesis to test on your own calls. The vendor behaviors below come from each vendor's own documentation, read on 2026-11-20; none of these pages carries a date.
An accent is a voice ID, not an instruction. In Cekura the simulated caller's accent comes only from its voice ID. The accent shown on a personality is a read-only label derived from that voice, and the docs say "Evaluator instructions cannot override a personality setting". Asking for a stronger accent in the instructions changes nothing, and the run still reports a result 6. Its Transcription Accuracy metric in simulation scores the testing agent's speech-to-text, not the agent under test 7. Our reading: before trusting any simulation metric, find out which side of the call it measures.
Interruptions on a clock. Coval's persona settings schedule proactive interruptions on a timer. None, Low, Medium or High means never, or roughly every 90, 45 or 30 seconds, regardless of what the agent is saying; a caller who interrupts when the amount is read has to be scripted by hand 8. A timed interruption exercises the barge-in detector. It does not test recovery, and T02 was a recovery failure.
Mixed speech rendered as one language. LiveKit's simulation docs say that mixed-language speech runs in the non-English language 9. A Spanglish caller rendered as Spanish never switches inside a clause, so no number of seeds in that suite reproduces T08. The same docs' outbound limits are covered in the outbound section below.
Clean pipelines smooth out disfluency. ServiceNow's EVA-Bench authors report that their clean STT-LLM-TTS user simulator under-represents real callers' disfluencies (v1 adds hesitations and emotional variation) and does not generate interruptions systematically by default 10. The 720 ms pause before "fifth" on T05 ("next month on the... fifth") is the kind of hesitation a clean simulator leaves out.
Each gap pushes the same way, toward a better-looking agent. Read a simulated pass rate as an upper estimate until the sim-vs-real check says how far to discount it.
How do you validate the simulator itself?
Before you score Hazel on a simulated call, run automated validators that read only the caller's side, reject any call where the simulated caller broke its card, and regenerate it with a fresh seed. The validators judge the simulator; they never look at the agent's result.
EVA-Bench passes every simulated call through validators for a valid call end, user behavioral fidelity and user speech fidelity, and regenerates the calls that fail. Across four systems, 12% of trials were regenerated for user simulator error, nearly all of it behavioral drift 10. Steelyard runs the same three, fitted to its cards, and adds a fourth:
| Validator | What it checks | Rejects when | Steelyard example |
|---|
| Valid call end | The call reached a terminal event the flow allows | No terminal event, or the caller hung up early for a reason its card does not give | P12 hangs up after the first clarification, though her card allows one |
| Behavioral fidelity | The caller did what the card requires and nothing it forbids; facts held constant | A required behavior is missing, a fact changed, or an interruption fired without its content trigger | P02 never says "ya lo pague" [I already paid it]; P12 gives her payday as Monday |
| Speech fidelity | The caller's audio and language match the card | Language mix off card, unintelligible TTS, audio missing the card's codec chain | P12 answers every clause in English; caller audio arrives at 16 kHz with no codec chain |
| Interruption trigger (Steelyard addition) | Every caller interruption follows its card's content trigger | An interruption with no trigger in the preceding agent text | A barge-in 40 s into a P02 call, before "ciento treinta dolares" [one hundred thirty dollars] was spoken |
A validator that reads Hazel's outcome can learn to reject her failures, which inflates the pass rate. Keep the agent's result out of every validator's inputs.
Test the validators too: run them on a few dozen hand-labeled simulated calls and count false rejections, each of which adds regeneration cost and tilts the suite toward calls the simulator finds easy.
Order the validators from cheapest and most structural to most expensive, and let the first failure set the rejection reason. Allow two retries. A scenario that fails validation three times is dropped from that night's score and listed by id, because dropping it silently shrinks the denominator to the scenarios the simulator handles well.
Worked example: 20 runs of P12 (synthetic Steelyard data). One night P12's card runs 20 times with different seeds through the audio path. The validators reject 3 runs on the first attempt:
- one call that ended without a terminal event (valid call end),
- one where the simulated Valeria spoke English in every clause (speech fidelity, language mix off card),
- one where she offered the twenty-third with no mention of the Friday paycheck (behavioral fidelity, required utterance missing).
All three pass on the retry. Of the 20 valid runs, 12 reach the gold end state: a PTP on 21 November or later, written after a closed read-back and an explicit yes, with no more than one clarification on the date. The per-trial pass rate is 12/20 = 0.60, Wilson 95% interval 0.39 to 0.78.
The rejected runs matter. The English-only run and the run without the paycheck both passed, because neither contained the clause-level switch that trips language identification; the truncated run failed. Their replacements went 1 pass and 2 fails, because two of them did switch inside a clause and Hazel looped. Without validators the count would have been 13/20 = 0.65.
Now carry both counts into pass^k at k = 4, the probability that all four trials of the scenario succeed. With the unbiased estimator, the validated runs give C(12,4)/C(20,4) = 495/4,845 = 0.102, and the unvalidated runs give C(13,4)/C(20,4) = 715/4,845 = 0.148. The naive power 0.60^4 = 0.130 overstates even the validated figure; reliability statistics owns the estimators. Two drifted simulated calls moved a reliability number by 4.5 points, both times in Hazel's favor.
The caller-simulator validator recipe runs the same loop on a synthetic night of 120 planned runs. It rejects 25 on the first attempt (20.8%) and places 152 simulator calls to keep 118, a 26.7% cost overhead.
Who caused a failed simulated call, the agent or the simulator?
Take each failed simulated run, have two raters listen, and ask each to attribute the first critical error to the agent or to the simulator. Report raw agreement, Cohen's kappa, and the simulator's share of failures with a Wilson interval. That share is the part of your failure count that says nothing about Hazel.
Validators catch the drift you predicted; the audit catches the rest. Its unit is the first critical error: the earliest event after which the call could no longer reach its gold end state. When that event belongs to the caller, such as a payment date the card never gave or an interruption with nothing to interrupt, the failure is the simulator's.
For a published reference point, the tau-Voice authors had two raters audit 91 failed runs from two selected cohorts. The raters agreed on 84% (76 of 91), and the simulated user caused the first critical error in 21% of one cohort's failures and 10% of the other's 4. That is two cohorts in English only, so keep the order of magnitude, between one failure in ten and one in five, as your starting assumption.
Put kappa beside agreement. Most failed runs are agent failures, so two raters who mostly answer "agent" will agree often by chance alone, and raw agreement overstates how much they agree beyond chance 11. In the recipe's synthetic audit of 42 failed runs, agreement is 0.810 while kappa is 0.475, and after adjudication the simulator's share is 7/42 = 16.7% (Wilson 95% interval 8.3% to 30.6%). All seven were interruptions on a timer. The audit's real output is the next validator to write, in this case the interruption-trigger check.
Steelyard audits 50 failed simulated runs per release with two raters, bilingual English-Spanish for the Spanish and Spanglish runs. At $3.125 per audited 150 s call, two raters on 50 runs come to about $312.50 a release, assuming simulated calls run as long as real ones. The coding method lives in error analysis on real calls; this audit adds a single binary code, agent or simulator.
How do you check sim-vs-real calibration?
Pick 30 consented, redacted real calls, build a simulated twin of each, score both with the same checks, and compare pass rates check by check with paired tests. The mean overestimate becomes the discount you apply to simulated pass rates until the next calibration.
Validators hold the simulator to its card. Calibration holds the card to reality. A twin is a simulated call built from one real call's persona, facts, language mix and audio condition. Both versions go through the same checks: task success from the CRM diff (task success and outbound call outcomes owns the scorer), Customer Advocate details stated (C06), no clarification loop, no talk-over, and the amount stated correctly.
Each real call and its twin share a scenario, so they are paired data. McNemar's exact test uses only the discordant pairs, the twins that passed in simulation and failed for real and the reverse 12. The mean overestimate gets a paired bootstrap interval. Comparing two independent Wilson intervals throws the pairing away.
In the recipe's synthetic set the simulated pass rate is higher on every check, and the mean overestimate across five checks is 8.7 points (paired bootstrap 95% interval 2.0 to 16.0). Task success has the widest gap, 0.83 simulated against 0.63 real, and with 30 twins even that gives McNemar p = 0.070. Thirty twins are enough to show that the simulator flatters the agent; they are not enough to put a price on each check.
Steelyard runs calibration quarterly and per language group (English, Spanish-dominant, Spanglish), as calibration rather than as a gate. Two outcomes change what the team does next:
- A steady overestimate on one check means the cards are too easy on that dimension. Fix the cards before raising the bar on Hazel.
- A simulated pass rate below the real one is a finding as well. An over-harsh card spends review time on failures no borrower produces.
Use only calls consented for this purpose and redacted, handled as safety, privacy and compliance checks describes.
Which scenarios run in audio, and how often?
Text scenarios run on every commit, because they are cheap and catch logic and policy regressions. Audio runs nightly and before every release, because only audio exercises STT, the turn detector, barge-in and TTS. At Steelyard's prices, the per-commit suite in audio would cost more than the monthly eval cap.
Steelyard's cadence policy: 600 text scenarios at k = 2 on every commit, 240 audio scenarios at k = 4 every night (20 per gated segment across 12 gated segments), and the audio suite again before each release. LiveKit's testing docs recommend the same split ("Run text simulations on every commit, and save audio runs for a nightly or pre-release pass") and advise reserving audio runs for turn-taking and speech-specific issues 13. That is a vendor's advice, not evidence. The evidence is the arithmetic.
With Steelyard's invented cost model ($0.025 per text scenario run, $0.30 per simulated audio call, about 10 commits a day over 22 working days):
| Run | Volume | Per run | Monthly |
|---|
| Text on every commit | 600 x 2 = 1,200 runs | $30.00 | $6,600.00 (220 commits) |
| Audio nightly | 240 x 4 = 960 calls | $288.00 | $8,640.00 (30 nights) |
| Commit suite in audio instead (not done) | 1,200 calls | $360.00 | $79,200.00 (220 commits) |
The last row alone is over the $50,000 monthly cap, and the whole baseline plan is about $45,673.03. The load and cost chapter prices the full plan.
Text mode hands Hazel the caller's words, so it tests tool order, policy checks such as C04 and C08, read-back logic and the clarification-loop rule. It cannot see T08, T02 or T05. Audio is required wherever the failure starts before the transcript exists: code-switching (P12), noise and barge-in (P02), hesitation pauses (T05), slow or hard-of-hearing callers (P13) and every machine path. The rule Steelyard applies: if the failure code is STT., TURN., LAT., OUT.AMD, or ENT. on the spoken side, the regression case runs in audio.
The nightly 240 follow the gated segments, with fixed slots for incident regressions. Which cells of the persona, language and audio grid to fill follows dimensional sampling: every pair of factor levels at least once, plus the three-way cells behind past incidents. Synthetic vs human data covers when synthetic examples help at all.
Which outbound paths need their own harness?
Voicemail, call-screening assistants, carrier intercepts and IVRs, family members, and transfers to a human. In the vendor docs we read, the simulated party either dials in or plays no machine openings, so an outbound agent's first seconds, AMD, voicemail and transfers go untested without a callee harness.
Hazel dials out. The person who answers says "Hello?" at once, or a call-screening assistant asks her to state her name and reason, or a carrier plays "the number you have dialed is not in service", or a voicemail greeting runs 9 seconds before the beep, and AMD has to decide by 2.5 s. The simulator has to play the callee.
LiveKit's docs say phone tests are "Inbound calls only": the simulated user is always the caller, mocks do not apply and tool calls are not evaluated. To simulate an outbound agent you skip the dial step and put the simulated user straight into the room. In audio mode that user speaks first only after about 10 seconds of silence, so it cannot answer the moment the call connects; it plays no voicemail greetings or phone menus, and transfers to a person are not simulated 9. The agent's logic gets tested and the outbound phone path does not. Coval runs outbound tests by POSTing to your trigger endpoint, so your own system dials its simulated customer 14. That is closer to the real path, but the page does not describe voicemail greetings, IVR menus or call-screening prompts, so plan to supply the US machine openings yourself.
The machine paths carry rules too. Steelyard collects its own debts in its own name, so the FDCPA and Regulation F largely do not bind it 15 16; it adopts their conduct rules as company policy, and state law, the TCPA and the Consumer Financial Protection Act's ban on unfair, deceptive or abusive practices still apply. Under that policy a voicemail is a limited-content message: a business name that does not indicate debt collection, a request that the borrower reply, the name of one or more natural persons to contact and a reply number, and nothing outside the listed optional items 16. Unanswered calls and calls that reach voicemail count toward the 7-in-7 limit, while busy and out-of-service calls do not 17, and a telephone conversation about the debt starts a 7-day period with no further calls 18. The harness scores what the dialer logged, not only what Hazel said.
Steelyard's callee harness is a SIP endpoint that the production dialer calls like any other number, with one mode per path:
| Path | Personas and flows | What the harness plays | What the checks score |
|---|
| Human answers at once | P01, P12 and most borrowers | "Hello?" the moment the call connects | AMD = human by 2.5 s; opener leads with the business identity, then AI disclosure and recording notice (C03) |
| Voicemail | P08, F07 | English and Spanish greeting, beep after 9 s | Limited-content message naming "Steelyard", Dana Brooks and the callback number; no amount, product, DPD, "past due", complaint script or SMS (C04); mark_contact(voicemail); attempt counted toward the cap (C11) |
| Call-screening assistant, carrier intercept or IVR | P09, F08 | A screening prompt asking for name and reason; "The number you have dialed is not in service"; "press 1" menus | Hang up within 4 s; mark_contact(ivr); no opener; attempt not logged as a conversation |
| Family member | P05, F05 | "He's at work. This is his brother. What's this about?" | No dues disclosed; mark_contact(third_party) |
| Transfer | P04, P07, P11 | A fake human queue that answers and records the handover | transfer_to_human lands on the right queue with the reason; no PTP request after hardship (C09) |
The machine modes need real US openings, not only voicemail greetings. T07 happened because AMD tuned on voicemail greetings labeled a call-screening assistant as human at 0.71; the attempt was logged as a conversation, so the dialer also started a 7-day no-call period and Steelyard lost a week of contact with that borrower. Record or license screening prompts, intercepts and IVR menus, and version them with the gold set. The AMD confusion matrix and the cost of each error belong to task success and outbound call outcomes; the harness puts the openings on the line.
What gate would have caught it?
An audio suite of Spanglish personas on the production path, scored with the deterministic clarification-loop check and with pass^k over seeds, plus a rule that a production failure cut into a scenario must fail on the unfixed agent before the fix ships. T08 fails all three.
gate: caller-simulation
suite:
scenarios: P12 card plus high code-switch density variants in every Spanglish gated segment
path: production audio (codec chain from the card, SIP, production media stream)
k: 4 seeds per scenario, nightly and before release
validators: [valid_call_end, behavioural_fidelity, speech_fidelity, interruption_trigger]
regeneration: up to 2 retries; still invalid -> dropped and listed by id
block_if:
- any run with more than two clarifications on one slot (LOOP.CLARIFY) # deterministic
- pass^4 for a code-switch scenario below the signed bar # chapter 13 estimator
- candidate worse than incumbent on the same scenarios and seeds # McNemar or paired bootstrap
regression_cases:
- T08 cut as a scenario; must FAIL on hazel-3.2.1 before the fix is accepted
report_always: [first_attempt_rejection_rate, rejection_reasons, dropped_ids,
simulator_share_from_last_audit, last_sim_vs_real_overestimate]
Interval rule: a per-trial pass rate is one binary proportion, so use Wilson; a candidate against the incumbent on the same scenarios and seeds is paired, so use McNemar or a paired bootstrap. Intervals for pass^k and seed counts are in reliability statistics; pass^k comes from tau-bench, where it is the probability that all k trials of a task succeed 19.
The simulator is part of the measuring instrument. A new caller model, TTS voice, impairment profile or card version changes the eval, not Hazel, so record each version in the run record. When one of them changes, rerun the incumbent on the new simulator before you compare a candidate against it; otherwise a simulator upgrade shows up as an agent regression or an agent win.
The fail-first rule is the one teams skip. If the T08 scenario passes on the agent that produced T08, the simulator cannot reproduce the failure: its caller does not switch inside a clause, or its audio skips the codec chain, or its STT hears clean speech. Per-turn and per-session scoring of the agent reuses the multi-turn conversation recipe, and the tau-bench recipe shows the same closed loop in text.
What goes in the decision memo?
Steelyard's memo builds its own outbound callee harness, scenario cards and validators, keeps text scenarios on a cheap framework, and sets the validator thresholds that pause a scenario family. Its evidence carries the recipe's rejection rate, simulator share and sim-vs-real overestimate, each with an interval.
Decision. Build Steelyard's outbound callee harness, Spanglish and Spanish scenario cards and caller validators in house; run text scenarios on whichever framework or vendor runs them cheaply on every commit. Validator thresholds: two retries, then drop and list by id; a scenario family whose first-attempt rejection rate doubles between releases is paused until its card or caller model is fixed.
Context. T08: a Spanglish borrower hung up after three clarifications once STT locked to English; no caller in the regression suite switched language inside a clause on the audio path.
Options considered. (1) Buy a vendor simulator for audio: its documented gaps cover outbound dialing, voicemail, IVR and call screening, transfers, mixed Spanish-English speech and content-triggered interruptions. (2) Build everything: duplicates text tooling that already works. (3) Build the audio path and validators, reuse text tooling: chosen.
Evidence. Recipe synthetic run, seed 20261120: 20.8% first-attempt rejection; simulator share 7/42 (Wilson 8.3% to 30.6%); sim-vs-real mean overestimate +8.7 points (paired bootstrap 2.0 to 16.0) on 30 twins. P12 worked example: pass^4 0.102 validated against 0.148 unvalidated. Gold set gold-v7.
Owner and cadence. QA lead owns cards and validators; the eval platform staff engineer owns the gate. Audio nightly, calibration quarterly per language group, two-rater audit of 50 failed runs per release.
Tripwires. First-attempt rejection rate rises between releases; the lower bound of the simulator share rises; a cut incident passes before its fix; a new check overestimates in calibration.
Dated facts. LiveKit, Coval and Cekura docs read 2026-11-20; tau-Voice March 2026; EVA-Bench v3 September 2026; Regulation F text read 2026-11-20.
What to do this week
Begin with fail-first: turn recent incidents into scenario cards and confirm each one fails on the agent version that produced it. A simulator that cannot reproduce your own incidents is not ready to gate a release.
- Write cards for your last three production incidents and run each against the agent version that caused it. Any card that passes points to a gap in the simulator.
- List every simulator setting you depend on (accent, interruptions, language mix) and check in the vendor docs whether each one is behavior or a label.
- Add a valid-call-end validator and a language-mix validator to your audio runs, log the first-attempt rejection rate, and publish the list of dropped runs.
- Take 30 consented, redacted real calls, at least a few per language group, build their twins, and compute the gap for each check with McNemar.
- Have two people attribute the first critical error on 20 failed simulated runs, and report agreement and kappa side by side.
NOTE
What will date on this page. LiveKit's simulation and testing docs, Coval's personas and outbound pages and Cekura's personality and accent-testing pages are undated and were read 2026-11-20; the quoted limits are the facts most likely to change. tau-Voice figures are from v1 of the paper (March 2026); later leaderboard runs use newer versions and are not comparable. VoiceBench figures are from v3 (December 2024). EVA-Bench figures are from v3 (September 2026). The 312-call incident is from a June 2026 post. The Pew Spanglish figure is from a survey fielded in August 2022 and published in September 2023. The language-forcing WER figures are from Table III of the PIER paper (v2). Regulation F's limited-content message definition and its commentary on which calls count were read on eCFR on 2026-11-20. Recipe numbers are tied to seed 20261120.