A simulated caller is a second model in your eval, and its errors land in your agent's pass rate. One engineer's regression suite of 312 recorded production calls all passed, and then a caller who switched between English and Hindi mid-sentence caused a 3am page: STT mis-segmented the audio and the agent looped on clarifications. The pattern appeared zero times in the 312 calls 1. Steelyard's version is T08. Valeria Cruz (persona P12) said "si, pero my paycheck comes el viernes" [yes, but my paycheck comes on Friday], STT locked onto English, and she hung up after the third clarification. Synthetic callers can produce that pattern on demand. This recipe checks that the synthetic caller behaves like its card before you count what the agent did.
The script and synthetic data for this recipe are also in the ai-evals-course repository on GitHub, in the folder voice-ai/labs/caller-simulator-validator/. That copy is the full file, which also carries the India edition.
What you build
You build one standard-library Python script that does four jobs. It builds a scenario grid that covers every pair of factor levels in 60 scenarios instead of 180. It runs three validators on each simulated call and regenerates the ones that fail. It compares 30 real calls with their simulated twins, check by check. And it scores two raters who decide whether the agent or the simulator caused each failure. The data is seeded and synthetic, so you can run it before you own a simulator, then swap in your own transcripts.
The validator design follows ServiceNow's EVA-Bench. Every simulated call goes through automated validators for a valid call end, user behavioral fidelity and user speech fidelity, and any call that fails is regenerated. Across four systems, 12% of trials were regenerated because of user simulator error, nearly all of it behavioral drift 2. This script works on transcripts, so its third validator checks the caller's turns (facts and language mix) where EVA checks speech.
| Part | What it does | What it prints |
|---|
| 1. Scenario grid | Persona-flow x audio condition x language mix, built as a Latin square | Pair coverage for each pair of factors |
| 2. Validators | Valid call end, behavioral fidelity to the card, caller-turn fidelity; up to 2 regenerations | First-attempt rejection rate, reasons, runs dropped, cost of the retries |
| 3. Sim vs real | 30 consented, redacted real calls paired with simulated twins | Per-check pass rates, the gap, McNemar's exact test, a paired bootstrap on the mean overestimate |
| 4. Attribution audit | Two raters label who caused the first critical error in each failed run | Agreement, Cohen's kappa, simulator share with a Wilson interval |
Step 1: Create a fresh environment
The script uses only the standard library, so the venv just pins the interpreter.
python3.11 -m venv .venv
.venv/bin/python --version
Step 2: Save the script
Save this as caller-simulator-validator.py. It starts with the same Steelyard constants block as the other voice recipes, adds the simulation cost and cadence from Steelyard's eval plan, and generates all of its own data. The persona cards are shortened versions of Steelyard's 13 callee personas. Each card has a defining line for each language mix (Spanglish, Spanish-dominant, English), required and forbidden behaviors, and the facts the caller is supposed to know. Spanish lines are written without accents, and G_GLOSS holds an English gloss for each one; the script refuses to run if a Spanish line has none. The simulate function is a stand-in that writes transcripts from a card and injects one of six kinds of drift at fixed rates. Five of them have a validator. The sixth, an interruption on a timer, has none on purpose.
The script takes --edition global|india (default global). The listing below is the global edition: it leaves out the India edition's block (its constants, persona cards and word lists), which sits between two marker comments in the full file in the course repository. Without that block, --edition india exits with a usage error. Both editions draw from one simulation seed, 20261120, so every count, rate and interval the script prints is the same in both; only names, lines and money labels change.
"""Validate a simulated caller (aievals.co cookbook recipe R7).
Dataset: steelyard-calls (synthetic, seed 20261121). Python 3.11 standard library, no network.
--edition india switches to the India edition's fictional lender, persona cards and
language mixes (its block sits between the INDIA EDITION markers below).
What it does:
1. Builds a pairwise scenario grid over persona-flow x audio condition x language mix.
2. Generates 120 simulated call transcripts (60 scenarios x 2 seeds) with seeded
simulator drift.
3. Runs three validators on every run (valid call end, behavioral fidelity to the
persona card, caller-turn fidelity) and regenerates a rejected run up to 2 times.
4. Compares 30 real calls with their simulated twins, per check (paired).
5. Scores a two-rater attribution audit of the failed simulated runs (agent or
simulator caused the first critical error), with Cohen's kappa.
Run:
python3.11 -m venv .venv
.venv/bin/python caller-simulator-validator.py # global edition
.venv/bin/python caller-simulator-validator.py --edition india # India edition
"""
from __future__ import annotations
import argparse
import itertools
import math
import random
import re
from collections import Counter
# --- GLOBAL constants (WORLD-GLOBAL world-global-v1.1) ---
GLOBAL = {
"lender_line": "Synthetic data.",
"dataset": "steelyard-calls (synthetic, seed 20261121)",
"seed": 20261121,
"products": {"AUTO": "auto loan", "PIL": "personal installment, digital",
"POS": "retail point-of-sale", "SBL": "small business (guarantor)",
"SDL": "credit-builder and small-dollar"},
"calling_window_local": {"default": ("08:00", "21:00"), "SDL": ("09:00", "20:00")},
"campaigns": { # (book, silence budget ms, amount-asked / date-asked budget ms)
"AUTO-early-nov": ("AUTO", 400, 600),
"PIL-digital-nov": ("PIL", 400, 600),
"POS-short-nov": ("POS", 400, 600),
"SBL-guar-nov": ("SBL", 450, 650),
"SDL-biwk-nov": ("SDL", 600, 800),
},
"languages": {"English": 0.64, "Spanish-dominant": 0.18, "Spanglish": 0.14, "other": 0.04},
"customer_advocate": {"name": "Ms Dana Brooks", "email": "advocate@steelyard.example",
"phone": "(520) 555-0199",
"address": "1200 Example Avenue, Tucson, AZ 85701"},
"media": {"sample_rate_hz": 8000, "codec": "G.711 mu-law", "frame_ms": 20, "frame_bytes": 160},
"money_unit": "integer cents",
"usd_1500_forms": ["fifteen hundred", "one thousand five hundred", "1,500", "1500",
"$1,500.00", "mil quinientos", "mil quinientos dolares",
"1.500 [es-ES]"],
"usd_1500_cents": 150000,
"sim_cost": {"audio_call": 0.30, "text_run": 0.025}, # WG20, US dollars
"sim_cadence": {"nightly_audio": (240, 4), "per_commit_text": (600, 2)}, # WG19
"twins_per_quarter": 30, # WG19
}
# --- end GLOBAL constants ---
# One simulation seed for both editions, so every computed line below is identical
# in the global and India runs (the voice-simulation chapters quote them).
SIM_SEED = 20261120
MAX_RETRIES = 2
AUDIO = ["clean-16k", "phone-8k-mulaw", "phone-8k-loss5-snr10"]
# ---------- persona cards (WORLD-GLOBAL WG8, shortened for the simulator) ----------
# key: the caller's defining line per language mix. required / forbidden: markers the
# behavioral validator looks for in caller turns. facts: values the caller-turn
# validator checks. machine: no language check, no human speech allowed.
G_LANGS = ["Spanglish", "Spanish-dominant", "English"] # mixed, other-dominant, English
G_CARDS = {
"P01": dict(name="Ryan Mitchell, AUTO borrower, will pay", machine=False,
key={"Spanglish": "si, le pago, el veinte I can do it",
"Spanish-dominant": "si, le pago el veinte",
"English": "yes I will pay it on the twentieth"},
required=[r"veinte|twentieth"], forbidden=[r"already|wrong|lost my job"],
facts={"date": r"veinte|twentieth"},
drift="no, no, ese monto esta mal"),
"P02": dict(name="Sofia Ramirez, POS borrower, already paid", machine=False,
key={"Spanglish": "pero I already paid, el tres, con la tarjeta in the app",
"Spanish-dominant": "ya lo pague, el tres, con la tarjeta de debito en la app",
"English": "I already paid on the 3rd, with my debit card in the app"},
required=[r"already|ya lo pague"], forbidden=[r"le pago manana|will pay"],
facts={"paid_on": r"el tres|3rd", "confirmation_no": r"\bconfirmacion\b \d{12}"},
content_interrupt=True,
drift="bueno, esta bien, le pago manana"),
"P03": dict(name="Marcus Reed, PIL borrower, disputes the late fee", machine=False,
key={"Spanglish": "ese late fee esta mal, yo pague on time",
"Spanish-dominant": "ese cargo por atraso esta mal, yo no lo voy a pagar",
"English": "this late fee is wrong, I paid on time"},
required=[r"esta mal|wrong"], forbidden=[r"le pago|will pay"],
facts={}, drift="bueno, le pago todo"),
"P04": dict(name="Daniela Torres, AUTO borrower, lost her job", machine=False,
key={"Spanglish": "perdi mi trabajo, ahorita I cannot pay anything",
"Spanish-dominant": "perdi mi trabajo, ahorita no puedo pagar nada",
"English": "I lost my job, I cannot pay anything right now"},
required=[r"perdi mi trabajo|lost my job"], forbidden=[r"le pago|will pay"],
facts={}, drift="si, le pago el cinco"),
"P05": dict(name="Kyle Mitchell, Ryan's adult brother", machine=False,
key={"Spanglish": "esta en el trabajo, soy su hermano, what is this about",
"Spanish-dominant": "esta en el trabajo, yo soy su hermano",
"English": "he is at work right now, I am his brother, what is this about"},
required=[r"trabajo|at work"], forbidden=[r"soy ryan|this is ryan"],
facts={}, drift="si, yo soy Ryan, soy Ryan"),
"P06": dict(name="Wrong number (recycled)", machine=False,
key={"Spanglish": "usted tiene el numero equivocado",
"Spanish-dominant": "usted tiene el numero equivocado, senor",
"English": "you have the wrong number"},
required=[r"numero equivocado|wrong number"], forbidden=[r"soy ryan|this is ryan"],
facts={}, drift="si, yo soy Ryan, soy Ryan"),
"P07": dict(name="Luis Herrera, SBL guarantor", machine=False,
key={"Spanglish": "si, digame, what is this about? mi zip ocho cinco siete uno nueve",
"Spanish-dominant": "si digame de que se trata, zip ocho cinco siete uno nueve",
"English": "yes, what is this about? my zip is eight five seven one nine"},
required=[r"de que se trata|what is this"], forbidden=[r"yo soy el deudor"],
facts={"zip": r"ocho cinco siete uno nueve|eight five seven one nine"},
drift="si, yo soy el deudor"),
"P08": dict(name="Carrier voicemail greeting", machine=True,
key={l: "[voicemail greeting] [beep after 9 s]" for l in G_LANGS},
required=[r"\[beep"], forbidden=[r"^(si|yes|hello)"], facts={},
drift="hello, si digame"),
"P09": dict(name="Call-screening assistant / intercept / IVR", machine=True,
key={l: "[call screening] the person you are calling is using a screening service,"
" say your name and why you are calling" for l in G_LANGS},
required=[r"\[call screening"], forbidden=[r"^(si|yes|hello)"], facts={},
drift="hello, si digame"),
"P10": dict(name="Javier Ortiz, PIL borrower, driving", machine=False,
key={"Spanglish": "voy manejando ahorita, llameme after six",
"Spanish-dominant": "voy manejando, llameme despues de las seis",
"English": "call me after six, I'm driving"},
required=[r"after six|despues de las seis"], forbidden=[],
facts={"callback": r"after six|despues de las seis"},
drift="okay fine, tell me the amount now"),
"P11": dict(name="Derek Wallace, SBL borrower, angry", machine=False,
key={"Spanglish": "voy a poner una queja con la CFPB, me llaman todos los dias",
"Spanish-dominant": "voy a poner una queja con la CFPB, me llaman todos los dias, todos",
"English": "I will complain to the CFPB, you call me every day"},
required=[r"queja|complaint|complain"], forbidden=[],
facts={}, drift="bueno, esta bien, no hay problema"),
"P12": dict(name="Valeria Cruz, PIL borrower, code-switcher", machine=False,
key={"Spanglish": "si, pero my paycheck comes el viernes, so after that I can pay, el veintitres",
"Spanish-dominant": "si pero mi cheque llega el viernes, despues de eso el veintitres",
"English": "yes but my paycheck comes on Friday, so after that, around the twenty third"},
required=[r"paycheck|cheque"], forbidden=[],
facts={"date_after_payday": r"twenty third|veintitres"},
drift="I can pay today"),
"P13": dict(name="Ramon Diaz, SDL borrower, 64", machine=False,
key={"Spanglish": "repitalo por favor, cuanto es the amount? no le escuche, more slowly please",
"Spanish-dominant": "repitalo por favor, cuanto es? no le escuche bien",
"English": "please repeat, how much is it? I did not hear properly"},
required=[r"repitalo|repeat"], forbidden=[],
facts={}, drift="si si, esta bien"),
}
G_TEXT = {
"opener": {"Spanglish": "Hi, this is Steelyard Credit. I'm Hazel, an AI assistant. This call is recorded.",
"Spanish-dominant": "Hola, le llamamos de Steelyard Credit. Soy Hazel, una asistente de"
" inteligencia artificial. Esta llamada se graba.",
"English": "Hi, this is Steelyard Credit. I'm Hazel, an AI assistant. This call is recorded."},
"answer": {"Spanglish": "si, speaking", "Spanish-dominant": "si, digame", "English": "yes, speaking"},
"verify": "Thank you. To verify your identity, can you tell me your ZIP code?",
"ask_ref": "Do you have the confirmation number of the payment?",
"give_ref": "si, confirmacion {ref}",
"fact_forgot": "no se, no me acuerdo",
"lang_swap_to_english": "yes okay I will see, please tell me again",
"lang_swap_to_other": "si, esta bien, repitamelo por favor, no le entiendo",
"hangup": "bueno, ya me voy, bye",
}
# English gloss for every Spanish line above (WG0). The example run prints them, and
# main() checks that no Spanish line is missing one.
G_GLOSS = {
"si, le pago, el veinte I can do it": "yes, I'll pay you, on the twentieth I can do it",
"si, le pago el veinte": "yes, I'll pay you on the twentieth",
"no, no, ese monto esta mal": "no, no, that amount is wrong",
"pero I already paid, el tres, con la tarjeta in the app":
"but I already paid, on the third, with the card in the app",
"ya lo pague, el tres, con la tarjeta de debito en la app":
"I already paid it, on the third, with the debit card in the app",
"bueno, esta bien, le pago manana": "okay, fine, I'll pay you tomorrow",
"ese late fee esta mal, yo pague on time": "that late fee is wrong, I paid on time",
"ese cargo por atraso esta mal, yo no lo voy a pagar":
"that late charge is wrong, I am not going to pay it",
"bueno, le pago todo": "okay, I'll pay you all of it",
"perdi mi trabajo, ahorita I cannot pay anything": "I lost my job, right now I cannot pay anything",
"perdi mi trabajo, ahorita no puedo pagar nada": "I lost my job, right now I can't pay anything",
"si, le pago el cinco": "yes, I'll pay you on the fifth",
"esta en el trabajo, soy su hermano, what is this about":
"he's at work, I'm his brother, what is this about",
"esta en el trabajo, yo soy su hermano": "he's at work, I'm his brother",
"si, yo soy Ryan, soy Ryan": "yes, I'm Ryan, I'm Ryan",
"usted tiene el numero equivocado": "you have the wrong number",
"usted tiene el numero equivocado, senor": "you have the wrong number, sir",
"si, digame, what is this about? mi zip ocho cinco siete uno nueve":
"yes, go ahead, what is this about? my ZIP eight five seven one nine",
"si digame de que se trata, zip ocho cinco siete uno nueve":
"yes, tell me what this is about, ZIP eight five seven one nine",
"si, yo soy el deudor": "yes, I am the borrower",
"hello, si digame": "hello, yes, go ahead",
"voy manejando ahorita, llameme after six": "I'm driving right now, call me after six",
"voy manejando, llameme despues de las seis": "I'm driving, call me after six",
"voy a poner una queja con la CFPB, me llaman todos los dias":
"I'm going to file a complaint with the CFPB, you call me every day",
"voy a poner una queja con la CFPB, me llaman todos los dias, todos":
"I'm going to file a complaint with the CFPB, you call me every day, every one",
"bueno, esta bien, no hay problema": "okay, fine, no problem",
"si, pero my paycheck comes el viernes, so after that I can pay, el veintitres":
"yes, but my paycheck comes on Friday, so after that I can pay, on the twenty-third",
"si pero mi cheque llega el viernes, despues de eso el veintitres":
"yes but my paycheck arrives on Friday, after that the twenty-third",
"repitalo por favor, cuanto es the amount? no le escuche, more slowly please":
"repeat it please, how much is the amount? I didn't hear you, more slowly please",
"repitalo por favor, cuanto es? no le escuche bien": "repeat it please, how much is it? I didn't hear you well",
"si si, esta bien": "yes yes, that's fine",
"Hola, le llamamos de Steelyard Credit. Soy Hazel, una asistente de inteligencia artificial."
" Esta llamada se graba.": "Hello, we are calling from Steelyard Credit. I'm Hazel, an"
" artificial intelligence assistant. This call is recorded.",
"si, speaking": "yes, speaking",
"si, digame": "yes, go ahead",
"si, confirmacion {ref}": "yes, confirmation {ref}",
"no se, no me acuerdo": "I don't know, I don't remember",
"si, esta bien, repitamelo por favor, no le entiendo":
"yes, that's fine, repeat it for me please, I don't understand you",
"bueno, ya me voy, bye": "okay, I'm going now, bye",
}
# Tokens for the language-mix check: Spanish words and English words in the caller turns.
G_OTHER = set("si le pago el veinte no ese monto esta mal pero tres con la tarjeta de debito en "
"lo pague bueno bien manana confirmacion se acuerdo cargo por atraso yo voy "
"pagar todo perdi mi trabajo ahorita puedo nada cinco soy su hermano que trata usted "
"tiene numero equivocado senor digame ocho siete uno nueve deudor manejando "
"llameme despues las seis poner una queja llaman todos los dias hay problema viernes "
"eso veintitres cheque llega repitalo favor cuanto es escuche repitamelo ya entiendo".split())
G_EN = set("yeah i can pay do it yes will the twentieth already paid on rd with my debit card in "
"app this late fee is wrong time lost job zip but cannot anything right now he at work am his "
"brother what about you have number eight five seven one nine call me after six m "
"driving complain to every day paycheck comes friday so that around twenty third "
"please repeat how much did not hear properly amount slowly okay see tell again "
"speaking fine today bye".split())
G_CHECKS = { # check: (simulated pass prob, real pass prob) for the synthetic generator
"task success (CRM diff)": (0.87, 0.70),
"advocate details stated (C06)": (0.97, 0.87),
"no clarification loop": (0.93, 0.77),
"no talk-over": (0.97, 0.77),
"amount stated correctly": (0.95, 0.80),
}
EDITIONS = {"global": dict(
world=GLOBAL, langs=G_LANGS, cards=G_CARDS, text=G_TEXT, gloss=G_GLOSS,
other_tokens=G_OTHER, en_tokens=G_EN, other_name="Spanish", checks=G_CHECKS,
header=f"Validate a simulated caller (R7). {GLOBAL['lender_line']}",
dataset_line=(f"Dataset: {GLOBAL['dataset']}; simulation seed {SIM_SEED}, shared with the"
" India edition. All data below is synthetic."),
money=lambda x: f"${x:,.2f}",
)}
# ---- INDIA EDITION START ------------------------------------------------------------
# (left out of this listing: the India edition's constants, persona cards, lines and
# word lists, which add EDITIONS["india"]; they are in the full file in the course repository)
# ---- INDIA EDITION END --------------------------------------------------------------
# The active edition. use_edition() fills these names; the code below reads only them.
ED: dict = {}
WORLD: dict = {}
LANGS: list[str] = []
CARDS: dict = {}
TEXT: dict = {}
CHECKS: dict = {}
def use_edition(name: str) -> None:
global ED, WORLD, LANGS, CARDS, TEXT, CHECKS
ED = EDITIONS[name]
WORLD, LANGS, CARDS, TEXT, CHECKS = (ED["world"], ED["langs"], ED["cards"], ED["text"],
ED["checks"])
# Valid persona-flow pairs (WG9 / W9 primary pairs plus plausible variants). 20 pairs,
# the same ids in both editions.
PERSONA_FLOWS = [
("P01", "F01"), ("P01", "F09"), ("P02", "F02"), ("P02", "F03"), ("P03", "F03"),
("P03", "F02"), ("P04", "F04"), ("P04", "F09"), ("P05", "F05"), ("P06", "F05"),
("P07", "F06"), ("P08", "F07"), ("P09", "F08"), ("P10", "F09"), ("P10", "F01"),
("P11", "F10"), ("P11", "F03"), ("P12", "F01"), ("P13", "F01"), ("P13", "F04"),
]
TERMINAL = {"F01": "end_call(ptp)", "F02": "end_call(paid)", "F03": "end_call(dispute)",
"F04": "transfer_to_human(hardship)", "F05": "mark_contact(third_party)",
"F06": "transfer_to_human(collections_l2)", "F07": "mark_contact(voicemail)",
"F08": "mark_contact(ivr)", "F09": "schedule_callback(borrower)",
"F10": "transfer_to_human(grievance)"}
# ---------- part 1: a pairwise scenario grid ----------
def build_grid() -> list[dict]:
"""3 rows per persona-flow pair; audio = j, language = (j + i) mod 3 (a Latin square)."""
rows = []
for i, (p, f) in enumerate(PERSONA_FLOWS):
for j in range(3):
rows.append({"sid": f"S{len(rows) + 1:03d}", "persona": p, "flow": f,
"audio": AUDIO[j], "lang": LANGS[(j + i) % 3]})
return rows
def pair_coverage(rows: list[dict]) -> dict[str, tuple[int, int]]:
levels = {"persona_flow": [f"{p}-{f}" for p, f in PERSONA_FLOWS], "audio": AUDIO, "lang": LANGS}
out = {}
for a, b in itertools.combinations(levels, 2):
need = set(itertools.product(levels[a], levels[b]))
got = {(r_val(r, a), r_val(r, b)) for r in rows}
out[f"{a} x {b}"] = (len(need & got), len(need))
return out
def r_val(r: dict, factor: str) -> str:
return f"{r['persona']}-{r['flow']}" if factor == "persona_flow" else r[factor]
# ---------- part 2a: a stand-in simulator with seeded drift ----------
DRIFT_MENU = [ # (kind, probability per attempt)
("truncated", 0.04), # media dropped, no terminal event
("early_hangup", 0.03), # simulated caller leaves before the flow ends
("behavior", 0.06), # caller abandons the card (the key line is replaced)
("fact", 0.04), # caller forgets or changes a fact the card says they know
("language", 0.03), # caller drops the card's language mix
("timer_interrupt", 0.08), # interruption on a timer, not on content: no validator checks it
]
def simulate(scn: dict, seed: int) -> dict:
rng = random.Random(seed)
card = CARDS[scn["persona"]]
u, drift, acc = rng.random(), None, 0.0
for kind, p in DRIFT_MENU:
acc += p
if u < acc:
drift = kind
break
if drift == "fact" and not card["facts"]:
drift = "language"
if drift == "language" and card["machine"]:
drift = "behavior"
lang = scn["lang"]
key = card["key"][lang]
turns = [("agent", TEXT["opener"][lang])]
if card["machine"]:
turns.insert(0, ("caller", key))
else:
turns.append(("caller", TEXT["answer"][lang]))
turns.append(("agent", TEXT["verify"]))
turns.append(("caller-interrupt" if card.get("content_interrupt") else "caller", key))
if scn["persona"] == "P02":
turns.append(("agent", TEXT["ask_ref"]))
ref = "".join(str(rng.randrange(10)) for _ in range(12))
turns.append(("caller", TEXT["give_ref"].format(ref=ref)))
if drift == "behavior":
turns = [(s, card["drift"] if t == key else t) for s, t in turns]
elif drift == "fact":
fact_re = re.compile("|".join(card["facts"].values()), re.I)
turns = [(s, fact_re.sub(TEXT["fact_forgot"], t) if s.startswith("caller") else t)
for s, t in turns]
elif drift == "language":
swap = TEXT["lang_swap_to_english"] if lang != LANGS[2] else TEXT["lang_swap_to_other"]
turns = [(s, swap if (s.startswith("caller") and t != key) else t) for s, t in turns]
elif drift == "timer_interrupt":
turns.append(("caller-timer", "hello? hello?"))
if drift == "truncated":
end = None
elif drift == "early_hangup":
turns.append(("caller", TEXT["hangup"]))
end = "caller_hangup"
else:
end = TERMINAL[scn["flow"]]
return {"sid": scn["sid"], "seed": seed, "scn": scn, "turns": turns, "end": end,
"drift": drift}
# ---------- part 2b: three validators and the regenerate loop ----------
def other_share(text: str) -> float | None:
"""Share of the non-English language's words among the words we can label."""
toks = re.findall(r"[a-z]+", text.lower())
other = sum(t in ED["other_tokens"] for t in toks)
en = sum(t in ED["en_tokens"] for t in toks)
return None if other + en == 0 else other / (other + en)
def v_call_end(run: dict) -> str | None:
if run["end"] is None:
return "no terminal event (call truncated)"
if run["end"] == "caller_hangup":
return "simulated caller hung up before the flow's terminal state"
return None
def v_behaviour(run: dict) -> str | None:
card = CARDS[run["scn"]["persona"]]
caller = [t for s, t in run["turns"] if s.startswith("caller")]
for pat in card["required"]:
if not any(re.search(pat, t, re.I) for t in caller):
return f"required behavior missing ({pat})"
for pat in card["forbidden"]:
if any(re.search(pat, t, re.I) for t in caller):
return f"forbidden behavior present ({pat})"
if card.get("content_interrupt") and not any(s == "caller-interrupt" for s, _ in run["turns"]):
return "content-triggered interruption missing"
return None
def v_caller_turns(run: dict) -> str | None:
card, lang = CARDS[run["scn"]["persona"]], run["scn"]["lang"]
caller = " ".join(t for s, t in run["turns"] if s.startswith("caller"))
for name, pat in card["facts"].items():
if not re.search(pat, caller, re.I):
return f"caller fact missing or changed ({name})"
if card["machine"]:
return None
share = other_share(caller)
if share is not None:
mixed, dominant, english = LANGS
what = f"{ED['other_name']} share {share:.2f}, card {lang}"
if lang == english and share > 0.4:
return f"language mix off card ({what})"
if lang == dominant and share < 0.6:
return f"language mix off card ({what})"
if lang == mixed and not 0.2 <= share <= 0.9:
return f"language mix off card ({what})"
return None
VALIDATORS = [("valid call end", v_call_end), ("behavioral fidelity", v_behaviour),
("caller-turn fidelity", v_caller_turns)]
def validate(run: dict) -> tuple[str, str] | None:
for name, fn in VALIDATORS:
reason = fn(run)
if reason:
return name, reason
return None
def run_with_regeneration(scn: dict, base_seed: int) -> tuple[dict, list[tuple[str, str]], bool]:
"""Simulate, validate, regenerate a rejected run up to MAX_RETRIES times."""
rejections = []
for attempt in range(MAX_RETRIES + 1):
run = simulate(scn, base_seed + 1000 * attempt)
verdict = validate(run)
if verdict is None:
return run, rejections, True
assert run["drift"] not in (None, "timer_interrupt"), f"false rejection: {verdict}"
rejections.append(verdict)
return run, rejections, False
# ---------- part 3: sim-vs-real on 30 twins ----------
def make_twins(n: int, seed: int) -> list[dict]:
rng = random.Random(seed)
twins = []
for i in range(n):
row = {"twin": f"R{i + 1:02d}", "lang": LANGS[i % 3], "sim": {}, "real": {}}
for chk, (ps, pr) in CHECKS.items():
u = rng.random() # shared draw: the twin is the same scenario
row["sim"][chk] = u < ps
row["real"][chk] = u < pr
if rng.random() < 0.05: # sim and real do not always fail on the same call
row["sim"][chk] = not row["sim"][chk]
twins.append(row)
return twins
def mcnemar_exact(b: int, c: int) -> float:
n = b + c
if n == 0:
return 1.0
k = min(b, c)
tail = sum(math.comb(n, i) for i in range(k + 1)) / 2 ** n
return min(1.0, 2 * tail)
def paired_bootstrap_mean_gap(twins: list[dict], reps: int, seed: int) -> tuple[float, float]:
rng = random.Random(seed)
n, stats = len(twins), []
for _ in range(reps):
sample = [twins[rng.randrange(n)] for _ in range(n)]
gaps = [sum(t["sim"][c] - t["real"][c] for t in sample) / n for c in CHECKS]
stats.append(sum(gaps) / len(gaps))
stats.sort()
return stats[int(0.025 * reps)], stats[int(0.975 * reps) - 1]
# ---------- part 4: two-rater attribution audit ----------
def cohen_kappa(a: list[str], b: list[str]) -> tuple[float, float]:
n = len(a)
po = sum(x == y for x, y in zip(a, b)) / n
ca, cb = Counter(a), Counter(b)
pe = sum(ca[k] * cb[k] for k in set(a) | set(b)) / n ** 2
return po, (po - pe) / (1 - pe)
def wilson(k: int, n: int, z: float = 1.959964) -> tuple[float, float]:
p = k / n
d = 1 + z * z / n
centre = (p + z * z / (2 * n)) / d
half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
return centre - half, centre + half
AGENT_PASS = {"P02": 0.70, "P11": 0.70, "P12": 0.55, "P13": 0.65} # others 0.80
def agent_outcome(run: dict) -> tuple[bool, str]:
"""Return (passed, true cause of failure). Synthetic: a stand-in for your checks."""
rng = random.Random(run["seed"] * 7 + 3)
if run["drift"] == "timer_interrupt" and rng.random() < 0.7:
return False, "simulator"
p = AGENT_PASS.get(run["scn"]["persona"], 0.80)
return (True, "") if rng.random() < p else (False, "agent")
def rate(rng: random.Random, truth: str, acc: float) -> str:
if rng.random() < acc:
return truth
return "simulator" if truth == "agent" else "agent"
def main() -> None:
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--edition", choices=list(EDITIONS), default="global",
help="which fictional world's cards and labels to use (default: global)")
use_edition(ap.parse_args().edition)
if ED["gloss"]: # every line with words of the other language carries an English gloss
lines = [t for c in CARDS.values() for t in [*c["key"].values(), c["drift"]]]
lines += [t for v in TEXT.values() for t in (v.values() if isinstance(v, dict) else [v])]
missing = [t for t in set(lines) if other_share(t) and t not in ED["gloss"]]
assert not missing, f"no gloss for: {missing}"
print(ED["header"])
print(ED["dataset_line"] + "\n")
# part 1
grid = build_grid()
full = len(PERSONA_FLOWS) * len(AUDIO) * len(LANGS)
cov = pair_coverage(grid)
print("== 1. scenario grid ==")
print(f"persona-flow pairs: {len(PERSONA_FLOWS)}; audio conditions: {len(AUDIO)};"
f" language mixes: {len(LANGS)}")
print(f"full factorial: {full} scenarios; pairwise grid: {len(grid)} scenarios")
for name, (got, need) in cov.items():
print(f" {name:<24} {got}/{need} pairs covered")
assert len(grid) == 60 and full == 180
assert all(got == need for got, need in cov.values())
assert {(r["persona"], r["flow"]) for r in grid} == set(PERSONA_FLOWS)
# part 2
runs, first_reasons, all_reasons, attempts, unrecovered = [], Counter(), Counter(), 0, 0
reason_text, dropped = Counter(), []
first_pass_rejects = 0
for k in range(2):
for scn in grid:
base = SIM_SEED + 10 * int(scn["sid"][1:]) + k
run, rejections, ok = run_with_regeneration(scn, base)
attempts += len(rejections) + (1 if ok else 0)
if rejections:
first_pass_rejects += 1
first_reasons[rejections[0][0]] += 1
reason_text[rejections[0][1]] += 1
for name, _ in rejections:
all_reasons[name] += 1
if ok:
runs.append(run)
else:
unrecovered += 1
dropped.append(f"{scn['sid']} {scn['persona']}-{scn['flow']} seed {k}")
planned = 2 * len(grid)
print(f"\n== 2. validators and regeneration (up to {MAX_RETRIES} retries) ==")
print(f"planned runs: {planned}; simulator calls placed: {attempts}")
print(f"rejected on first attempt: {first_pass_rejects}/{planned}"
f" = {first_pass_rejects / planned:.1%}")
for name, _ in VALIDATORS:
print(f" first-attempt rejections by {name:<22} {first_reasons[name]}")
for text, count in reason_text.most_common():
print(f" {count:>2} {text}")
print(f"all rejections incl. retries: {sum(all_reasons.values())}")
print(f"still invalid after {MAX_RETRIES} retries (dropped, reported): {unrecovered}"
f" ({', '.join(dropped)})")
print(f"valid runs kept: {len(runs)}")
undetected = sum(r["drift"] == "timer_interrupt" for r in runs)
print(f"kept runs with a timer-based interruption (no validator checks it): {undetected}")
extra = attempts - planned
nightly_calls = WORLD["sim_cadence"]["nightly_audio"][0] * WORLD["sim_cadence"]["nightly_audio"][1]
per_call = WORLD["sim_cost"]["audio_call"]
print(f"regeneration overhead: {extra / planned:.1%} extra calls; at the nightly"
f" {nightly_calls} audio calls x {ED['money'](per_call)} that is about"
f" {ED['money'](nightly_calls * per_call * extra / planned)} a night")
example = next(r for r in runs if r["scn"]["persona"] == "P12" and r["scn"]["lang"] == LANGS[0])
assert attempts == planned + extra and len(runs) + unrecovered == planned
assert first_pass_rejects > 0 and all(first_reasons[n] > 0 for n, _ in VALIDATORS)
# regression lock for simulation seed 20261120, the same in both editions (pages quote these)
assert (attempts, first_pass_rejects, unrecovered, len(runs), undetected) == (152, 25, 2, 118, 10)
assert [first_reasons[n] for n, _ in VALIDATORS] == [11, 7, 7]
assert sum(all_reasons.values()) == 34
assert dropped == ["S008 P02-F02 seed 0", "S031 P07-F06 seed 0"]
# part 3
twins = make_twins(WORLD["twins_per_quarter"], SIM_SEED + 5)
n = len(twins)
print(f"\n== 3. sim vs real on {n} consented, redacted real calls and their twins ==")
print(f"{'check':<32}{'sim':>6}{'real':>6}{'gap':>7}{'b':>4}{'c':>4}{'McNemar p':>11}")
gaps = []
for chk in CHECKS:
s = sum(t["sim"][chk] for t in twins)
r = sum(t["real"][chk] for t in twins)
b = sum(t["sim"][chk] and not t["real"][chk] for t in twins)
c = sum(t["real"][chk] and not t["sim"][chk] for t in twins)
gaps.append((s - r) / n)
print(f"{chk:<32}{s / n:>6.2f}{r / n:>6.2f}{(s - r) / n:>+7.2f}{b:>4}{c:>4}"
f"{mcnemar_exact(b, c):>11.3f}")
assert s > r, f"simulator should flatter on {chk}"
mean_gap = sum(gaps) / len(gaps)
lo, hi = paired_bootstrap_mean_gap(twins, 2000, SIM_SEED + 6)
print(f"mean overestimate: {mean_gap:+.3f} (paired bootstrap 95% CI {lo:+.3f} to {hi:+.3f})")
print("b = sim pass, real fail; c = real pass, sim fail")
width = max(map(len, LANGS)) + 1
for lang in LANGS:
sub = [t for t in twins if t["lang"] == lang]
ts = sum(t["sim"]["task success (CRM diff)"] for t in sub)
tr = sum(t["real"]["task success (CRM diff)"] for t in sub)
print(f" task success, {lang:<{width}} sim {ts}/{len(sub)} real {tr}/{len(sub)}")
assert mean_gap > 0 and lo > 0
assert round(mean_gap, 3) == 0.087 and round(gaps[0], 2) == 0.20
assert (round(lo, 3), round(hi, 3)) == (0.020, 0.160)
# part 4
failed = [r for r in runs if not agent_outcome(r)[0]]
truth = [agent_outcome(r)[1] for r in failed]
rng = random.Random(SIM_SEED + 7)
r1 = [rate(rng, t, 0.90) for t in truth]
r2 = [rate(rng, t, 0.88) for t in truth]
po, kappa = cohen_kappa(r1, r2)
agreed = [a for a, b in zip(r1, r2) if a == b]
final = [a if a == b else t for a, b, t in zip(r1, r2, truth)] # disagreements adjudicated
sim_k = final.count("simulator")
wl, wh = wilson(sim_k, len(final))
print(f"\n== 4. two-rater attribution of {len(failed)} failed simulated runs ==")
print(f"rater 1: {Counter(r1)['agent']} agent, {Counter(r1)['simulator']} simulator;"
f" rater 2: {Counter(r2)['agent']} agent, {Counter(r2)['simulator']} simulator")
print(f"agreement {po:.3f}; Cohen's kappa {kappa:.3f}; disagreements adjudicated:"
f" {len(failed) - len(agreed)}")
print(f"simulator caused the first critical error: {sim_k}/{len(final)} = {sim_k / len(final):.1%}"
f" (Wilson 95% {wl:.3f} to {wh:.3f})")
tim = sum(r["drift"] == "timer_interrupt" for r in failed)
print(f"failed runs with a timer-based interruption: {tim}")
assert 0 < kappa < 1 and sim_k > 0 and len(failed) >= 20
assert (len(failed), sim_k, round(po, 3), round(kappa, 3)) == (42, 7, 0.810, 0.475)
assert (round(wl, 3), round(wh, 3), tim) == (0.083, 0.306, 7)
assert all(r["drift"] == "timer_interrupt" for r, f in zip(failed, final) if f == "simulator")
print(f"\nP12 example run {example['sid']} ({example['scn']['lang']},"
f" {example['scn']['audio']}), end = {example['end']}:")
for spk, text in example["turns"]:
print(f" {spk:<16} {text}")
if text in ED["gloss"]:
print(f" {'':<16} gloss: {ED['gloss'][text]}")
print("\nAll checks passed.")
if __name__ == "__main__":
main()
Step 3: Run it
.venv/bin/python caller-simulator-validator.py
Running it with Python 3.11 prints:
Validate a simulated caller (R7). Synthetic data.
Dataset: steelyard-calls (synthetic, seed 20261121); simulation seed 20261120, shared with the India edition. All data below is synthetic.
== 1. scenario grid ==
persona-flow pairs: 20; audio conditions: 3; language mixes: 3
full factorial: 180 scenarios; pairwise grid: 60 scenarios
persona_flow x audio 60/60 pairs covered
persona_flow x lang 60/60 pairs covered
audio x lang 9/9 pairs covered
== 2. validators and regeneration (up to 2 retries) ==
planned runs: 120; simulator calls placed: 152
rejected on first attempt: 25/120 = 20.8%
first-attempt rejections by valid call end 11
first-attempt rejections by behavioral fidelity 7
first-attempt rejections by caller-turn fidelity 7
8 no terminal event (call truncated)
3 simulated caller hung up before the flow's terminal state
2 required behavior missing (veinte|twentieth)
2 caller fact missing or changed (confirmation_no)
2 required behavior missing (esta mal|wrong)
2 required behavior missing (after six|despues de las seis)
1 language mix off card (Spanish share 0.47, card English)
1 caller fact missing or changed (paid_on)
1 language mix off card (Spanish share 0.40, card Spanish-dominant)
1 required behavior missing (\[beep)
1 caller fact missing or changed (date_after_payday)
1 language mix off card (Spanish share 0.50, card Spanish-dominant)
all rejections incl. retries: 34
still invalid after 2 retries (dropped, reported): 2 (S008 P02-F02 seed 0, S031 P07-F06 seed 0)
valid runs kept: 118
kept runs with a timer-based interruption (no validator checks it): 10
regeneration overhead: 26.7% extra calls; at the nightly 960 audio calls x $0.30 that is about $76.80 a night
== 3. sim vs real on 30 consented, redacted real calls and their twins ==
check sim real gap b c McNemar p
task success (CRM diff) 0.83 0.63 +0.20 7 1 0.070
advocate details stated (C06) 1.00 0.97 +0.03 1 0 1.000
no clarification loop 0.87 0.83 +0.03 3 2 1.000
no talk-over 0.97 0.83 +0.13 5 1 0.219
amount stated correctly 0.93 0.90 +0.03 2 1 1.000
mean overestimate: +0.087 (paired bootstrap 95% CI +0.020 to +0.160)
b = sim pass, real fail; c = real pass, sim fail
task success, Spanglish sim 7/10 real 5/10
task success, Spanish-dominant sim 10/10 real 8/10
task success, English sim 8/10 real 6/10
== 4. two-rater attribution of 42 failed simulated runs ==
rater 1: 32 agent, 10 simulator; rater 2: 32 agent, 10 simulator
agreement 0.810; Cohen's kappa 0.475; disagreements adjudicated: 8
simulator caused the first critical error: 7/42 = 16.7% (Wilson 95% 0.083 to 0.306)
failed runs with a timer-based interruption: 7
P12 example run S053 (Spanglish, phone-8k-mulaw), end = end_call(ptp):
agent Hi, this is Steelyard Credit. I'm Hazel, an AI assistant. This call is recorded.
caller si, speaking
gloss: yes, speaking
agent Thank you. To verify your identity, can you tell me your ZIP code?
caller si, pero my paycheck comes el viernes, so after that I can pay, el veintitres
gloss: yes, but my paycheck comes on Friday, so after that I can pay, on the twenty-third
All checks passed.
Every printed number is checked by an assert before the script prints its last line, including a lock on the values for simulation seed 20261120. If you change the generator or a card and an assert fails, the numbers on this page no longer describe your run. The lock caught a real case while this edition was written: dropping two ambiguous words from the Spanish word list shifted two language-mix verdicts and moved the call count.
Step 4: Check that the grid covers every pair
A pairwise grid puts every level of each factor next to every level of every other factor at least once. Here that takes 60 scenarios where the full factorial needs 180. With 20 persona-flow pairs and two 3-level factors, 60 is the minimum: each persona-flow pair has to meet all three audio conditions. The Latin square (audio = j, language = (j + i) mod 3) also covers all 9 audio x language pairs, and the script checks all three coverage counts by brute force.
The persona-flow factor is a list of valid pairs, not a product. P05 (the borrower's adult brother) never runs F01, because a promise to pay from a third party is a scenario-design bug, not a test. Pairwise coverage catches failures caused by two factors together, such as P12 on 8 kHz audio with a Spanglish mix. It does not promise any three-way combination, so add the three-way cells you already know are risky, starting with the ones behind your incidents.
Step 5: Read the validator rejections
The validators rejected 25 of the 120 planned runs on the first attempt. Valid call end caught 11 of those (8 truncated calls, 3 early hang-ups). Behavioral fidelity caught 7, for example a P03 caller who dropped the dispute ("esta mal" [it's wrong]) the card requires. Caller-turn fidelity caught 7, among them a P02 caller who forgot the payment confirmation number her card says she has, and three calls whose language mix drifted off the card. After up to two regenerations, 118 runs were kept. Two runs, S008 and S031 at seed 0, failed all three attempts and were dropped and listed by id. Report dropped runs. Hiding them shrinks the denominator to the scenarios your simulator handles well.
The validators run in a fixed order and the first failure decides the reason. Put the cheapest, most structural check first. A truncated call has no meaningful behavior to judge.
The language validator matters most for a book with code-switching borrowers. Simulators drop the switch quietly. LiveKit's simulation docs, read on 2026-11-20, say that mixed-language speech runs in the non-English language (their example is Hindi mixed with English), and that the simulated user does not play voicemail greetings or phone menus 3. A Spanglish caller rendered as Spanish never switches inside a clause, so a suite like that cannot reproduce T08, and it cannot reproduce T07's call-screening assistant either, however many seeds it runs. This script scores the card's language mix with a small word list and the share of Spanish words among the words it can label, with one band per mix. That is crude, and it is enough to flag a Spanish-dominant caller who answers in English. Words that belong to both languages, such as "me" and "a", stay out of both lists.
Regeneration costs money. In this synthetic run, 152 simulator calls produced 118 kept runs, a 26.7% overhead. At Steelyard's nightly audio cadence of 240 scenarios at k=4 (960 calls at $0.30 each, $288.00 a night), the same rate adds about $76.80 a night, or $2,304.00 over 30 nights. The monthly plan's $8,640.00 nightly simulation line assumes no regeneration, so it would rise to $10,944.00. The plan total goes from about $45,673.03 to about $47,977.03, still inside the $50,000 cap. Your own rejection rate replaces the synthetic 26.7%.
Ten kept runs carry a timer-based interruption that no validator looked at. That is the gap the attribution audit in Step 7 finds. Vendor simulators can have the same blind spot. Coval's persona settings, read on 2026-11-20, schedule proactive interruptions on a timer that is not tied to what the agent is saying, so a caller who cuts in when the amount is read out has to be scripted by hand 4. Steelyard's cards specify content-triggered interruptions, and P02's card requires one: Sofia Ramirez cuts in with "ya lo pague" [I already paid it] while Hazel states the dues.
Step 6: Compare simulated twins with real calls
Run 30 consented, redacted real calls and simulated twins of the same scenarios, then compare pass rates check by check. In this synthetic set the simulated pass rate is higher on every check, and the mean overestimate across the five checks is 8.7 points (paired bootstrap 95% interval 2.0 to 16.0). Use it as the correction to simulated pass rates until the next calibration, and keep the per-check gaps beside it: task success runs 20 points high, three of the checks only 3.
The twins are paired, so the script compares them with tests for paired data, not with two independent intervals. McNemar's exact test uses only the discordant pairs: b twins passed in simulation and failed for real, c the reverse 5. Task success shows the largest gap (0.83 simulated against 0.63 real, b = 7, c = 1), and with 30 twins even that gives p = 0.070. Thirty twins are enough to see that the simulator flatters the agent. They are not enough to price each check separately, and the 10 per language group are thinner still. Steelyard runs the comparison quarterly, per language group, and treats it as calibration, not as a gate.
The tau-Voice authors list ways their simulated user is easier than a real caller: more patience, perfect memory and instant tool calls. It reads the agent's transcript instead of transcribing its speech, and its accent results from TTS personas are indicative only 6. Expect your twins to show the same direction. A simulator whose pass rates come out lower than reality is also worth investigating. Usually it means the card is harsher than real callers are.
Step 7: Audit who caused each failure
For every failed simulated run, two raters listen and attribute the first critical error to the agent or to the simulator. In this run, 42 of the 118 kept runs failed. The raters agreed on 81.0% of them, but Cohen's kappa was only 0.475, because most failures are agent failures and two raters who mostly say "agent" agree often by chance 7. Report kappa next to raw agreement. Agreement alone overstates how much the raters agree beyond chance.
After the 8 disagreements were adjudicated, the simulator caused the first critical error in 7 of 42 failures, 16.7% (Wilson 95% interval 8.3% to 30.6%). All 7 were timer-based interruptions, the drift no validator checks. A single proportion like this one is where a Wilson interval belongs. A published audit gives a reference point. In tau-Voice, two raters audited 91 failed runs from two selected cohorts, agreed on 84% of them (76 of 91), and found that the simulated user caused the first critical error in 21% of one cohort's failures and 10% of the other's 6.
The audit tells you what validator to write next. Here the fix is a check that every interruption in a simulated call follows the content trigger on its card. Steelyard's policy is a two-rater audit of 50 failed simulated runs per release, with bilingual English-Spanish raters for the Spanish and Spanglish runs. This synthetic night had 42 failures, so audit all of them.
Step 8: Plug in your own simulator
Replace simulate with a loader for your simulator's transcripts. Each run needs the scenario (persona, flow, audio condition, language mix), the ordered turns with the speaker and text, a marker for content-triggered interruptions, and the terminal event or none. Keep the validators and the regenerate loop as they are, and extend each card's required, forbidden and facts from your gold-set scenario cards. If your callers speak another language pair, replace the two word lists and keep the three bands. Then replace make_twins with per-check results from your real calls and their twins, and replace rate with your two raters' labels.
Two rules carry over. The simulated call has to go through the production audio path (8 kHz, the codec chain, the trunk), or the audio conditions in the grid are labels with nothing behind them; the phone-path degradation recipe builds that path offline. And a production failure you cut into a scenario must fail in simulation before the fix. If it passes, the simulator cannot reproduce the failure, and its pass rate tells you nothing about that pattern.
Run the full file with --edition india for the India edition's persona cards, language mixes and money labels; the India edition of the simulation chapter explains that output. Every count, rate and interval comes out the same in both editions.
What to do next
Read the voice simulation chapter for persona design, outbound paths and Steelyard's simulation cadence: text scenarios on every commit, audio nightly and before release. Once the validators hold, compute pass^k over the kept runs with the pass^k planner recipe, and reuse the per-turn and per-session checks from the multi-turn conversation recipe as the agent-side scoring for each simulated call.