Ten calls broke Asha in November, each exposing a missing component: a wrong amount written to the CRM (T01), an ignored barge-in (T02), dues disclosed to a borrower's wife (T03), a 3,990 ms silence at peak (T04), and six more. This chapter treats that log as the acceptance test for an evaluation platform: a design passes only if it names the component, the gate and the time to detection for every trace. Earlier chapters own the methods; this one fits them together, prices them, decides what to build, and ends with the capstone.
All ten in the T01-T10 log, each by a named check with a gate and a detection time set by its cadence. Cheap deterministic checks on every call catch most within minutes of call end; the rest live in audio, timing or load and need nightly or pre-release suites.
| Trace | Failure | Component | Check | Gate | Detected |
|---|
| T01 | "dhai" heard as "do"; PTP before read-back | STT entity suite; production checks | Amount entity error rate; C12 order | Nightly upper bound; C12 blocks | Before merge (C12); minutes in production |
| T02 | Barge-in ignored | Audio simulation; production checks | Stop latency, recovery; C08 tool sequence | Nightly; C08 blocks | Next nightly; minutes |
| T03 | Dues told to a third party | Wrong-party personas; production checks | C04 ordered events, C05 contact role | Any failure blocks | Before merge; first call |
| T04 | Tool-turn tail, duplicate reply at peak | Load generator; production checks | P95 tool-turn turn gap at 1.0x and 1.5x; duplicate reply | Pre-release load gate | Before release; minutes |
| T05 | False cutoff on a mid-turn pause | Endpointing suite | False-cutoff rate at the campaign budget | Nightly; detector or budget change | Next nightly |
| T06 | TTS read 05/12 as 12 May | Phone-path TTS suite | Spoken-entity round trip | Nightly; TTS or prompt change | Next nightly |
| T07 | Operator announcement taken as human | AMD suite | Six-class confusion matrix, errors costed | Pre-release; AMD change | Before release |
| T08 | Code-switch clarification loop | Audio simulation; production checks | Code-switching callers; LOOP.CLARIFY | Nightly; monitored | Next nightly; minutes |
| T09 | Threat on the angry path | Lexicon on 100%; J1 on angry calls | Hit or flag to human audit; regression case | Audit confirms | Same day |
| T10 | Money request after hardship | Production checks; J2 | C09 tool order; J2 pacing (advisory) | C09 blocks | Minutes |
T01 (entity suite and C12), T03 (wrong-party simulation and C04) and T09 (lexicon and judge) are each caught twice, independently: the lexicon missed "ghar aa", and a judge on every angry call was the second chance.
Three planes follow: data (each call on one clock), eval (scoring) and control (gold sets, judge registry, gate logic).
flowchart LR
subgraph DP["Data plane: every call"]
A["Dialer and SIP trunk"] --> B["Asha: STT, turn detector, LLM, tools, TTS"]
B --> C["Dual-channel recording"]
B --> D["Spans and events on one clock"]
B --> E["CRM before and after"]
end
subgraph EP["Eval plane"]
F["Trace store: redacted copies, content-free spans"]
G["Deterministic checks C01-C16 on 100%"]
H["Judges J1-J3 on the stratified sample"]
I["Human audit, 400 a week"]
J["Simulation fleet and offline suites"]
end
subgraph CP["Control plane"]
K["Gold sets gold-v n"]
L["Judge registry"]
M["Gate logic and release record"]
end
C --> F
D --> F
E --> F
F --> G
F --> H
H --> I
K --> J
J --> M
G --> M
L --> H
I --> L
What is the data model?
The call is the unit, on one clock: milliseconds since media start at the agent's ingress. Nine record types, each keyed so a reviewer can jump from a failed check to the audio offset that caused it, and each carrying the version vector of every component that touched it.
The call record is the traces chapter's schema; the platform adds keys, PII classes and retention:
| Record | Key | Holds | PII | Retention |
|---|
| Call | call_id | Campaign, pipeline, codec chain, AMD label, versions | Metadata | Metric |
| Turn | (call_id, turn_id) | Speaker, turn type, VAD times, STT final, reference | Content | Eval |
| Span | (call_id, turn_id, stage) | Start and end ms, no text | None | Metric |
| Audio segment | (call_id, channel, start_ms, end_ms) | URI to raw or redacted audio | Content | Raw or eval |
| Tool call | (call_id, turn_id, seq) | Name, redacted args, result, timings | Content | Eval |
| CRM diff | call_id | Before and after JSON | Content | Eval |
| Check result | (call_id, check_id, check_version) | Verdict, evidence turns, policy pack | None | Metric |
| Label | (call_id, turn_id, labeler, taxonomy) | Failure code, note | Reference | Eval |
| Judge score | (call_id, judge_id@version, dimension) | Score, rationale, input hash | Reference | Metric |
Raw recordings of recovery calls are kept at least six months from the call date, or until disposal if sub judice, as RBI's amended recovery directions require from 1 January 2027 1. Eval copies are redacted at ingest and kept while a gold release references them. Metric records hold no content, so strip the transcripts and tool arguments frameworks export inside latency spans.
Task success is a diff of CRM state against the scenario's gold end state, the tau-bench idea of comparing the final database 2. Keep spoken-communication checks (was the grievance officer named) as separate results: tau-Voice folds an LLM evaluator for spoken communications into its task score, so that score is not fully deterministic 3. A diff and a judged check fail in different ways; never merge them. Versioned check results mean changing C06 adds results instead of rewriting last month's compliance rate.
How do you version gold sets?
As immutable releases, gold-v{n}, with lineage: source, consent, redaction version, labelers and adjudication. Metrics are compared only within one release, and the incumbent is rerun on each new release so comparisons stay paired.
gold-v7 holds 600 scenario cards, 1,200 reference-transcribed utterances from 180 speakers, 500 TTS lines (400 of them amounts), 400 labelled pauses per language group, 120 human-labelled calls per language group per judged dimension, and 200 labelled AMD openings. The general method is on dataset versioning and lineage; three rules are voice-specific.
Incident intake, fail-first. Each production incident becomes a scenario card within two working days, and it must fail on the version that produced the incident; one that passes has not captured the failure.
Coverage by flow and party, not volume. T03 passed regression because the borrower answered every gold scenario. Each gated language group needs flows F01-F10, including wrong party, guarantor, voicemail and IVR.
A new language is a gold-set project. Marathi or Kannada each need 400 labelled pauses, 1,200 utterances from at least 150 speakers and 120 labelled calls per judged dimension before any gate in that language means anything. Cut releases monthly, never mid-comparison.
What runs per commit, nightly and before release?
Four tiers, priced by run cost and sized by what each gate's statistics need: text scenarios on every commit, phone-path audio nightly, load and the paired segment gate before release, and a production canary. Plumbing is on CI integration, gate logic on release gates.
| Tier | What runs | Size | Gate | Monthly cost |
|---|
| Per commit | Text scenarios and all deterministic checks | 600 at k=2, about 10 commits a day | C01-C16 failure blocks; pass^2 per flow vs incumbent | Rs 1,32,000 |
| Nightly | Phone-path audio scenarios; STT and TTS entity suites; endpointing; AMD | 240 at k=4, 20 per gated segment | Interval rules below | Rs 1,72,800 |
| Pre-release | Load at 1.0x and 1.5x of peak plus a 3x spike; paired segment gate | 12 segments | Holm | Rs 1,00,000 |
| Canary | 5% of traffic | Until the confidence sequence decides | North star, guardrails; compliance failure halts | Production cost |
Text runs go per commit because an audio scenario costs 12 times as much (Rs 6 against Rs 0.50). They catch prompt and tool-order regressions such as T03, never the audio-only T01, T05 or T06.
Every gate is an interval rule using the reliability chapter's statistics:
- Amount and date entity error rate: the speaker-clustered bootstrap upper bound stays below the campaign threshold.
- P95 platform turn gap on tool turns: the call-clustered upper bound stays at or below silence budget plus 2,200 ms (2,600 ms on TW-early-nov, 2,800 ms on MFI-fort-nov; terms from the latency chapter).
- False-cutoff rate at the campaign budget, per language and call state (turn-taking chapter).
- pass^4 per flow, paired against the incumbent with McNemar or a paired bootstrap.
- Compliance: one failure blocks.
Multiplicity. Twelve segments (three language groups times four flow groups) at 0.05 each block good releases by chance. Holm's step-down method compares the smallest p-value with 0.05/12, about 0.0042, then relaxes one segment at a time 4.
Load. Ramp to projected peak, hold, then spike to 3x the busiest hour 5; stop conditions are in the load chapter.
What is in the simulation fleet?
Persona callers that reach Asha through the production phone path, with impairments, content-triggered interruptions, a sandbox CRM to diff, and validators that discard runs where the simulator failed. Fidelity is checked against real calls quarterly (simulation chapter).
- Scenario cards. Personas P01-P13 with SNR, codec chain, loss profile, a content-triggered interruption script and gold CRM end state.
- Audio-path parity. SIP egress, so the 8 kHz mu-law stream, AMD and the turn detector see production audio; clean websocket audio tests a different system.
- Impairments and outbound. The telephony chapter's condition matrix; callees: borrowers, wives, voicemail, operator announcements, IVRs.
- Validators. EVA-Bench validates each simulated call (valid call end, user behavioral and speech fidelity) and regenerates failures; across four systems, 12% of trials were regenerated for simulator error, almost all behavioral drift 6. A rising regeneration rate is a simulator regression.
- Audit. Two raters audit 50 failed runs per release; quarterly, 30 consented real calls per language get simulated twins that must fail where the real calls failed.
Concurrency. Nightly is 960 audio calls, 1,44,000 call-seconds at 150 s each: 3 h 20 min at 12 simultaneous calls, 96 minutes at 25. A vendor's concurrency limit decides whether nightly finishes overnight.
Simulation is the strongest buy candidate; read the documented limits.
CAUTION
Vendor-reported. Treat it as a hypothesis to test on your own calls.
From vendor docs read on 2026-11-20:
- LiveKit. Reports pass@k and pass^k. Outbound agents can be simulated in a room, but phone-number testing is inbound only; no voicemail greetings or phone menus; Hindi mixed with English runs in the non-English language 7.
- Vapi. "Synthetic callers do not fully represent real callers and their environments" 8.
- Retell. Simulation is a text conversation, with no custom-LLM agents; voicemail and IVR detection, DTMF and transfers need web or phone calls 9. Tests bill at production rates: 20 cases run three times bill 60 runs 10.
- Roark. Seeded packet loss in 20 ms packets, presets up to 10% in 120 ms gaps 11; an
#{interrupt} cue fires at a stated point in the script 12.
- Coval. Outbound via a webhook that makes your system dial the simulated customer 13. Interruptions fire on a timer (about every 90, 45 or 30 seconds); its five Indian-accent voices sit in a tier limited to about 12 simultaneous connections 14.
- Cekura. "Evaluator instructions cannot override a personality setting": an instruction to interrupt constantly has no effect, yet the call returns a result 15. Outbound tests with bring-your-own Twilio and Plivo numbers are documented 16. Its simulation Transcription Accuracy scores the testing agent's speech-to-text, not yours.
- Bluejay. Voicemail simulation plays a greeting and a beep, outbound only, at most 100 Digital Humans per test 17. Self-reported pay-as-you-go: $0 a month plus usage, $25 in credits, up to 25 concurrent simulations 18.
Two limits decide Tarazu's shortlist: T02 needs an interruption triggered by what Asha says, which a timer cannot reproduce, and T07 needs an Indian operator announcement as the callee.
How does online sampling work?
Deterministic checks and the lexicon run on every call; judges on every risk-stratum call plus a small share of the rest; human audits on both strata. Reported rates are reweighted and corrected for judge error, as the monitoring chapter designs.
Tarazu's plan since T09: every angry, complaint, dispute, hardship and window-edge call (about 4% of connects, 5,200 a day) plus 2% of the rest (2,496 a day), about 7,696 judged calls a day, and 400 human audits a week. The old 2% uniform sample had about a 28% chance of seeing even one threat in the three days the bad prompt was live.
Rare-violation sizing. Seeing a 1-in-2,000 violation within a week with 95% probability takes 5,990 sampled calls, since 1 - (1 - 1/2000)^5990 = 0.95000. The non-risk sample alone is 2,496 x 6 = 14,976 a week (probability 0.9994). That covers C07 judging and the C04 judge backstop; C04's deterministic check runs on every call.
Reported rates. A judge's raw flag rate on a stratified sample is biased twice, by the strata and by judge error. Report a stratum-weighted prediction-powered estimate, which corrects the judge with the human-audited subset 19; see also online versus offline evaluation.
What is a judge registry?
A versioned record per judge, dimension and language: inputs, validation against human labels, gating or advisory status, revalidation triggers, owner and cost. Gate logic reads it, so an unvalidated judge cannot gate a release in that language.
J1's entry after its November 2026 Hinglish validation (method in the judges chapter):
{
"id": "J1", "version": "v1", "dimension": "C07 coercion and consequence misstatement",
"input": "agent-turn transcript plus lexicon hits", "owner": "staff engineer, evaluation",
"cost_rs_per_call": 0.15,
"validation": {
"hinglish": {"gold": "gold-v7", "n_calls": 120, "violations": 18, "caught": 15, "false_alarms": 6,
"recall": 0.833, "recall_wilson95": [0.608, 0.942],
"specificity": 0.941, "specificity_wilson95": [0.878, 0.973],
"precision": 0.714, "kappa": 0.725, "bar": "recall lower bound >= 0.90",
"result": "fail"},
"hindi_dominant": null, "indian_english": null
},
"status": "advisory", "routing": "flags go to human audit",
"revalidate_on": ["judge model or prompt change", "upstream STT or LLM version change",
"taxonomy change", "new language"],
"anchor_set": "frozen 200-call anchor set"
}
Admit on the lower bound, not the point estimate: J1's recall of 0.833 looks usable, but its lower bound of 0.608 says it may miss two threats in five. With 18 violations no recall can clear the bar: even 18 of 18 has a lower bound of about 0.82. So a new language gets a gating judge only after about 100 native-labelled violations: 97 caught of 100 gives a Wilson lower bound of 0.915, while 96 of 100 gives 0.902, barely.
Agreement and kappa are recorded but never decide admission: at 3% prevalence, an always-"compliant" judge agrees 97% of the time with kappa 0.
What does it cost?
About Rs 9,13,460 a month today, inside a Rs 10 lakh cap, at Tarazu's invented unit prices. At next quarter's 2.5x volume the three volume-driven lines reach about Rs 10,00,818 and the plan about Rs 15,13,951, just over the Rs 15 lakh cap before any Marathi or Kannada judge.
Unit costs and policy pricing are in the load and cost chapter.
| Line | Today | At 2.5x volume |
|---|
| Deterministic checks on 100% | Rs 6,760 | Rs 16,900 |
| Judges J1, J3 (text), J2 (blueprint), 7,696 calls a day | Rs 1,40,067 | Rs 3,50,168 |
| STT re-transcription on 5% of connects | Rs 2,53,500 | Rs 6,33,750 |
| Human audit, 400 calls a week | Rs 1,08,333 | Rs 1,08,333 |
| Nightly audio simulation, 240 x k=4 | Rs 1,72,800 | Rs 1,72,800 |
| Per-commit text runs, 600 x k=2 | Rs 1,32,000 | Rs 1,32,000 |
| Two pre-release load tests | Rs 1,00,000 | Rs 1,00,000 |
| Total | about Rs 9,13,460 | about Rs 15,13,951 |
First lever: STT re-transcription. It buys interval width, not violations found. Cutting it from 5% to 3% of connects saves Rs 2,53,500, for a plan of Rs 12,60,451.
What the saving pays for. Gating Marathi and Kannada takes segments from 12 to 20 and nightly simulation from 240 to 400 scenarios (Rs 2,88,000). The plan is then Rs 13,75,651, leaving about Rs 1.2 lakh for new-language judges and labels.
Never cut. Checks and the lexicon on 100%, and judging every risk-stratum call. Cost per detected violation allocates money between lines, never whether a compliance line exists.
What do you build and what do you buy?
Own what encodes Tarazu's rules or must survive a vendor exit: policy checks, gold sets, scenario cards, the CRM diff, the judge registry and gate logic. Buy impairment, load generation and labelling from vendors that meet six criteria and sign exit terms.
| Component | Decision | Why | Exit terms |
|---|
| Policy checks C01-C16, CRM diff | Build | Tarazu's reading of RBI, TRAI and DPDP | None needed |
| Gold sets, scenario cards | Build | The test is the asset | None needed |
| Judge registry, gate logic | Build | Releases must be reproducible from Tarazu's records | None needed |
| Caller simulation | Buy if a vendor passes; else build | Persona voices and SIP plumbing are costly | Scenario cards in Tarazu's format; raw audio export |
| Telephony impairment | Buy | Commodity | Seeded profiles exported per run |
| Load generation | Buy | Commodity, if it dials 40 call starts a second (6,000 / 150 s) | Raw per-call spans |
| Judge models | Buy the model; own prompt and validation | Validation is per language | Prompts and validation sets stay |
| Dashboards, labelling UI | Buy or reuse | Commodity over Tarazu's stores | No metric or label held only by the vendor |
| Listening review tool | Build unless a vendor shows the timeline with check results | Reviewers need audio, transcripts and evidence turns together | None needed |
The criteria, tested on Tarazu's own calls:
- Outbound support: voicemail, operator and IVR openings as callees.
- Hinglish fidelity: code-switched callers; vendor STT measured on Tarazu's references.
- Audio-path parity: SIP at 8 kHz through the production media stream.
- Indian data residency and DPDP processor terms: the DPDP Rules require security safeguards, including masking or tokenisation, and processor contracts from around May 2027 20.
- Raw-result export: per-call audio, spans, transcripts and verdicts.
- Cost at volume: 960 nightly audio calls and 1,200 text runs per commit, finishing overnight.
What changes if a speech-to-speech model replaces the cascade?
Three families of evals break: per-stage latency attribution, every compliance check that reads TTS input text, and the turn-taking suites, because the model now owns endpointing. The CRM diff, tool-order checks and gold scenarios carry over. Run asha-s2s-pilot paired against the cascade as control.
| Eval | Cascade | Speech-to-speech | What to do |
|---|
| Latency | Per-stage spans: end-of-turn delay, LLM TTFT, tool time, TTS TTFB | LLM and TTS blocks absent, not zero | Mouth-to-ear turn gap from the dual-channel recording |
| C03, C04, C13 | Deterministic on TTS input text | No TTS text; transcript may lag the reply | Agent-channel STT, whose entity error rate the check inherits |
| TTS entity round trip | TTS text against the tool result | No text to compare | Agent-channel STT entities against get_dues |
| Endpointing and barge-in | Per-campaign silence budgets | The model decides when to speak | Rerun false-cutoff, talk-over and recovery suites |
| Task success | CRM diff | Unchanged | Unchanged |
| Judges | Transcript plus audio features | Transcript from agent-channel STT | Revalidate per language |
The compliance row is the expensive one. "No dues before verification" is an exact check on Tarazu's own text in the cascade; in speech-to-speech it reads a transcript of the model's audio, with its own error rate on amounts, so it needs the validation a judge gets.
Barge-in recovery is where real-time models are weakest: in the EchoChain preprint, four models had to revise their answer after a scripted barge-in, and none fully passed more than half of 200 interrupted conversations (best 48.5%, lowest 16.5%) 21. That is T02.
Architecture does not settle quality: since late October 2026 EVA's leaderboard shows a managed cascade, Cartesia Managed Agents with GPT 5.5, leading on both axes (0.87 EVA-A, 0.91 EVA-X pass@1) 22. Compare on Tarazu's scenarios, paired per segment (McNemar or paired bootstrap), under the same twelve-segment gate.
The design review
The capstone is a design document defended in a review. Quizzes and the exam are graded on the course; the design and memo appendix are self-assessed against the rubric below, with optional peer review.
You are the staff engineer for evaluation at Tarazu Finance. Next quarter Asha goes from 5 campaigns to 9 (adding a second PL book, an SBL top-up campaign, a larger MFI book and a Marathi pilot), from Hinglish, Hindi and Indian English to five languages (Marathi and Kannada handled by the bot for the first time), and from 2,400 to 6,000 peak concurrent calls. A speech-to-speech model (asha-s2s-pilot) is under evaluation as a replacement for the cascade. The RBI recovery amendments take effect on 1 January 2027 and DPDP Rule 6 around May 2027. Compliance wants evidence that no call discloses dues before verification and that calling-window violations are zero. The CFO has capped eval spend at Rs 15,00,000 a month; today's plan costs about Rs 9.1 lakh and scales to about Rs 15.1 lakh at the new volume (see the cost section above). Two vendors have pitched a managed testing platform. Write a design of at most six pages plus an appendix of your memos, and defend it in a 30-minute review with the head of engineering, the chief compliance officer and the CFO.
Cover:
- Problem statement: the three incidents from T01-T10 you consider most expensive, and why.
- Data model: call, turn, span, audio segment, tool call, CRM diff, check result, label, judge score; the clock; versions; PII classes and retention; how speech-to-speech traces differ.
- Gold sets: contents, versioning, refresh, incident intake.
- CI gates: per commit, nightly, pre-release, canary; gated metrics as interval rules; per-segment gating and multiplicity.
- Simulation fleet: personas, audio-path parity, impairments, outbound coverage, simulator validation and audit rate, concurrency and cost.
- Online sampling: 100% checks, strata and rates, human audit, how rates are reported, rare-violation sizing for C07 and for the C04 judge backstop (C04's deterministic check runs on every call).
- Judge registry: fields, admission rule, per-language status (including Marathi and Kannada), revalidation triggers, ownership.
- Cost budget: monthly cost by line from the unit costs on this page and cost per detected violation, inside the Rs 15 lakh cap.
- Build vs buy for simulation, telephony impairment, load generation, judges, dashboards and labelling, with criteria (outbound support, Hinglish fidelity, audio-path parity, Indian data residency and DPDP processor terms, raw-result export, cost at volume) and exit terms.
- Speech-to-speech: which evals disappear or must be rebuilt, and how you would run the comparison.
- Incident coverage table: T01-T10, the component and gate that catches each, and the expected time to detection.
- Evidence from your own work: cite at least four outputs from the course recipes or sims (for example your endpointing table from the endpointing recipe, your judge registry entry from the judge validation recipe).
How is the design scored?
Against a 100-point rubric: "meets" at 70 with no row at zero, "strong" at 85. Five errors cost 5 points each automatically, whatever else the design gets right.
| Row | Points | Meets | Strong |
|---|
| Incident coverage | 15 | All ten traces mapped to a component, a gate and a detection time | At least three caught by two independent components; detection times justified by cadence and sample sizes |
| Audio in the loop | 10 | STT, TTS, turn-taking and simulation evaluated on phone-path audio; wideband results non-gating | Real 8 kHz holdout, live impairment tests, caller-heard latency from recordings |
| Entity and task metrics | 10 | Entity error rate on amounts and dates as a gate with intervals; CRM-diff task scoring per flow | Spoken checks separated from the diff; confirmation polarity; verified PTP |
| Latency and turn-taking | 10 | One vocabulary; P95 per turn type; endpointing point per language and campaign | Per-call-state budgets, consequence weighting, SDK pinning, speech-to-speech implications |
| Statistics and gates | 10 | pass^k by the unbiased estimator; power-planned run counts; per-segment paired gates with Holm; clustered intervals | No-peeking canary; zero-event bounds for compliance; no Wilson on differences or ratio metrics |
| Judge registry | 10 | Per-language, per-dimension validation against human labels; lower-bound admission; revalidation triggers | Prevalence handled; gating vs advisory status; PPI for reported rates; no gating judge in a language not yet validated |
| Compliance as code | 10 | C01-C16 or equivalent with types and inputs; 100% coverage for cheap checks; product-aware windows | Express-request evidence, voice threat tests, redaction audit, dated legal assumptions |
| Production and rare events | 10 | Stratified sampling with sizing; drift signals; incident-to-regression with fail-first | Sizing shown for a 1-in-2,000 violation within a week; noise-season and carrier drift |
| Cost and build vs buy | 10 | Line-item budget within the Rs 15 lakh cap using the unit costs on this page; component-level decision with criteria | Cost per detected violation; exit terms; what is never cut |
| Clarity under review | 5 | Six pages; decisions with trade-offs; open risks named | Every number has an interval or an as-of date |
Automatic deductions, 5 points each: any claim this course corrects, stated in its uncorrected form (for example pass^8 at 80% given as anything other than 0.8^8, about 17%); a latency number without a start and end point; a judge used for gating without per-language validation; a vendor number cited without "self-reported" or an as-of date; a Wilson interval on a difference or on WER.
Reference design (write your own first)
Draft your design before reading this. It is one "strong" design, detailed in the sections above; yours can differ on any decision if the trade-off is argued.
- Most expensive. T03 (regulatory, every wrong-party pickup), T09 (rare, reputational, a complaint trigger), T01 (silent revenue loss on the commonest flow).
- Data model. The nine records; S2S traces carry
pipeline: "s2s", missing stages shown as missing, and agent-channel STT for compliance.
- Gold sets and CI. Immutable gold-v{n}, fail-first cases within two working days; the four tiers, with load at 1.0x and 1.5x of 6,000 plus a 3x spike and a 5% canary on confidence sequences.
- Simulation. Content-triggered personas with consented-voice or licensed TTS over SIP, regeneration rate reported, quarterly sim-vs-real.
- Online. About 7,696 judged calls a day; 400 audits a week with Neyman allocation; stratum-weighted PPI; C07 and the C04 backstop sized at 5,990 calls a week.
- Judges. J1 v1 advisory; no gating judge in Marathi or Kannada before about 100 native-labelled violations.
- Cost. STT re-transcription to 3% first; compliance never cut.
- Build vs buy and S2S. The matrix and the S2S table above, with the cascade as control.
- Coverage. The incident table, with T01, T03 and T09 caught twice.
What goes in the decision memo?
Tarazu's build vs buy memo for the head of engineering and the CFO, in the seven-part format every chapter uses.
Decision. Build policy checks, gold sets, the CRM diff, the judge registry and gate logic; buy telephony impairment, load generation and labelling; buy caller simulation only from a vendor that passes a T02 and T07 pilot on Tarazu's audio.
Context. Five to nine campaigns, three to five languages, 2,400 to 6,000 peak concurrent calls; RBI amendments from 1 January 2027.
Options considered. (a) One managed platform: fastest, but Tarazu's compliance reading and release record would live in a vendor. (b) Build everything: full control, months of SIP work. (c) Split by component: chosen.
Evidence. Vendor docs read 2026-11-20 (timer interruptions, inbound-only phone tests, text-only simulation); the 2.5x plan at Rs 15,13,951, Rs 12,60,451 after the STT cut; gold-v7.
Owner and cadence. Staff engineer, evaluation; quarterly and at contract renewal.
Tripwires. A pilot that fails T02 or T07; no raw audio and span export; regeneration rate above the agreed level for two nightly runs; DPDP processor terms unsigned before Rule 6 takes effect.
Dated facts. Vendor capabilities and Bluejay's self-reported pricing as of 2026-11-20; DPDP Rule 6 around May 2027.
What to do this week
Build the coverage table for your own incident log first. Every row without a component, a gate and a detection time is the backlog.
- Store version vectors and evidence turns with check results; strip content from latency spans.
- Turn last month's worst incident into a fail-first scenario card.
- Register judges per language and make the gate code read the registry.
- Pilot each simulation vendor on your two hardest incidents.
- Reread the chapters behind each row: error analysis, metric layers, Hinglish normalization, TTS, task success and compliance.
NOTE
What will date on this page. Vendor simulation capabilities and limits for LiveKit, Vapi, Retell, Roark, Coval, Cekura and Bluejay, and Bluejay's self-reported pricing (vendor docs, undated, read 2026-11-20). The EVA leaderboard leader (as of late October 2026). EchoChain's barge-in results (preprint, April 2026). The RBI recovery amendments (effective 1 January 2027) and DPDP Rule 6 commencement (around May 2027).