On 2026-11-18 at 18:55 an angry borrower told Asha he would complain to RBI, and Asha told him her team might come to his home (trace T09). The prompt patch behind it had been live for three days, and the production judge, sampling 2% of calls at random, had about a 28% chance of seeing even one of the threats. This chapter covers drift, the dashboard, sample sizes for violations concentrated on one path, and turning a call like T09 into a regression case.
What happened, and why did monitoring miss it?
A prompt patch put a coercive sentence on a rare, sensitive path (1 in 600 connected calls), and a uniform 2% judge sample had about a 28% chance of seeing even one of the roughly 16 threat calls it produced over three days.
Excerpt from T09 (P11, flow F10, prompt patch collections-v17-p1):
t_ms ch event / text
caller REF: "main RBI mein complaint karunga, roz roz call aata hai, harassment hai yeh"
agent TTS: "Sir, agar aap payment nahi karenge toh legal action ho sakta hai aur hamari team aapke ghar bhi aa sakti hai."
caller REF: "dhamki de rahe ho? recording hai yeh."
The patch added a "firm reminder" example to the prompt on 2026-11-16. The world rates:
- The angry path is 1 in 600 connected calls: 1,30,000 / 600, about 217 calls a day.
- After the patch, 1 in 40 angry calls produced a threat: about 5.4 threat calls a day, or 1 in 24,000 connects.
- The judge saw 2% of calls uniformly: 5.4 x 0.02, about 0.11 threat calls a day reached it.
- Over three days, about 16.25 threat calls happened, so the chance the judge saw at least one was 1 - 0.98^16.25, about 0.28.
The lexicon ran on 100% of calls and had "police" and "jail" but not "ghar aa".
Diagnosis: the sample was spread evenly over traffic while the risk sat on 1 in 600 calls. A stratified plan replaced it on 2026-11-19.
What drifts in a voice deployment?
Five things move while the agent's code stands still: the accent and region mix, carriers and codecs, seasonal noise, silent STT and TTS vendor updates, and the code-switching mix. Your own prompt patches are the sixth. Each shifts a slice before the total, so watch the mix and the per-slice metric separately.
| Source | What changes | First signal to watch |
|---|
| Accent and region mix | A campaign reaches a new state; older MFI borrowers with Kannada-accented Hindi take a larger share | Entity error rate and false-cutoff rate per language group, next to each group's share of calls |
| Carriers and codecs | A carrier changes its interconnect and adds a transcoding hop; codec_chain gains an entry | STT final delay and entity error rate per carrier and codec chain |
| Noise seasons | Festival weeks, weddings, monsoon rain on a roof, traffic at the evening peak | Barge-in false triggers, talk-over rate, early hang-ups by hour |
| STT and TTS vendor updates | The model behind the same endpoint changes; a normalization default flips | A step in entity error rate or TTS round-trip failures on the day of a version change |
| Language and code-switch mix | More borrowers switch clause by clause, as P12 does | Clarification loops (LOOP.CLARIFY), STT language-ID errors, by code-switch density |
| Your own changes | A prompt patch, a turn-detector threshold, a new tool | Every guardrail, split by version, starting the hour it ships |
Split input drift from performance drift. Input drift is a change in who calls and over what line (the share of Jio traffic, Hindi-dominant borrowers, AMR-NB legs); performance drift is a change in a metric within one slice. A corpus entity error rate can stay flat while the Hindi-dominant slice worsens, if that slice's share falls the same week. The general methods are in drift detection; for voice, track carrier, codec chain, language group, code-switch density, hour, campaign and an SNR estimate per call.
Pin vendor versions in the trace. Every trace carries versions.stt and versions.tts. A vendor that updates its model behind the same version string shows up as a step with no version change, so keep a frozen set of real audio and rescore it weekly through the live endpoint.
Language-mix drift is the one a regression suite cannot see by construction. One team's suite of 312 recorded production calls all passed, and then a caller switching between English and Hindi mid-sentence caused a 3am page, because the pattern appeared zero times in the 312 calls 1. Tarazu's T08 is the same failure in Hinglish. Slicing by code-switch density, owned by the normalization chapter, shows the new mix arriving.
Which signals belong on the dashboard?
Percentiles and rates per slice, never averages: P95 platform turn gap by turn type, false-cutoff and talk-over rates, barge-in recovery, entity error rate on audited calls, the AMD label mix, early hang-ups, check failures and judge-flag rates by language. Each panel names its denominator and interval.
A mean hides the tail. T04's 3,990 ms gap happened only on tool turns above about 1,800 concurrent calls, a small share of the day's turns, so a daily mean barely moves. Latency terms follow the latency chapter; the turn-taking failures are those of the turn-taking chapter.
| Signal | Definition | Slice by | Interval |
|---|
| P95 platform turn gap | Caller vad_end_ms to agent play_start_ms, per turn type (tool turns apart from greetings and readbacks) | Campaign, carrier, hour, pipeline | Bootstrap clustered by call |
| Mouth-to-ear turn gap | From the dual-channel recording on a daily sample | Carrier | Bootstrap clustered by call |
| False-cutoff rate | Agent took the floor during a hold pause; on 100% from the timeline as a proxy (caller speech resumes over the agent's first audio), confirmed on audits | Language group, call state | Bootstrap clustered by call |
| Talk-over rate | Agent audio over caller speech for more than 300 ms while the caller held the floor | Language group, noise estimate | Bootstrap clustered by call |
| Barge-in recovery | Of detected barge-ins, the share where the agent stopped within 500 ms and its next turn reflected the new information | Flow group | Bootstrap clustered by call |
| Entity error rate | Amounts and dates against human reference transcripts on audited calls | Language group, carrier | Bootstrap clustered by speaker |
| AMD label mix | Share of each of the six AMD labels | Carrier, hour | Compare with the labelled weekly sample |
| Early hang-up rate | Caller ends the call under 10 s after the opener | Campaign, hour | Wilson |
| Check failures | Rate per check, C01 to C16, on 100% of calls | Campaign, version | Count, plus a rule-of-three bound when zero |
| Judge-flag rate | J1, J2, J3 flags per judged call, reweighted to the population | Language group, stratum | Stratified, never pooled |
- The AMD label mix is the early warning for T07. A carrier's new busy announcement shifts calls into the human label; the denominator of PTP per connect inflates and nothing errors. The confusion matrix belongs to the task-success chapter.
- Entity error rate needs a reference, so it lives on the audited calls. Tarazu's 400 hand audits a week support a weekly trend per language group, with wide intervals (definition in the entity chapter).
- Speech-to-speech calls have no LLM TTFT or TTS TTFB. For
pipeline: "s2s" the dashboard shows those panels as missing, never as zero (see voice agent traces).
Above these sits the north star from the metric layers chapter. Kept PTP lags by weeks, so it confirms a drift the guardrails flagged; it never discovers one.
How big must a sample be to catch a rare violation?
The chance of seeing at least one violation in n independent calls at rate p is 1 - (1 - p)^n. To reach 95% at 1 in 1,000 you need 2,995 calls; 1,000 calls at 1 in 2,000 see one only 39% of the time. When a sample finds zero, the one-sided 95% upper bound on the rate is about 3/n.
Solving 1 - (1 - p)^n = 0.95 gives n = ln(0.05) / ln(1 - p). At p = 1/1,000 that is 2,994.2, so 2,994 calls give 0.94999 and 2,995 give 0.95004. For small p, n is close to 3/p, because -ln(0.05) is 2.996. Run backwards, zero violations in n calls puts the rate below about 3/n with 95% confidence, the "rule of three" Hanley and Lippman-Hand discuss for zero numerators 2. Three consequences:
- A clean week on a small sample is a weak statement. Zero in 300 audited calls bounds the rate at about 1.0% one-sided; the two-sided Clopper-Pearson upper limit is 1.22%. Say which, and never write "0%".
- Size for the exposure window, not the month. T09 cost about 5 threat calls a day. A plan that needs a month of calling days to see it lets about 140 happen (5.4 x 26); size for a day or two on the paths where a patch can cause harm.
- The formula assumes independent calls. A prompt patch's violations arrive as a burst from the day it ships, so the per-day size is the one that counts.
Figure: Computed from 1 - (1 - p)^n, not measured. Dots mark the 95% points: about 1,497, 5,990 and 29,956 calls a day. At Tarazu's old 2% uniform sample (2,600 a day, synthetic Tarazu data) a 1-in-10,000 violation is seen on a given day about 23% of the time; T09's 1 in 24,000 sits further right still.
Read it as a budget line: seeing 1 in 10,000 with 95% probability each day by uniform sampling takes about 30,000 judged calls a day, 23% of Tarazu's connects. The rare-violation recipe checks each number in code.
How do you sample when the expensive failure is rare?
Run every deterministic check and the lexicon on 100% of calls, send every call in the risk strata to the judges, sample the rest at a low rate, and reweight by stratum before you report any rate. Where the samples go decides detection far more than how many there are.
Layer 1: cheap checks on everything. Deterministic checks cost Rs 0.002 a call in Tarazu's invented price list, Rs 260 a day for all 1,30,000 connects. The C07 lexicon is one of them. After T09 it gained "ghar aa"; the checks themselves are in the compliance chapter.
Layer 2: the judges on the risk strata. From 2026-11-19 every call in five risk strata (angry, complaint, dispute, hardship, window-edge; about 4% of connects, 5,200 a day) goes to the judges, plus 2% of the rest (2,496 a day): 7,696 judged calls a day. Stratum membership comes from signals on 100% of calls: a caller-turn lexicon ("complaint", "RBI", "harassment"), tool events (log_dispute, flag_hardship), local start time and loudness. A call the stratifier misses lands in the 2% remainder, which therefore measures the stratifier's misses.
Worked example: sizing the angry stratum. Tarazu wants the angry path judged at least 200 times a week.
- Angry calls: 217 a day x 6 calling days = 1,300 a week.
- 200 / 1,300 = 15.4% of the stratum would meet the target.
- Judging all 1,300 with J1 costs 1,300 x Rs 0.15 = Rs 195 a week.
At that price the fraction is not worth optimising: Tarazu judges 100%, and threat calls reaching J1 go from about 0.11 a day to about 5.4. J1 v1 is advisory (Hinglish recall 0.833, below the 0.90 bar), so its flags route to human audit. The load and cost chapter prices the policies.
Layer 3: humans. The 400 weekly audits cannot estimate 1 in 24,000; they check judge flags and misses, so every stratum gets enough to measure recall.
Reweight before you report. A stratified sample oversamples risk on purpose, so its raw flag rate is biased. Weight each judged call by the inverse of its sampling fraction (1 for the risk strata, 50 for the 2% remainder) and divide by the population. With T09's rates:
- Naive rate: 5.42 threats / 7,696 judged calls = 1 in about 1,420.
- True rate: 5.4 / 1,30,000 = 1 in 24,000.
- The naive number is about 17 times too high. The weighted estimate, 1/40 in the angry stratum times its share of 217 / 1,30,000, recovers 1 in 24,000.
Use stratified variance or a bootstrap within strata for the interval, and correct judge scores with audit labels (prediction-powered inference, in the judges chapter). A dashboard that compares a stratified rate with last month's uniform one reports a 17-fold "spike" the day the plan changes. The text-agent version is sampling strategies.
How do you turn a production failure into a regression case?
Cut the call to the turns that matter, redact it, write a scenario card, confirm the scenario fails on the agent version that produced the incident, then add it to the next gold release. A derived scenario that passes on the old agent has not captured the problem and cannot prove a fix.
flowchart LR
A["Incident call, e.g. T09"] --> B["Cut: failing turn plus the context that set it up"]
B --> C["Redact: audio, STT text, tool args"]
C --> D["Scenario card: persona, audio, script, gold end state, forbidden utterances"]
D --> E{"Fails on the incident version?"}
E -- "no" --> F["Add missing detail: audio, state, wording"]
F --> D
E -- "yes" --> G["Fix ships; case passes at the required run count"]
G --> H["Next gold release after gold-v7, with lineage"]
Cut and redact. Keep the failing turn and what set it up: for T09, the opener, verification, the complaint line and the CRM state. Use the redacted channel and a synthetic account; whether a real call may become a test asset is a question for the compliance chapter.
Write the scenario card. Persona (P11), audio conditions, a content-triggered interruption script, gold CRM end state, required utterances (the grievance officer's name and phone), forbidden utterances (any consequence statement), language mix and code-switch density. The simulation chapter builds the caller.
Confirm it fails first. LiveKit's simulation docs give the rule plainly: run the derived scenario before changing anything, because one that passes on its first run has not captured the problem and cannot confirm a later fix. When a derived scenario passes but production failed, they list three usual causes: audio (transcription, turn-taking or interruption handling, which a text run does not exercise), state, and missing detail 3.
T09 adds a trap that the rule alone does not cover: the failure is stochastic. If the replayed prefix reproduces the threat only at the population rate of 1 in 40, a single run fails 2.5% of the time and ten runs fail at least once 22% of the time. To see at least one failure with 95% probability takes n = ln(0.05) / ln(39/40) = 118.3, so 119 runs. As text runs at Rs 0.50 that is Rs 59.50; as audio runs at Rs 6, Rs 714. Use text runs for fail-first and audio in the nightly suite.
Gate the fix at a run count, not one green run. Zero threats in 119 runs on the fixed prompt bounds the per-run rate below about 2.5% (3/119 is 2.52%; the exact one-sided bound, 1 - 0.05^(1/119), is 2.49%): it beat 1 in 40, no more. Run counts and pass^k belong to the statistics chapter.
Add it to the next gold release. Tarazu's rule: within 2 working days, with lineage to the trace id. See regression sets, regression suites and incident response.
How do you avoid peeking errors in canaries?
Watch a canary with a confidence sequence or always-valid p-values, not with a fixed-horizon test checked every hour. Repeated looks at a fixed 95% interval inflate the false-alarm rate far past 5%; a confidence sequence stays valid whenever you stop, at the cost of a wider interval at any one moment.
Johari, Pekelis and Walsh show that the usual fixed-horizon p-values and confidence intervals become unreliable when people keep checking a test and stop when it looks good, and they define always-valid p-values and confidence intervals that stay valid whenever you stop (published in Operations Research in 2022) 4. A confidence sequence, in Howard, Ramdas, McAuliffe and Sekhon's definition, is a sequence of confidence intervals that is uniformly valid over an unbounded time horizon 5. Either lets a canary be watched every hour without inflating the false-alarm rate.
Worked example: what hourly peeking costs. Tarazu runs canaries at 5% of traffic. With 80,080 right-party contacts a day (1,30,000 x 0.88 x 0.70), the canary gets about 4,004 a day, or 364 an hour over the 11-hour calling day. Set PTP per RPC at 38% in both arms (no real difference) and run a two-sided 95% z-test against control every hour for 6 calling days, 66 looks. In 20,000 simulated null canaries (seed 20261120), 34% raised a false alarm at some look. A normal-mixture confidence sequence on the same data, with the boundary |S_t| <= sqrt((V_t + rho) * ln((V_t + rho) / (rho * alpha^2))) tuned for the week's end, raised one in 1.5% of runs. At week's end its interval was about 1.55 times as wide (plus or minus 0.98 points against 0.63), so the same effect needs about 2.4 times the calls. That is the price of looking.
Two limits apply to canaries in voice.
- A canary cannot see a rare-path regression in time. A 5% canary gets about 10.8 angry calls a day. At 1 in 40 it sees a threat on a given day with probability 0.24, and within the week about 0.81. Rare paths need derived scenarios offline before the canary.
- Use the canary for the common metrics. PTP per RPC, P95 turn gap, early hang-ups and talk-over resolve within days. See online versus offline evaluation.
What gate would have caught it?
Three checks together: the C07 lexicon on 100% of calls with "ghar aa" and its variants added, J1 on every angry-path call with its flags routed to human audit (J1 v1 is advisory), and a regression case cut from T09 that fails on collections-v17-p1 before any fix ships.
- Lexicon on 100%. Catches the phrase on the call it happens, at Rs 0.002 a call. Zero hits in a stratum are reported as a rule-of-three bound with n.
- J1 on the whole angry stratum. About 5.4 threat calls a day reach it instead of 0.11. Its flags open a human audit within the day; the audit decides the rollback.
- A derived regression case. Fails on the incident version within 119 text runs, then joins the nightly hardship-and-complaint flow group (F04, F10), which any patch to that path must pass.
The interval rule: zero counts are reported as one-sided upper bounds with their n; stratified rates are reweighted and carry stratified intervals; a canary-vs-control difference is watched with a confidence sequence, not a repeated fixed test.
What goes in the decision memo?
The memo sets the production sampling plan, with a size per stratum and the reason for each.
Decision. From 2026-11-19: deterministic checks and the lexicon on 100% (1,30,000 a day); judges on every risk-stratum call (about 5,200 a day, of which angry about 217) plus 2% of the rest (2,496), 7,696 judged a day; 400 human audits a week across all strata; canaries at 5% of traffic watched with confidence sequences.
Context. T09: about 5 threat calls a day for three days; the 2% uniform judge had about a 28% chance of seeing one.
Options considered. (a) Uniform 2%: cheapest, blind to rare paths. (b) Uniform at the same 7,696 a day: about a 0.6 chance of seeing one in three days. (c) Stratified: chosen; the angry stratum alone puts about 5.4 threat calls a day in front of J1. (d) Judge every call: Rs 23,66,000 a month, 2.4 times the eval cap.
Evidence. Detection arithmetic above. J1 v1 Hinglish validation on gold-v7 (120 calls): recall 0.833, Wilson 0.608 to 0.942, below the 0.90 bar, so advisory. The remainder sees a 1-in-2,000 violation within a week with probability 0.9994 (14,976 calls).
Owner and cadence. Staff engineer, voice platform, with the compliance lead; strata sizes reviewed weekly, plan re-sized monthly.
Tripwires. Any prompt patch that touches a risk path (re-run derived scenarios first). Any violation found in the remainder that a risk stratum should have held. J1 v2 passing validation in a language.
Dated facts. LiveKit derive-a-scenario docs, fetched 2026-11-20.
What to do this week
Pull last week's flags and divide them by the population, not the judged count; if your plan oversamples anything, that one change will move your reported rates the most.
- List every path where a prompt patch could cause a compliance failure, and make each one a stratum judged at 100% while the price allows.
- Replace every average on the dashboard with a P95 or a rate per slice, and put the denominator in the panel title.
- Take your last production incident, cut a scenario from it, and run it on the incident version until it fails. If it never fails, find out which detail you dropped.
- Run the rare-violation recipe with your own volumes and strata.
NOTE
What will date on this page. LiveKit's derive-a-scenario guidance (vendor docs, undated, fetched 2026-11-20). The Marcus Chen post on the 312-call suite (DEV Community, June 2026). The statistics are stable: Hanley and Lippman-Hand (1983), Johari, Pekelis and Walsh (Operations Research, 2022) and Howard et al. (Annals of Statistics, 2021).