At 18:55 Central on 18 November 2026, Derek Wallace, a small-business borrower in Houston, told Hazel he was filing a complaint with the CFPB, and Hazel answered that Steelyard Credit might "have to send someone out to your house" (trace T09). A prompt patch had been producing lines like that for three days, and the production judge, which then read a uniform 2% of calls, had roughly a 28% chance of hearing a single one of them. This page covers what drifts, what the dashboard shows, how many calls a rare violation needs, where the judge budget goes, and how the call becomes a test.
What happened, and why did monitoring miss it?
A patch made Hazel threaten borrowers on the angry path, which carries 1 in 600 connected calls. About 16 threat calls happened over three days, and a judge reading a uniform 2% of traffic had about a 28% chance of seeing even one, because the sample was spread over all calls while the risk sat on one path.
Excerpt from T09 (P11, flow F10, prompt patch collections-v17-p1):
t_ms ch event / text
caller REF: "I'm filing a complaint with the CFPB. You people call me every single day, this is harassment."
agent TTS: "Sir, if you don't make a payment, legal action may follow, and we may have to send someone out to your house."
caller REF: "Are you threatening me? This call is recorded."
On 16 November the team added a "firm reminder" example to prompt collections-v17. Steelyard's world rates, at 130,000 connected calls a day:
- Angry-borrower calls: 130,000 / 600, about 217 a day.
- With the patch live, 1 angry call in 40 drew a threat: about 5.4 a day, or 1 in 24,000 connects.
- Threat calls reaching the judge at 2%: 5.4 x 0.02, about 0.11 a day.
- Over three days, about 16.25 threat calls, so the judge saw at least one with probability 1 - 0.98^16.25, about 0.28.
The C07 lexicon ran on every call, but it held "police", "jail" and "arrest", not "send someone out".
Steelyard bans the statement by policy. Steelyard collects its own loans in its own name, so the FDCPA and Regulation F largely do not bind it 1 2; it adopts their conduct rules as company policy, and state law such as California's Rosenthal Act, the TCPA and the Consumer Financial Protection Act's ban on unfair, deceptive or abusive practices still apply. How each rule becomes a check belongs to the compliance chapter.
Diagnosis: uniform sampling put the judge where the traffic was, not where the harm was. A stratified plan took over on 19 November.
What drifts in a voice deployment?
The agent's code can stay frozen while six inputs move under it: the accent and region mix, carriers and codecs, seasonal noise, silent STT and TTS model updates, the language and code-switch mix, and the team's own patches. Each shows up first in one slice, so chart every slice's share of calls next to its metric.
| Source | What moves | Watch first |
|---|
| Accent and region mix | A campaign opens in a new state; older SDL borrowers, many of them Spanish-dominant, grow as a share of calls | Entity error rate and false-cutoff rate per language group, beside each group's call share |
| Carriers and codecs | A mobile carrier reroutes its interconnect and adds a transcoding hop, so codec_chain grows by one entry | STT final delay and entity error rate per carrier and codec chain |
| Noise seasons | Holiday shopping crowds, storms on a car roof, the evening commute | Barge-in false triggers, talk-over and early hang-ups by hour |
| STT and TTS vendor updates | A new model ships behind the same endpoint, or a normalization default changes | A step in entity error rate or TTS round-trip failures on the release day |
| Language and code-switch mix | More borrowers switch every clause, the way P12 Valeria Cruz does | Clarification loops (LOOP.CLARIFY) and language-ID errors by code-switch density |
| Your own changes | A prompt patch, a detector threshold, a new tool | Every guardrail split by version, from the hour of release |
Keep two kinds of drift apart. Input drift changes who calls and on what line (the share of one carrier, of Spanish-dominant borrowers, of AMR-NB legs); performance drift changes a metric inside one fixed slice. The two can cancel: if the Spanish-dominant slice worsens in the week its share drops, the corpus entity error rate barely moves. The general detectors are in drift detection; for voice, tag each call with carrier, codec chain, language group, code-switch density, local hour, campaign and an SNR estimate.
Record vendor versions, and distrust them. Each trace stores versions.stt and versions.tts, but a vendor can swap the model behind the same string. Push a frozen set of real calls through the live endpoint every week.
A recorded regression suite cannot see language-mix drift, since it holds only the past mix. In a June 2026 post, an engineer described 312 recorded production calls that all passed, followed by a 3am page when a caller switched between English and another language mid-sentence; that pattern occurred zero times in the 312 3. Steelyard's T08 is the Spanglish version. Slicing by code-switch density, taught in the normalization chapter, makes the new mix visible the week it arrives.
Which signals belong on the dashboard?
Per-slice rates and percentiles, never means: P95 turn gap per turn type, false-cutoff and talk-over rates, barge-in recovery, entity error rate on audited calls, the AMD label mix, early hang-ups, check failures, judge flags by language, and calls after a stop request. Each panel states its denominator and interval.
Averages bury the tail: T04's 3,990 ms gap occurred only on tool turns above about 1,800 concurrent calls, so the daily mean hardly shifts. Terms follow the latency chapter and the turn-taking chapter.
| Signal | Definition | Sliced by | Interval |
|---|
| P95 platform turn gap | Caller vad_end_ms to agent play_start_ms, per turn type, with tool turns apart from greetings and read-backs | Campaign, carrier, hour, pipeline | Bootstrap clustered by call |
| Mouth-to-ear turn gap | Measured on the dual-channel recording for a daily sample | Carrier | Bootstrap clustered by call |
| False-cutoff rate | Agent took the floor in a hold pause; a timeline proxy on 100% (caller speech resumes over the agent's first audio), confirmed on audited calls | Language group, call state | Bootstrap clustered by call |
| Talk-over rate | Agent audio over caller speech for more than 300 ms while the caller held the floor | Language group, noise estimate | Bootstrap clustered by call |
| Barge-in recovery | Share of detected barge-ins where Hazel stopped within 500 ms and her next turn used the new information | Flow group | Bootstrap clustered by call |
| Entity error rate | Amounts and dates against human reference transcripts | Language group, carrier | Bootstrap clustered by speaker |
| AMD label mix | Share of each of the six AMD labels | Carrier, hour | Against the labeled weekly sample |
| Early hang-up rate | Caller hangs up within 10 s of the opener | Campaign, hour | Wilson |
| Check failures | Rate per check, C01 to C16, on every call | Campaign, version | Count; rule-of-three bound when zero |
| Judge-flag rate | J1, J2 and J3 flags per judged call, reweighted to the population | Language group, stratum | Stratified, never pooled |
| Calls after a stop request | Attempts to a person after a logged revocation event; open revocations by business days elapsed | Campaign, language | Count, listed call by call |
- The AMD mix warns of T07. A call-screening assistant labeled human swells the PTP-per-connect denominator and no error fires (task-success chapter).
- Entity error rate needs a reference, so it rides on audited calls. 400 hand audits a week give a weekly trend per language group, with wide intervals (entity chapter).
- Speech-to-speech calls have no LLM TTFT or TTS TTFB. For
pipeline: "s2s" those panels show missing data, never zero (voice agent traces).
Stop requests get their own panel, whatever the frequency count says. Regulation F's frequency presumption counts placed calls: seven within seven days, and none for seven days after a conversation 4. Steelyard follows it as policy, but the cap is not enough. The CFPB's 2025 FDCPA report describes collectors who placed over 100 calls to consumers who had asked them to stop; the calls stayed within the frequency limits, yet examiners found the volume had the effect of harassing the consumers 5. So every call after a logged "stop calling me" or "deje de llamarme" [stop calling me] is listed one by one, not as a rate. The TCPA consent-revocation rule, which takes effect on 25 November 2026, keeps the duty to honor a revocation within a reasonable time of no more than ten business days 6. Steelyard's policy is immediate; the panel still counts business days on each open revocation, so a dialer bug shows up long before the legal limit.
Above all of this sits the north star from the metric layers chapter. Kept PTP arrives weeks late, so it confirms a drift the guardrails already flagged and never finds one first.
How big must a sample be to catch a rare violation?
With n independent calls and a violation rate p, the chance of seeing at least one is 1 - (1 - p)^n. At 1 in 1,000, reaching 95% takes 2,995 calls, and 1,000 calls at 1 in 2,000 find one only 39% of the time. A sample with zero violations bounds the rate at about 3/n, one-sided, with 95% confidence.
Set 1 - (1 - p)^n equal to 0.95 and solve: n = ln(0.05) / ln(1 - p). For p = 1/1,000 the answer is 2,994.2, and the integer matters: 2,994 calls give 0.94999, 2,995 give 0.95004. Since -ln(0.05) is 2.996, n sits close to 3/p for small p. Turned around, zero violations in n calls says the rate is below about 3/n with 95% confidence; Hanley and Lippman-Hand's paper on zero numerators is the usual reference for this rule of three 7, and the numbers here are derived, not quoted. What follows for a monitoring plan:
- "Zero this week" on a small sample says little. No violations in 300 audited calls puts the one-sided 95% bound near 1.0%; the two-sided Clopper-Pearson upper limit is 1.22%. Name the bound you report, and never print "0%".
- Size the sample to the exposure window. T09 ran at about 5 threat calls a day. A plan that needs a month of calling days to see one lets about 140 through (5.4 x 26). On paths where a patch can do harm, size for one or two days.
- Violations from a patch are not independent. They start together on release day and continue at a steady rate, so the daily sample size is the one to budget.
Figure: Computed from 1 - (1 - p)^n, not measured. The dots are the 95% points: about 1,497, 5,990 and 29,956 calls a day. Synthetic Steelyard data: at the old 2% uniform sample of 130,000 connects (2,600 a day), a 1-in-10,000 violation is seen on a given day about 23% of the time, and T09's 1 in 24,000 lies further right.
As a budget line: catching a 1-in-10,000 violation with 95% probability each day by uniform sampling means judging about 30,000 calls a day, 23% of connects. The rare-violation recipe reproduces each number in code.
How do you sample when the expensive failure is rare?
Put every deterministic check and the lexicon on all calls, send every call in a risk stratum to the judges, sample the remaining traffic thinly, and reweight by stratum before any rate leaves the team. Where the judged calls come from matters far more than how many there are.
Layer 1: cheap checks on 100%. In Steelyard's invented price list a deterministic check costs $0.0001 a call, about $13 a day across 130,000 connects. The C07 lexicon is one of these checks. After T09 it gained "send someone out" and its Spanish counterpart "mandar a alguien" [send someone]; the full check list sits in the compliance chapter.
Layer 2: judges on the risk strata. Since 19 November, every call in six risk strata (angry, complaint, dispute, hardship, stop request and window-edge; about 4% of connects, 5,200 a day) goes to the judges, plus 2% of everything else (2,496 a day), for 7,696 judged calls a day. Membership comes from signals present on every call: words in caller turns ("complaint", "CFPB", "harassment"), tool events such as log_dispute, flag_hardship and set_do_not_call, the local start time and loudness. A risky call the stratifier misses falls into the 2% remainder, which therefore measures the stratifier's misses.
Worked example: how much of the angry stratum to judge. The target is at least 200 judged angry-path calls a week.
- Angry calls per week: 217 a day x 6 calling days = 1,300.
- Fraction needed: 200 / 1,300 = 15.4% of the stratum.
- Cost of judging all 1,300 with J1: 1,300 x $0.0075 = $9.75 a week.
At that price Steelyard judges the whole stratum, and threat calls in front of J1 rise from about 0.11 a day to about 5.4. J1 v1 is advisory: it failed its Spanglish validation (recall 0.833, below the 0.90 bar), so a J1 flag opens a human audit and never gates on its own. What each policy costs is worked in the load and cost chapter.
Layer 3: people. A 400-call weekly audit cannot estimate a 1-in-24,000 rate; it checks judge flags and unflagged calls so every stratum yields enough labels to measure judge recall.
Reweight before you report. A stratified sample overweights risk on purpose, so its raw flag rate is biased upward. Give each judged call the inverse of its sampling fraction as a weight (1 in the risk strata, 50 in the 2% remainder) and divide by the population. With T09's rates:
- Raw rate: 5.42 threats / 7,696 judged calls, about 1 in 1,420.
- True rate: 5.4 / 130,000, 1 in 24,000.
- The raw figure is about 17 times too high. Weighting recovers the truth: 1 in 40 inside the angry stratum, times that stratum's share of 217 / 130,000, is 1 in 24,000.
Compute the interval with stratified variance or a bootstrap inside each stratum, and correct judge scores against audit labels with prediction-powered inference (derived in the judges chapter). A chart comparing the new raw rate with last month's uniform one shows a 17-fold "spike" the day the plan changes. The text-agent treatment is sampling strategies.
How do you turn a production failure into a regression case?
Trim the call to the turns that matter, redact it, write it up as a scenario card, show that the scenario fails on the agent version that produced the incident, and only then add it to the next gold release. A derived case that passes on the broken agent has missed the problem and cannot vouch for a fix.
flowchart LR
A["Production incident, e.g. T09"] --> B["Trim: failing turn and the turns that led to it"]
B --> C["Redact audio, STT text and tool args"]
C --> D["Scenario card: persona, audio, script, gold end state, forbidden lines"]
D --> E{"Fails on the incident version?"}
E -- "no" --> F["Restore the missing detail: audio, state or wording"]
F --> D
E -- "yes" --> G["Fix ships; the case passes at the required run count"]
G --> H["Next gold release after gold-v7, with lineage to the trace"]
Trim and redact. Keep the failing turn and what set it up: for T09, the opener, verification, Derek Wallace's complaint line and the CRM state. Use the redacted channel and a synthetic account; whether a real call may become a test asset at all is decided in the compliance chapter.
Write the card. Persona P11; audio conditions; an interruption script triggered by content, not a timer; the gold CRM end state; required lines (the Customer Advocate's name and phone); forbidden lines (any consequence statement); language mix and code-switch density. The caller itself is built in the simulation chapter.
CAUTION
Vendor-reported. Treat it as a hypothesis to test on your own calls. LiveKit's derive-a-scenario page is undated vendor documentation, read on 2026-11-20.
Make it fail before you fix anything. LiveKit's simulation docs advise running a scenario derived from a failed session before changing the agent: if it passes on its first run, it has not captured the problem and cannot confirm a later fix. When a derived scenario passes though production failed, they name three usual causes: audio (transcription, turn-taking or interruption handling, which a text run never exercises), state, and missing detail 8.
T09 adds a wrinkle the rule does not cover: the failure is random. If the replayed prefix produces the threat only at the population rate of 1 in 40, one run fails 2.5% of the time and ten runs show at least one failure only 22% of the time. A 95% chance of at least one failure needs n = ln(0.05) / ln(39/40) = 118.3, so 119 runs. As text runs at $0.025 that is about $2.98; as audio runs at $0.30, $35.70. Do fail-first in text and keep audio for the nightly suite.
Gate the fix on a run count. Zero threats in 119 runs on the fixed prompt bounds the per-run rate below about 2.5% (3/119 is 2.52%; the exact one-sided bound, 1 - 0.05^(1/119), is 2.49%). That proves the fix beats 1 in 40 and nothing more. Run counts and pass^k are covered in the statistics chapter.
Release it with lineage. Steelyard's rule: the case lands in the next gold release within 2 working days, linked to the trace id. See regression sets, regression suites and incident response.
How do you avoid peeking errors in canaries?
Watch canaries with a confidence sequence or always-valid p-values, not a fixed-horizon test re-run every hour. Each extra look at a fixed 95% interval adds a chance of a false alarm, so the total climbs far past 5%. A confidence sequence holds at any stopping time but is wider at each moment.
Johari, Pekelis and Walsh show that ordinary fixed-horizon p-values and confidence intervals stop being reliable when people keep checking a test and stop once it looks good; their always-valid p-values and intervals remain valid whenever the test is stopped (Operations Research, 2022) 9. Howard, Ramdas, McAuliffe and Sekhon define a confidence sequence as a sequence of confidence intervals that is uniformly valid over an unbounded time horizon 10.
Worked example: the cost of looking every hour. Steelyard runs canaries on 5% of traffic. Right-party contacts are 130,000 x 0.88 x 0.70 = 80,080 a day, so the canary gets about 4,004 a day, about 250 an hour across the 16-hour dialing day that runs from 08:00 Eastern to 21:00 Pacific. Give both arms the same PTP per RPC, 38%, so there is no real effect, and run a two-sided 95% z-test against control every hour for 6 calling days: 96 looks. In 20,000 simulated null canaries (seed 20261120), 37% raised a false alarm at some look. A normal-mixture confidence sequence on the same runs, with boundary |S_t| <= sqrt((V_t + rho) * ln((V_t + rho) / (rho * alpha^2))) tuned for the end of the week, raised one in 1.5%. At the end of the week its interval was about 1.55 times as wide (plus or minus 0.98 points against 0.63), so detecting the same effect takes about 2.4 times the calls. That is the price of looking.
Two limits for voice canaries:
- Rare-path regressions are too sparse for a canary. A 5% canary sees about 10.8 angry calls a day. At 1 in 40 it catches a threat on a given day with probability 0.24, and within the week with about 0.81. Rare paths need derived scenarios offline, before any canary starts.
- Give the canary the common metrics. PTP per RPC, P95 turn gap, early hang-ups and talk-over settle within days. See online versus offline evaluation.
What gate would have caught it?
Three checks, run together: the C07 lexicon on every call with "send someone out", "mandar a alguien" [send someone] and their variants added; J1 on the whole angry stratum with every flag sent to a human (J1 v1 is advisory); and a regression case cut from T09 that fails on collections-v17-p1 before any fix ships.
- Lexicon on 100%. Catches the phrase on the very call, at $0.0001 a call. A stratum with zero hits is reported as a rule-of-three bound next to its n.
- J1 on the full angry stratum. About 5.4 threat calls a day reach it instead of 0.11. A flag opens a human audit the same day, and the audit decides on a rollback.
- A derived regression case. Fails on the incident version within 119 text runs, then joins the nightly hardship-and-complaint flow group (F04, F10), which every patch touching that path must pass.
Interval rule: report zero counts as one-sided upper bounds with their n; reweight stratified rates and give them stratified intervals; follow a canary-versus-control difference with a confidence sequence, never with a repeated fixed test.
What goes in the decision memo?
The memo fixes the production sampling plan: a size for each stratum and a reason for each size.
Decision. From 2026-11-19: deterministic checks and the lexicon on 100% (130,000 calls a day); judges on every call in the six risk strata (about 5,200 a day, of which about 217 angry) and on 2% of the rest (2,496), 7,696 judged a day; every call after a stop request listed for review; 400 human audits a week across strata; canaries at 5% of traffic, watched with confidence sequences.
Context. T09: about 5 threat calls a day for three days; the 2% uniform judge had about a 28% chance of seeing one.
Options considered. (a) Uniform 2%: cheapest and blind to rare paths. (b) Uniform at the same 7,696 a day: about a 0.6 chance of one sighting in three days. (c) Stratified: chosen; the angry stratum alone puts about 5.4 threat calls a day in front of J1. (d) Judge every call: $118,300 a month, 2.4 times the $50,000 cap.
Evidence. Detection arithmetic above. J1 v1 Spanglish validation on gold-v7 (120 calls): recall 0.833, Wilson 0.608 to 0.942, under the 0.90 bar, so advisory. The remainder alone sees a 1-in-2,000 violation within a week with probability 0.9994 (14,976 calls).
Owner and cadence. Staff engineer for the voice platform, with the compliance lead; stratum sizes reviewed weekly, the plan re-sized monthly.
Tripwires. Any prompt patch touching a risk path (run the derived scenarios first). Any violation found in the remainder that a stratum should have caught. Any call placed after a stop request. J1 v2 passing validation in a language group.
Dated facts. LiveKit derive-a-scenario docs, fetched 2026-11-20; TCPA revocation rule effective 2026-11-25.
What to do this week
Take last week's judge flags and divide by the population instead of the judged count. If your plan oversamples anything, no other change moves your reported rates as much.
- List each path where a prompt patch could cause a compliance failure, and judge each as its own stratum at 100% while the cost allows.
- Swap every average on the dashboard for a P95 or a per-slice rate, and put the denominator in the panel title.
- Query your call log for any attempt placed after a stop request, and make that query a standing panel.
- Take your most recent production incident, cut a scenario from it and run it on the incident version until it fails. If it never fails, find the detail you dropped.
- Run the rare-violation recipe with your own volumes and strata.
NOTE
What will date on this page. LiveKit's derive-a-scenario guidance (vendor docs, undated, fetched 2026-11-20). The 312-call incident post (DEV Community, June 2026). The CFPB's FDCPA Annual Report 2025 (November 2025, covering 2024). The TCPA consent-revocation rule takes effect on 25 November 2026; re-read it after that date. Regulation F 12 CFR 1006.14 as read on eCFR in November 2026. The statistics are stable: Hanley and Lippman-Hand (1983), Johari, Pekelis and Walsh (Operations Research, 2022) and Howard et al. (Annals of Statistics, 2021).