Between 10 and 18 November 2026, Hazel failed ten times in ten different ways: she stored a promise for $215 when the borrower said $250 (T01), kept talking over a borrower who had already paid (T02), told a borrower's brother about his past-due auto loan (T03), went silent for 3,990 ms at peak (T04), and six more. Read that log as an acceptance test: a platform design passes only if every trace has a named component, a gate and an expected time to detection. Earlier chapters own the methods; this one connects them, prices them, splits build from buy, and ends with the capstone.
All ten of Hazel's logged failures, T01-T10, each through a check whose gate and cadence set how fast it fires. Most leave evidence in text, tool order or CRM state, so near-free checks on every call fire within minutes of hang-up; the rest live in audio, timing or load and need nightly or pre-release suites.
| Trace | What broke | Owning component | Check | Gate | Time to detection |
|---|
| T01 | "two fifty" transcribed as "two fifteen"; promise stored before any read-back | STT entity suite; production checks | Amount entity error rate; C12 write order | Nightly upper bound; C12 blocks | C12 before merge; minutes in production |
| T02 | "ya lo pague" [I already paid it] talked over for 900 ms, plan unchanged | Audio simulation; production checks | Stop latency and recovery; C08 tool sequence | Nightly; C08 blocks | Next nightly run; minutes |
| T03 | Product, days past due and amount told to the borrower's brother | Wrong-party personas; production checks | C03 opener, C04 ordered events, C05 contact role | A single failure blocks | Before merge; first affected call |
| T04 | 3,990 ms tool turn, then a doubled reply, above about 1,800 concurrent calls | Load generator; production checks | Tool-turn P95 turn gap at 1.0x and 1.5x of peak; duplicate-reply check | Pre-release load gate | Before release; minutes |
| T05 | Turn closed inside a 720 ms pause just before a date | Endpointing suite | False-cutoff rate at the campaign budget | Nightly; reruns on any detector or budget change | Next nightly run |
| T06 | "12/03" meant 3 December; the es-US voice said "doce de marzo" [twelfth of March] | Phone-path TTS suite | Spoken-entity round trip per locale | Nightly; reruns on TTS or prompt change | Next nightly run |
| T07 | Call-screening assistant labeled human at 0.71 | AMD suite | Six-class confusion matrix, each error costed | Pre-release; reruns on AMD change | Before release |
| T08 | Spanglish turn decoded as English; three clarifications, then a hang-up | Audio simulation; production checks | Code-switching callers; LOOP.CLARIFY | Nightly; monitored in production | Next nightly run; minutes |
| T09 | "send someone out to your house" on the angry path | Lexicon on 100%; J1 on every angry call | Lexicon hit or J1 flag sent to human audit; regression case | Audit confirms | Same day |
| T10 | Money request straight after "I lost my job", over crying | Production checks; J2 | C09 tool order; J2 pacing (advisory) | C09 blocks | Minutes |
Three traces have two independent catchers: T01 (entity suite and C12), T03 (wrong-party simulation and C04) and T09 (lexicon and judge). T09 is why: the lexicon held "police", "jail" and "arrest" but not "send someone out", so the second chance was a judge on every angry call.
Three planes: data (each call on one clock), eval (scoring) and control (gold sets, judge registry, gate logic).
flowchart LR
subgraph DP["Data plane: every Hazel call"]
A["Dialer with attempt counter, SIP trunk with signed caller ID"] --> B["Hazel: STT, turn detector, LLM, tools, TTS"]
B --> C["Dual-channel recording"]
B --> D["Spans and events on one clock"]
B --> E["CRM state before and after"]
end
subgraph EP["Eval plane"]
F["Trace store: redacted copies, spans without content"]
G["Deterministic checks C01-C16 on every call"]
H["Judges J1-J3 on risk strata plus 2% of the rest"]
I["Human audit, 400 calls a week"]
J["Simulation fleet and offline suites"]
end
subgraph CP["Control plane"]
K["Gold set releases gold-v n"]
L["Judge registry"]
M["Gate logic and release record"]
end
C --> F
D --> F
E --> F
F --> G
F --> H
H --> I
K --> J
J --> M
G --> M
L --> H
I --> L
What is the data model?
The unit is the call, on a single clock: milliseconds since media start at the agent's ingress. Nine record types, each keyed so a reviewer can go from a failed check to the audio offset behind it, and each carrying the version vector of every component that touched the call.
Steelyard's call record is the schema from the traces chapter, down to the dial block that logs every attempt. On top of it the platform adds keys, a PII class and a retention class:
| Record | Key | Contents | PII class | Retention class |
|---|
| Call | call_id | Campaign, pipeline, codec chain, AMD label, dial outcome, versions | Metadata | Metric |
| Turn | (call_id, turn_id) | Speaker, turn type, VAD times, STT final, reference text | Content | Eval |
| Span | (call_id, turn_id, stage) | Start and end in ms; never text | None | Metric |
| Audio segment | (call_id, channel, start_ms, end_ms) | Pointer to raw or redacted audio | Content | Raw or eval |
| Tool call | (call_id, turn_id, seq) | Tool name, redacted arguments, result, timings | Content | Eval |
| CRM diff | call_id | Before and after JSON, amounts in cents | Content | Eval |
| Check result | (call_id, check_id, check_version) | Verdict, evidence turns, policy pack | None | Metric |
| Label | (call_id, turn_id, labeler, taxonomy) | Failure code from gl-fail-v4, note | Reference | Eval |
| Judge score | (call_id, judge_id@version, dimension) | Score, rationale, input hash | Reference | Metric |
Steelyard keeps raw recordings three years from the call date as company policy, and plays the recording notice at the start of every call: federal wiretap law needs one party's consent, but some states, California among them for confidential communications, require every party's 1 2, and the stricter state's law can apply on an interstate line. On the UK and Spain books, recordings and transcripts are personal data under UK and EU GDPR, and a voiceprint extracted to identify a speaker is special category data 3, so Hazel never authenticates by voice. Eval copies are redacted at ingest and kept while a gold release references them. Metric records hold no content, so strip the transcripts and tool arguments that agent frameworks put inside latency spans by default.
Task success is the CRM state diffed against the scenario's gold end state, tau-bench's final-database comparison 4; T01's diff reads ptps.amount_cents expected 25000 got 21500. Spoken checks (did Hazel name the Customer Advocate) stay separate results: tau-Voice folds an LLM evaluator of spoken communication into its task score, so that score is not fully deterministic 5. A diff and a judged check fail for different reasons; never merge them. Versioned check results mean a change to C06 adds results instead of rewriting last month's compliance rate.
How do you version gold sets?
As immutable releases named gold-v{n}, each with recorded lineage: source, consent, redaction version, labelers and adjudication. Compare metrics only inside one release, and rerun the incumbent agent on every new release so each comparison stays paired.
gold-v7 holds 600 scenario cards, 1,200 caller utterances with reference transcripts from 180 speakers (Spanish references keep their accents), 500 TTS lines of which 400 are amounts, 400 labeled pauses per language group, 120 human-labeled calls per language group per judged dimension, and 200 labeled AMD openings across the six classes, carrier intercept and call-screening assistant included. The general method is in dataset versioning and lineage. Voice adds three rules.
Incidents arrive as failing cases. Within two working days each production incident becomes a scenario card that must fail on the version that caused it; a card that passes has not captured the failure.
Cover flows and parties, not volume. T03 got through regression because every gold scenario was answered by the borrower. Each gated language group (English, Spanish-dominant, Spanglish) needs all of F01-F10, which means a family member on the borrower's phone, a guarantor, voicemail, and machine openings.
A new language group starts as a gold-set project. Spain Spanish is a good example. In CLDR the string 1.234 means about one in Mexico and one thousand two hundred thirty-four in Spain 6, so a scorer tuned on es-US speech will canonicalize Spain's amounts wrongly. Any new group needs 400 labeled pauses, 1,200 utterances from at least 150 speakers and 120 labeled calls per judged dimension before a gate in that group means anything. Releases are cut monthly and never in the middle of a comparison.
What runs per commit, nightly and before release?
Four tiers, each priced by what a run costs and sized by what its gate's statistics need: text scenarios on every commit, phone-path audio every night, load and the paired segment gate before a release, then a production canary.
CI integration covers the plumbing, and release gates covers the decision logic.
| Tier | Contents | Size | Gate | Cost per month |
|---|
| Per commit | Text scenarios plus every deterministic check | 600 at k=2, about 10 commits a day | Any C01-C16 failure blocks; pass^2 per flow against the incumbent | $6,600.00 |
| Nightly | Phone-path audio scenarios; STT and TTS entity suites; endpointing; AMD | 240 at k=4, 20 per gated segment | The interval rules below | $8,640.00 |
| Pre-release | Load at 1.0x and 1.5x of projected peak, a 3x spike, the paired segment gate | 12 segments | Holm across segments | $5,000.00 |
| Canary | 5% of live traffic | Until the confidence sequence decides | North star and guardrails; one compliance failure halts | Production cost |
Text runs go on every commit because an audio scenario costs twelve times as much ($0.30 against $0.025). They catch prompt and tool-order regressions such as T03's personalised opener, never the audio-only failures: T01's misheard amount (its write order is C12's job), T05 or T06.
Each gate is an interval rule, using the statistics from the reliability chapter:
- Amount and date entity error rate: the bootstrap upper bound, clustered by speaker, stays below the campaign threshold.
- P95 platform turn gap on tool turns: the upper bound, clustered by call, stays at or below the silence budget plus 2,200 ms. That is 2,600 ms on AUTO-early-nov and 2,800 ms on SDL-biwk-nov, using the terms from the latency chapter.
- False-cutoff rate at the campaign budget, per language group and call state, as in the turn-taking chapter.
- pass^4 per flow, paired against the incumbent with McNemar or a paired bootstrap.
- Compliance: one failure blocks.
Multiplicity. Steelyard gates twelve segments: three language groups times four flow groups. Testing each at 0.05 blocks good releases by chance. Holm's step-down method compares the smallest p-value with 0.05/12, about 0.0042, and relaxes the threshold one segment at a time 7.
Load. Ramp to the projected peak, hold there, then run a separate spike test up to three times the busiest hour 8. Today the projected peak is 2,400 concurrent calls. The load chapter sets the stop conditions.
What is in the simulation fleet?
Persona callers that reach Hazel over the production phone path, with impairments, interruptions triggered by what Hazel says, a sandbox CRM to diff, and validators that discard runs where the simulator went wrong. The fleet is checked against real calls quarterly (simulation chapter).
- Scenario cards. Personas P01-P13 in English or Spanish, each with SNR, codec chain, loss profile, an interruption script keyed to content, and a gold CRM end state.
- Audio-path parity. Simulated calls leave through SIP egress, so AMD, the turn detector and STT all hear the 8 kHz mu-law stream that production gets. A test over clean websocket audio is testing a different system.
- Impairments and callees. The condition matrix from the telephony chapter. Callees include borrowers, a brother answering the borrower's phone, voicemail greetings, call-screening assistants, carrier intercepts and IVRs.
- Validators. EVA-Bench validates every simulated call for a valid ending and for the user's behavioral and speech fidelity, then regenerates the ones that fail. Across four systems, 12% of trials were regenerated because of simulator error, nearly all of it behavioral drift 9. If the regeneration rate rises, treat it as a simulator regression.
- Audit. Two raters review 50 failed runs per release. Every quarter, 30 consented real calls per language group get simulated twins, and each twin must fail where the real call failed.
Concurrency. Nightly is 960 audio calls, 144,000 call-seconds at 150 s each: 3 h 20 min at 12 simultaneous calls, 96 minutes at 25. A vendor's concurrency limit decides whether nightly finishes before morning.
Simulation is the best candidate to buy, so read the documented limits before signing.
CAUTION
Vendor-reported. Treat it as a hypothesis to test on your own calls.
From vendor documentation read on 2026-11-20:
- LiveKit. Reports pass@k and pass^k. Outbound agents can be simulated in a room, but testing through a phone number covers inbound calls only. It plays no voicemail greetings or phone menus, and the docs say mixed-language speech runs in the non-English language 10.
- Vapi. Its testing guide says "synthetic callers do not fully represent real callers and their environments" 11.
- Retell. Simulation runs as a text conversation and does not support custom-LLM agents. Voicemail and IVR detection, DTMF and transfers have to be tested with web or phone calls 12. Tests bill at production rates, so 20 cases run three times bill as 60 runs 13.
- Roark. Packet loss is seeded and applied in 20 ms packets, with presets up to 10% loss in 120 ms gaps 14. An
#{interrupt} cue fires at a point the script names 15.
- Coval. Outbound tests use a webhook: your system dials the simulated customer 16. Interruptions fire on a timer, roughly every 90, 45 or 30 seconds, and its Higher Realism voice tier allows only about 12 simultaneous connections 17.
- Cekura. "Evaluator instructions cannot override a personality setting": an instruction to interrupt constantly does nothing, yet the call still returns a result 18. Outbound tests with your own Twilio and Plivo numbers are documented 19. In simulation, its Transcription Accuracy metric scores the testing agent's speech-to-text, not your agent's.
- Bluejay. Voicemail simulation plays a greeting and a beep, for outbound agents only, with at most 100 Digital Humans per test 20. Its self-reported pay-as-you-go plan is $0 a month plus usage, with $25 in credits and up to 25 concurrent simulations 21.
Two requirements settle Steelyard's shortlist: T02 needs an interruption triggered by what Hazel is saying, which no timer produces, and T07 needs a call-screening assistant as the callee.
How does online sampling work?
Deterministic checks and the lexicon read every call. Judges read every call in a risk stratum plus a small share of the rest, and human audits draw from both groups. Reported rates are reweighted by stratum and corrected for judge error, using the design in the monitoring chapter.
Steelyard's plan since T09 judges every angry, complaint, dispute, hardship, stop-request and window-edge call (about 4% of connects, 5,200 a day) plus 2% of the remaining calls (2,496 a day). That is about 7,696 judged calls a day, alongside 400 human audits a week. Under the 2% uniform sample it replaced, the chance of seeing even one threat during the three days the bad prompt patch was live was about 28%.
Sizing for a rare violation. To see a 1-in-2,000 violation within a week with 95% probability, you need 5,990 sampled calls, because 1 - (1 - 1/2000)^5990 = 0.95000. The non-risk sample alone is 2,496 x 6 calling days = 14,976 a week, which gives a probability of 0.9994. That covers both C07 judging and the C04 judge backstop. The C04 ordered-event check itself is deterministic and reads every call, so it needs no sizing.
Reported rates. A judge's raw flag rate on a stratified sample carries two biases, one from the strata and one from the judge's own errors. Report a stratum-weighted prediction-powered estimate, which uses the human-audited subset to correct the judge 22. Online versus offline evaluation covers the wider split.
What is a judge registry?
A versioned record per judge, dimension and language group: inputs, validation against human labels, gating or advisory status, revalidation triggers, owner and cost. Gate logic reads the registry, so a judge not validated in a language group cannot gate a release in that group.
Here is J1's entry after its November 2026 Spanglish validation. The method is in the judges chapter.
{
"id": "J1", "version": "v1", "dimension": "C07 coercion and consequence misstatement",
"input": "agent-turn transcript plus lexicon hits", "owner": "staff engineer, evaluation",
"cost_usd_per_call": 0.0075,
"validation": {
"spanglish": {"gold": "gold-v7", "n_calls": 120, "violations": 18, "caught": 15, "false_alarms": 6,
"recall": 0.833, "recall_wilson95": [0.608, 0.942],
"specificity": 0.941, "specificity_wilson95": [0.878, 0.973],
"precision": 0.714, "kappa": 0.725, "bar": "recall lower bound >= 0.90",
"result": "fail"},
"english": null, "spanish_dominant": null
},
"status": "advisory", "routing": "flags go to human audit",
"revalidate_on": ["judge model or prompt change", "upstream STT or LLM version change",
"taxonomy change", "new language group"],
"anchor_set": "frozen 200-call anchor set"
}
Admit on the lower bound, not the point estimate. J1's recall of 0.833 sounds usable, but its lower bound of 0.608 says it may miss two threats in five, and with 18 violations nothing can clear the bar: even 18 of 18 has a lower bound of about 0.82. So a new language group gets a gating judge only after about 100 natively labeled violations: 97 caught of 100 gives a Wilson lower bound of 0.915, 96 of 100 gives 0.902, barely.
The registry records agreement and kappa but never admits a judge on them. At 3% prevalence, a judge that always answers "compliant" agrees with humans 97% of the time, with a kappa of 0.
What does it cost?
About $45,673.03 a month today, inside the $50,000 cap, at Steelyard's invented unit prices. At next quarter's 2.5x volume the plan reaches about $75,697.57, just over the CFO's $75,000 cap, before any judge for a new language group is added.
Only three lines scale with calls (checks, judges and STT re-transcription); at 2.5x volume they reach about $50,040.90. Unit costs and the pricing of each sampling policy are in the load and cost chapter. All prices are invented for the fictional world and are not any vendor's list price.
| Line | Today | At 2.5x volume |
|---|
| Deterministic checks on every call | $338.00 | $845.00 |
| Judges: J1 and J3 (text), J2 (blueprint), 7,696 calls a day | $7,003.36 | $17,508.40 |
| STT re-transcription on 5% of connects | $12,675.00 | $31,687.50 |
| Human audit, 400 calls a week | $5,416.67 | $5,416.67 |
| Nightly audio simulation, 240 x k=4 | $8,640.00 | $8,640.00 |
| Per-commit text runs, 600 x k=2 | $6,600.00 | $6,600.00 |
| Two pre-release load tests | $5,000.00 | $5,000.00 |
| Total | about $45,673.03 | about $75,697.57 |
Cut STT re-transcription first. It narrows intervals but finds no violations. Moving it from 5% to 3% of connects saves $12,675.00 at the new volume and brings the plan to $63,022.57.
What the saving buys. Adding two gated language groups, UK English for the UK book and Spain Spanish for the Spain book, raises the segment count from 12 to 20 and nightly simulation from 240 to 400 scenarios at k=4, which lifts that line from $8,640.00 to $14,400.00. The plan becomes $68,782.57, leaving about $6,200 for judges and labels in the new groups.
Never cut these. Checks and the lexicon on every call, and judging of every risk-stratum call. Cost per detected violation decides how money moves between lines. It never decides whether a compliance line exists.
What do you build and what do you buy?
Build anything that encodes Steelyard's rules or has to survive leaving a vendor: policy checks, gold sets, scenario cards, the CRM diff, the judge registry and the gate logic. Buy impairment, load generation and labeling from vendors that meet six criteria and sign exit terms.
Steelyard collects its own debts in its own name, so the FDCPA and Regulation F largely do not bind it 23 24; it adopts their conduct rules as policy, and state law (California's Rosenthal Act, Massachusetts 940 CMR 7.00), the TCPA and the Consumer Financial Protection Act's ban on unfair, deceptive or abusive practices still apply. This is not legal advice: Steelyard's checks implement Steelyard's reading of the rules, which is why they are built in-house.
| Component | Decision | Reason | Exit terms |
|---|
| Policy checks C01-C16, CRM diff | Build | They encode Steelyard's reading of Regulation F, the TCPA, state law, CONC and the AI Act | None needed |
| Gold sets, scenario cards | Build | The test set is the asset | None needed |
| Judge registry, gate logic | Build | Every release must be reproducible from Steelyard's own records | None needed |
| Caller simulation | Buy if a vendor passes the pilot; otherwise build | English and Spanish persona voices and SIP plumbing are expensive to build | Scenario cards in Steelyard's format; raw audio export |
| Telephony impairment | Buy | A commodity | Seeded profiles exported with each run |
| Load generation | Buy | A commodity, provided it sustains 40 call starts a second (6,000 / 150 s) | Raw per-call spans |
| Judge models | Buy the model; own the prompt and the validation | Validation is per language group | Prompts and validation sets stay with Steelyard |
| Dashboards, labeling UI | Buy or reuse | A commodity over Steelyard's stores | No metric or label held only by the vendor |
| Listening review tool | Build, unless a vendor shows the timeline with check results | Reviewers need audio, both transcripts and evidence turns in one view | None needed |
Test each criterion on Steelyard's own calls:
- Outbound support: voicemail greetings, call-screening assistants, carrier intercepts and IVRs as callees.
- Spanish and Spanglish fidelity: callers who switch language mid-clause, and vendor STT measured against Steelyard's references.
- Audio-path parity: SIP at 8 kHz through the production media stream.
- UK and EU data handling: recordings and transcripts from the UK and Spain books are personal data, so the normal rules (lawful basis, transparency, storage limitation) follow them into any vendor that stores them, and any voiceprint feature used to identify a speaker puts it in special category data 3.
- Raw-result export: per-call audio, spans, transcripts and verdicts.
- Cost at volume: 960 nightly audio calls and 1,200 text runs per commit, finished overnight.
What changes if a speech-to-speech model replaces the cascade?
Three groups of evals break: per-stage latency attribution, compliance checks that read TTS input text, and turn-taking suites, since the model now decides when to speak. The CRM diff, tool-order checks and gold scenarios carry over. Run hazel-s2s-pilot paired against the cascade as the control.
| Eval | Under the cascade | Under speech-to-speech | What to do |
|---|
| Latency | Per-stage spans: end-of-turn delay, LLM TTFT, tool time, TTS TTFB | The LLM and TTS blocks are missing, not zero | Measure the mouth-to-ear turn gap on the dual-channel recording |
| C03, C04, C13 | Deterministic checks on TTS input text | There is no TTS text, and the transcript can arrive after the reply | Run STT on the agent channel; the check inherits that STT's entity error rate |
| TTS entity round trip | TTS text compared with the tool result | No text to compare | Compare agent-channel STT entities with get_dues |
| Endpointing and barge-in | Silence budgets per campaign | The model decides when to speak | Rerun the false-cutoff, talk-over and recovery suites |
| Task success | CRM diff | Unchanged | Unchanged |
| Judges | Transcript plus audio features | Transcript from agent-channel STT | Revalidate per language group |
The compliance row is the expensive one. "No debt details before verification" is an exact check on Steelyard's own text in the cascade; under speech-to-speech it reads a transcript of the model's audio, with its own error rate on amounts, so it needs the validation a judge gets.
Barge-in recovery is where real-time models are weakest. In the EchoChain preprint, four models had to revise their answer after a scripted barge-in, and none fully passed more than half of 200 interrupted conversations (best 48.5%, lowest 16.5%) 25. That is T02's failure.
Choosing an architecture does not settle quality. Since late October 2026, EVA's leaderboard has shown a managed cascade, Cartesia Managed Agents with GPT 5.5, leading on both axes (0.87 EVA-A and 0.91 EVA-X pass@1) 26. Compare the two pipelines on Steelyard's scenarios, paired per segment (McNemar or a paired bootstrap), under the same twelve-segment gate.
The design review
The capstone is a design document defended in a review. The course grades the quizzes and the exam; you self-assess the design and memo appendix against the rubric below, with optional peer review. Answer it for either edition: the 15-point incident row scores against the T01-T10 log of the edition you pick, and this brief uses the global one.
You are the staff engineer for evaluation at Steelyard Credit. Next quarter Hazel grows from 5 campaigns to 9 (a second PIL book, an SBL top-up campaign, and the first campaigns for Steelyard Credit UK Ltd and Steelyard Credit Espana S.L.). She grows from three language groups (English, Spanish-dominant, Spanglish) to five, with UK English and Spain Spanish handled by the bot for the first time, and from 2,400 to 6,000 peak concurrent calls. A speech-to-speech model, hazel-s2s-pilot, is being evaluated as a replacement for the cascade. The TCPA consent-revocation rule takes effect on 25 November 2026 27. On the Spain book, AI Act Article 50 has applied since 2 August 2026, and systems already on the market have until 2 December 2026 to mark synthetic audio 28 29. CCD2's forbearance duty applies from 20 November 2026, with transposition still incomplete in several Member States 30. Compliance wants evidence that no call reveals debt details before verification, that calling-window violations are zero, and that every stop request becomes a revocation event. The CFO has capped eval spend at $75,000 a month. Today's plan costs about $45,673 and scales to about $75,698 at the new volume (see the cost section above). Two vendors have pitched a managed testing platform. Write a design of no more than six pages plus an appendix of your memos, and defend it in a 30-minute review with the head of engineering, the chief compliance officer and the CFO.
Cover:
- Problem statement: the three incidents from T01-T10 that you consider most expensive, and why.
- Data model: call, turn, span, audio segment, tool call, CRM diff, check result, label and judge score; the clock; versions; PII classes and retention per jurisdiction; how speech-to-speech traces differ.
- Gold sets: contents, versioning, refresh and incident intake.
- CI gates: per commit, nightly, pre-release and canary; gated metrics as interval rules; per-segment gating and multiplicity.
- Simulation fleet: personas, audio-path parity, impairments, outbound coverage, simulator validation and audit rate, concurrency and cost.
- Online sampling: checks on every call, strata and rates, human audit, how rates are reported, and rare-violation sizing for C07 and for the C04 judge backstop (C04's deterministic check reads every call).
- Judge registry: fields, admission rule, status per language group (UK English and Spain Spanish included), revalidation triggers and ownership.
- Cost budget: monthly cost by line from the unit costs on this page, and cost per detected violation, inside the $75,000 cap.
- Build vs buy for simulation, telephony impairment, load generation, judges, dashboards and labeling, with criteria (outbound support, Spanish and Spanglish fidelity, audio-path parity, UK and EU data handling, raw-result export, cost at volume) and exit terms.
- Speech-to-speech: which evals disappear or must be rebuilt, and how you would run the comparison.
- Incident coverage table: T01-T10, the component and gate that catch each one, and the expected time to detection.
- Evidence from your own work: at least four outputs from the course recipes or sims, for example your endpointing table from the endpointing recipe or your judge registry entry from the judge validation recipe.
How is the design scored?
The capstone is scored on a 100-point rubric. "Meets" is 70 points with no row at zero, and "strong" is 85. Five errors each cost 5 points automatically, regardless of what else the design gets right.
| Row | Points | Meets | Strong |
|---|
| Incident coverage | 15 | Every trace in the chosen edition's log mapped to a component, a gate and a detection time | At least three caught by two independent components; detection times argued from cadence and sample size |
| Audio in the loop | 10 | STT, TTS, turn-taking and simulation evaluated on phone-path audio; wideband results never gate | A real 8 kHz holdout, live impairment tests, caller-heard latency taken from recordings |
| Entity and task metrics | 10 | Entity error rate on amounts and dates gated with intervals; CRM-diff task scoring per flow | Spoken checks kept apart from the diff; confirmation polarity; verified PTP |
| Latency and turn-taking | 10 | One vocabulary; P95 per turn type; an endpointing point per language group and campaign | Budgets per call state, consequence weighting, pinned SDKs, speech-to-speech implications |
| Statistics and gates | 10 | pass^k by the unbiased estimator; run counts from a power plan; paired gates per segment with Holm; clustered intervals | A canary that does not peek; zero-event bounds for compliance; no Wilson interval on differences or ratio metrics |
| Judge registry | 10 | Validation against human labels per language group and dimension; admission on the lower bound; revalidation triggers | Prevalence handled; gating vs advisory status; PPI for reported rates; no gating judge in a group not yet validated |
| Compliance as code | 10 | C01-C16 or an equivalent with types and inputs; cheap checks on every call; windows that know the product and location | Evidence of express requests, voice threat tests, a redaction audit, dated legal assumptions per jurisdiction |
| Production and rare events | 10 | Stratified sampling with sizing; drift signals; incident-to-regression with fail-first cases | Sizing shown for a 1-in-2,000 violation within a week; seasonal noise and carrier drift |
| Cost and build vs buy | 10 | A line-item budget inside the $75,000 cap from the unit costs on this page; a decision per component with criteria | Cost per detected violation; exit terms; a list of what is never cut |
| Clarity under review | 5 | Six pages; decisions with their trade-offs; open risks named | Every number carries an interval or an as-of date |
Automatic deductions, 5 points each: stating a claim this course corrects in its uncorrected form (for example, pass^8 at 80% given as anything other than 0.8^8, about 17%); a latency number with no start and end point; a judge used for gating without validation per language group; a vendor number with no "self-reported" label or as-of date; a Wilson interval on a difference or on WER.
Reference design (write your own first)
Draft your design before reading this. It is one "strong" design, detailed in the sections above; yours can differ on any decision if you argue the trade-off.
- Most expensive incidents. T03 (a breach of Steelyard policy on every wrong-party pickup), T09 (rare and reputational, said to a borrower who was already filing a CFPB complaint), T01 (silent revenue loss on the most common flow).
- Data model. The nine records. S2S traces carry
pipeline: "s2s", missing stages appear as missing, and compliance runs on agent-channel STT. Retention is three years by policy in the US, with storage limitation on the UK and Spain books.
- Gold sets and CI. Immutable gold-v{n} releases, with fail-first cases added within two working days. The four tiers, with load at 1.0x and 1.5x of 6,000 plus a 3x spike, and a 5% canary decided by confidence sequences.
- Simulation. Personas whose interruptions are triggered by content, voiced by consented recordings or licensed TTS in English and Spanish, over SIP. The regeneration rate is reported. Simulation is checked against real calls quarterly.
- Online. About 7,696 judged calls a day; 400 audits a week with Neyman allocation; stratum-weighted PPI; C07 and the C04 backstop sized at 5,990 calls a week.
- Judges. J1 v1 stays advisory. No gating judge in UK English or Spain Spanish before each group has about 100 natively labeled violations.
- Cost. STT re-transcription goes to 3% first, which funds the two new groups at $68,782.57; compliance coverage is never cut.
- Build vs buy and S2S. The matrix and the S2S table above, with the cascade as the control.
- Coverage. The incident table, with T01, T03 and T09 each caught twice.
What goes in the decision memo?
Steelyard's build vs buy memo, written for the head of engineering and the CFO in the seven-part format every chapter uses.
Decision. Build the policy checks, gold sets, CRM diff, judge registry and gate logic. Buy telephony impairment, load generation and labeling. Buy caller simulation only from a vendor that passes a T02 and T07 pilot on Steelyard's own English and Spanish audio.
Context. Campaigns go from five to nine, language groups from three to five, and peak concurrency from 2,400 to 6,000 calls. The UK and Spain books go live, and the TCPA revocation rule takes effect on 25 November 2026.
Options considered. (a) One managed platform: fastest to stand up, but Steelyard's reading of Regulation F, CONC and the AI Act, and its release record, would live inside a vendor. (b) Build everything: full control, but months of SIP work. (c) Split by component: chosen.
Evidence. Vendor docs read on 2026-11-20 (interruptions on a timer, phone tests inbound only, text-only simulation). The 2.5x plan at $75,697.57, falling to $63,022.57 after the STT cut. Gold set gold-v7.
Owner and cadence. The staff engineer for evaluation. Reviewed quarterly and at every contract renewal.
Tripwires. A pilot that fails T02 or T07; a vendor with no raw audio and span export; a regeneration rate above the agreed level for two nightly runs in a row; data-handling terms for the UK and Spain books not signed before those books go live.
Dated facts. Vendor capabilities and Bluejay's self-reported pricing as of 2026-11-20. The TCPA revocation rule takes effect on 25 November 2026. The AI Act's legacy deadline for marking synthetic audio is 2 December 2026.
What to do this week
Build the coverage table for your own incident log first. Every row that lacks a component, a gate or a detection time goes on the backlog.
- Store version vectors and evidence turns with each check result, and strip content from latency spans at the collector.
- Turn last month's worst incident into a scenario card, and confirm it fails on the version that shipped.
- Register judges per language group, and make the gate code read the registry.
- Pilot each simulation vendor on your two hardest incidents, with a machine callee in at least one of them.
- Reread the chapters behind each row: error analysis, metric layers, code-switching normalization, TTS, task success and compliance.
NOTE
What will date on this page. The simulation capabilities and limits of LiveKit, Vapi, Retell, Roark, Coval, Cekura and Bluejay, and Bluejay's self-reported pricing (vendor docs, undated, read 2026-11-20). The EVA leaderboard leader (as of late October 2026). EchoChain's barge-in results (preprint, April 2026). The TCPA revocation rule (takes effect 25 November 2026). The AI Act's legacy deadline for marking synthetic audio (2 December 2026). CCD2 transposition (applies from 20 November 2026, incomplete in several Member States). ICO biometric guidance (under review after the Data (Use and Access) Act 2025).