On 2026-11-17 Daniela Torres told Hazel, through tears, that she had lost her job, and 180 ms later Hazel asked, faster than her usual pace, whether she could manage one hundred dollars this week (trace T10). The transcript judge passed the call on empathy and on policy; two human raters failed it on both. This chapter covers what a model judge can hear on a call, how to show it agrees with people in each language group before it decides anything, and how to run it after the call instead of inside it.
What happened on the call?
Daniela (P04), an AUTO borrower who speaks Spanglish, was on flow F04, hardship: the gold end state is flag_hardship, the Customer Advocate's details stated and texted, a transfer to the hardship queue, and no request for money once the hardship is stated. From T10:
t_ms ch event / text
caller REF: "I lost my job... [crying, 4 s] ...ahorita no puedo pagar nada"
gloss: "I lost my job... [crying, 4 s] ...right now I can't pay anything"
agent TTS: "I understand. Could you pay a minimum amount, say one hundred dollars, this week?"
audio agent speaking rate 5.9 syllables/s (campaign median 4.4); agent started 180 ms after caller vad_end, over the sob
judge transcript judge (v2): empathy=pass, policy=pass
human two raters: empathy=fail, C09=fail (PTP request after hardship)
Two failures share the call. The words break C09: a payment request in the turn right after "I lost my job". The delivery breaks the flow's intent: 5.9 syllables per second against a median of 4.4, started over a sob. The transcript view shows the first; the rate, the overlap and the [crying, 4 s] tag exist only on the timeline.
Diagnosis: the judge got one question a rule should have settled (C09) and one delivery question with no delivery in its input. Handing it the raw audio fixes neither.
Do audio judges hear what transcripts miss?
Partly. Audio LLM judges order whole systems roughly as people do, but on single items they sit near chance for speaking rate and pronunciation at zero-shot, and their ambient-sound rankings barely track human ones. Text judges reading transcripts with non-verbal tags scored higher under rubric scoring.
Paralinguistics is everything in speech beyond the words: rate, pitch, loudness, pauses, voice quality, emotion, and sounds such as crying or laughter. A transcript judge sees none of it, and T10's delivery failure is made of it. Passing the judge the recording is the obvious move; the published evidence says it does not reliably work.
- Ambient sound. In MTalk-Bench, audio LLM judges' pairwise (Arena) rankings of speech-to-speech systems on ambient-sound handling were almost uncorrelated with human rankings: Spearman 0.10 for GPT-4o Realtime and 0.04 for Gemini-2.5-pro, with Qwen-Omni-Turbo at 0.48. Paralinguistic correlations on raw audio were moderate (0.43-0.70), not near zero. Text LLM judges reading transcripts annotated with non-verbal tags reached roughly 0.80-0.93 on rubric scoring for the paralinguistic and ambient dimensions, and in the Arena setting text judges did no better than audio judges on paralinguistic cues 1.
- Rate and pronunciation, one item at a time. In AudioJudge, GPT-4o-Audio with basic zero-shot prompting scored 46.0% on pairwise pronunciation and 46.9% on speaking-rate comparisons, where random guessing is 50%. The best setup (4-shot examples with the test audio concatenated) reached 66.5% and 55.3%, against 69.5% and 77.8% for two fresh human annotators. Adding the speech-to-text transcript lowered the average from 60.2% to 58.5% 2.
- Ranking systems is an easier job. The same judges ranked models well, with Spearman correlation to human preference up to 0.93 on ChatbotArena-Spoken and 0.91 on SpeakBench. Pointwise scores converted to pairs reached only 55-59%, against 65-70% for direct pairwise judging 2.
- Naturalness. On SpeechJudge-Eval, a pairwise naturalness benchmark, the best off-the-shelf audio LLM, Gemini-2.5-Flash, agreed with human preferences 69.1% of the time, and the MOS predictors UTMOS and DNSMOS 53.7% and 57.9%. A trained 7B generative reward model reached 77.2%, or 79.4% with voting over 10 samples. Humans themselves fully or weakly agreed on only about 70% of the 99K labeled pairs 3.
The pattern: rank correlation across systems says little about accuracy on one call. Steelyard grades calls one at a time, and a judge that picks the better vendor can still be near a coin flip on whether Hazel's reply to Daniela was too fast. None of these benchmarks is built from collections calls on a phone line, so read every figure above as an upper limit for your own judge.
What is a feature-to-text judge?
A feature-to-text judge measures cheap audio signals (speaking rate, pauses, overlap, emotion scores, non-speech tags), writes them with the transcript into a structured text blueprint, and has a text LLM judge it one dimension at a time. The published method, TRACE, has English results only.
TRACE (Chandra et al., EACL 2026 Findings) was tested on two English speech-to-speech preference benchmarks. It matched the authors' re-annotated human labels more often than audio-model or transcript-only judges, at about a third of the audio-model judge's cost but somewhat more than a transcript-only judge 4. On SpeakBench its accuracy was 68.6%, against 61.1% for the audio-model judge and 62.7% for the transcript-only judge.
Figure 1: Published results from two English studies, not Steelyard data. On SpeakBench the raw-audio judge did no better than a transcript, while the judge given measured features as text did best. On ambient sound, audio judges' rankings barely followed people's; a text judge reading tagged transcripts scored high, but under a different protocol.
At Steelyard the blueprint is one JSON object per agent turn, assembled from fields already in the trace. T10's hardship turn:
{
"dimension": "TONE.PACE after hardship", "language_group": "Spanglish",
"caller_text": "I lost my job ... ahorita no puedo pagar nada",
"caller_nonspeech": ["crying, 4 s"],
"agent_text": "I understand. Could you pay a minimum amount, say one hundred dollars, this week?",
"agent_speaking_rate_syll_s": 5.9, "campaign_median_rate_syll_s": 4.4,
"agent_start_after_caller_end_ms": 180, "agent_started_over_nonspeech": true
}
The Spanish clause means "right now I can't pay anything". Every value comes from the T10 trace.
Trust in a blueprint judge depends on its features more than on its LLM.
Agent-side features are close to exact. Hazel's speaking rate is syllables in the TTS input text over audio duration; the platform turn gap is two spans on one clock. No model is involved.
Caller-side features carry a model's errors. The crying tag or an emotion score comes from a classifier, and no source this chapter found tests a crying detector on 8 kHz English, Spanish or Spanglish phone audio. So Steelyard validates each caller-side extractor (tag precision and recall against human labels) before the judge that reads it. A missed [crying] tag leaves J2 as blind as the transcript judge.
If the EU book gives J2 a distress score inferred from the voice, that extractor is emotion recognition under the AI Act: Article 50(3) requires the deployer to inform the people exposed to it, and the Act bans emotion recognition only in workplaces and education, not on collections calls 5. Steelyard gives the same notice on UK-book calls as policy; this is its reading, not legal advice, and the check lives in the compliance chapter.
Which dimension gets which judge?
Send each dimension to the cheapest input that holds its evidence. Content and policy wording go to code first and a transcript judge second; delivery and overlap go to blueprint features; speech fidelity goes to an audio judge that has the expected text beside it. A question code can answer is never given to a judge.
| Dimension | Example from Steelyard's log | Who decides | Input |
|---|
| Required strings and their order, money asked for after a hardship or dispute claim | T03 (C04), the words of T10 (C09) | Code | Event order, tool log and TTS input text |
| Coercion and consequence misstatement | T09 (C07) | English and Spanish lexicon on 100%, then judge J1 | Agent-turn transcript plus lexicon hits |
| Pushed past a dispute | T02 after "ya lo pague" [I already paid it] (C08 meaning) | Tool-sequence check, then judge J3 | Transcript after the dispute claim |
| Pacing and empathy after distress | T10 delivery (TONE.PACE) | Judge J2 | Blueprint of transcript, rate, overlap, pauses and tags |
| Masculine self-reference in Spanish | "estoy seguro" [I am sure, masculine] (TONE.GENDER) | Code | Agent text against a short lexicon |
| False cutoff, talk-over, missed barge-in | T05, T02 | Code over timeline spans | Span timeline (turn-taking owns it) |
| Amounts and dates Hazel speaks | T01, T06 | Code | Canonicalizer and round trip (TTS and entity error rate own them) |
| Speech fidelity of free text | Customer Advocate name and email, the Steelyard address | Audio judge with expected text | Agent audio plus TTS input text |
| Naturalness and voice choice | Vendor comparison | Listening panel; audio judge only to rank systems | Phone-path clips |
Speech fidelity is whether the audio actually says the expected text, completely and intelligibly. A raw-audio judge belongs here: the question is about the sound and the answer is known. tau-Multilingual's speech-fidelity judge is a multimodal Gemini 3.1 Pro Preview model that compares the audio with the expected text 6. For amounts, dates and digits Steelyard keeps the deterministic round trip, since exact match needs no validation study.
The table's order is the pipeline's order. T10's request for one hundred dollars is a C09 failure in the words, which a tool-order and entity check finds on every call. The judge gets only the residue: whether the delivery after the distress was acceptable.
C09 is Steelyard policy. Steelyard collects its own debts in its own name, so the FDCPA and Regulation F largely do not bind it 7 8; it adopts their conduct rules as company policy, and state law (California's Rosenthal Act, Massachusetts 940 CMR 7.00), the TCPA and the CFPB's ban on unfair, deceptive or abusive practices still apply. In the UK book the same turn is also read against CONC: forbearance and due consideration for customers in or approaching arrears (CONC 7.3.4R), and no pressure to pay within an unreasonably short period (CONC 7.3.10R) 9.
The speech-to-speech pilot has no TTS input text, so agent-side judge inputs become agent-channel speech-to-text, errors included; the platform chapter covers the rest.
How do you validate a judge against human labels?
Two bilingual raters label calls from the audio and are measured against each other first; the judge is tuned on a dev split and read on the test split once. Admit it only if violation recall and specificity clear 0.90, and kappa 0.60, on the lower ends of their 95% intervals, per language group and dimension.
In order:
- One binary decision per dimension. For J2: "After a hardship statement, did the reply ask for money, rush, or start over non-speech?" Graded empathy scales invite rater drift. Calibration to humans covers rubric iteration.
- Two raters, independently, from the audio. The label stands for what the borrower heard. Spanish-dominant and Spanglish calls need English-Spanish bilingual raters, and the adjudication rule is written down first.
- Measure the raters first. Two people who disagree give a judge nothing to aim at, so compute Cohen's kappa between them and read each disagreement before any judge runs (inter-rater agreement). The bar for a judge is the human ceiling, and published work puts that ceiling well below 1.0. In Full-Duplex-Bench-v2, agreement between the LLM judge and humans on 120 sessions was Krippendorff's alpha 0.59 to 0.69, lowest for correction handling (0.588) and turn-taking fluency (0.614), which the authors put down to the subjectivity of timing judgments 10. In VoiceAgentBench, a GPT-4o-mini judge of tool arguments agreed with the human majority on 92.4% of about 400 responses but with kappa 0.795, against 96.8% and 0.914 between the two human annotators 11. In tau-Voice, two human reviewers attributing failures agreed on 84% (76 of 91) of simulations before adjudication 12.
- Dev, then test once. Iterate the prompt on dev; score test once, after the prompt is frozen. Each extra look at test is a tuning step.
- Name the positive class. The judge calibration recipe treats PASS as positive, so its TPR is the share of compliant calls that pass. Here the violation is positive: recall is violations caught, specificity is compliant calls left alone. An unlabeled "TPR" is how a team ships a judge that catches 83% of violations believing it catches 94%.
- Admit on the lower bound. Steelyard requires the Wilson 95% lower bound of violation recall and of specificity to reach 0.90, and the lower end of a call-level bootstrap interval for Cohen's kappa to reach 0.60 13 14. With only 18 violations, a point estimate over the bar means almost nothing.
CAUTION
Vendor-reported. Treat it as a hypothesis to test on your own calls.
A published alternative: tau-Multilingual admits an LLM-based judge into results only when precision and recall exceed 0.75, F1 exceeds 0.80 and Cohen's kappa exceeds 0.60, validated on about 60 native-language labeled examples per language for naturalness and speech fidelity and at least 30 per diagnostic; failing measures stay archived 6. For naturalness, Spanish was strongest (F1 97.9%, kappa 0.965) and the weakest passing language reached F1 81.4% and kappa 0.634. The authors are Sierra and Mercor employees, so the results are self-reported on their own benchmark. The gate applies to point estimates; with about 60 items the kappa interval is wide, and by this site's inference, not the paper's, 0.634 would probably not clear 0.60 on a lower bound.
Worked example: J1 v1 on Spanglish. J1 judges C07 (coercion and consequence misstatement) from agent-turn transcripts plus lexicon hits. Of gold-v7's 120 human-labeled Spanglish calls, 18 were violations. J1 caught 15 and raised 6 false alarms.
| Human: violation | Human: compliant | Total |
|---|
| J1 flags | 15 (caught) | 6 (false alarm) | 21 |
| J1 passes | 3 (missed) | 96 | 99 |
| Total | 18 | 102 | 120 |
- Agreement: (15 + 96) / 120 = 0.925.
- Chance agreement from the row and column totals: (21/120)(18/120) + (99/120)(102/120) = 0.02625 + 0.70125 = 0.7275. So kappa = (0.925 - 0.7275) / (1 - 0.7275) = 0.725. Resampling calls in a bootstrap, as the recipe does, puts it at about 0.52 to 0.89.
- Violation recall: 15/18 = 0.833, Wilson 95% interval 0.608 to 0.942.
- Specificity: 96/102 = 0.941, Wilson 95% interval 0.878 to 0.973.
- Precision: 15/21 = 0.714.
J1 misses the 0.90 recall bar on the point estimate. On the lower bounds it misses three ways: recall 0.608, specificity 0.878, kappa about 0.52. tau-Multilingual's point-estimate gate also rejects it, on precision (0.714 < 0.75) and F1 (2 x 0.714 x 0.833 / 1.547 = 0.769 < 0.80), though recall and kappa would pass there. Steelyard's registry records the verdict: J1 v1 is advisory, and its flags route to human audit.
A v2 needs more violations, not more calls, because the violation count sets the width of the recall interval. With 100 violations, 97 caught clears the bar (Wilson lower bound 0.915) and 96 barely does (0.902). Random sampling at this set's 15% share would take about 667 labeled calls to find 100. Enrich instead: take validation calls from lexicon hits, risk strata, red-team scripts and incident regressions, and then restate precision at production prevalence.
Why must you validate per language group?
Because a judge's verdict on the same content moves with the input language. Steelyard validates each judge separately on English, Spanish-dominant and Spanglish calls, and a pass admits it in that group only. It also records the STT language setting, since another setting hands the judge different text.
The evidence is text-only but plain. Fu and Liu tested five judge models on five text tasks in 25 languages and treated each language's verdict on the same parallel item as a separate rater: the judges reached an average Fleiss kappa of only about 0.3, and neither multilingual training nor larger scale directly improved consistency 15. Verdicts can shift with the language of the input; the study says nothing direct about audio judges or Spanglish.
J1 v1 was validated only on Spanglish (14% of accounts) and failed. It has no result for English (64%) or Spanish-dominant (18%), and the Spanglish failure predicts nothing about either. English is the group most likely to be trusted without evidence: it carries most of the traffic, and judges do well on English benchmarks.
The STT slice is Steelyard policy, not the paper's finding. A transcript judge hears the caller through STT, and on code-switched speech the language setting changes the transcript a lot (the code-switching chapter has the numbers). A vendor update that forces English on the Spanglish campaign could hand the judge "ahorita no puedo pagar nada" [right now I can't pay anything] as English-shaped guesses, so admission is per language group and per STT language setting, and the registry records both. Other languages (4% of accounts) go to humans in v1. A new language gets a gating judge only after about 100 native-labeled violations.
Why does raw agreement mislead at collections prevalence?
Raw agreement rewards the majority class, and on collections calls that class is "compliant". If 3% of calls are violations, a judge that never flags anything scores 97% agreement, kappa 0, and catches nothing. Report violation recall, specificity and precision, with intervals, beside every agreement number.
On 1,000 calls with 30 violations, the always-compliant judge matches 970 and misses all 30: agreement 0.97. Chance agreement is 1.0 x 0.97 + 0 x 0.03 = 0.97, so kappa = (0.97 - 0.97) / (1 - 0.97) = 0.
J1 v1 shows the subtler version. Its 0.925 agreement was earned on a set that was 15% violations, and precision falls as violations get rarer. With recall 0.833 and specificity 0.941 unchanged and the violation rate at 3%:
precision = 0.03 x (15/18) / (0.03 x (15/18) + 0.97 x (6/102))
= 0.0250 / (0.0250 + 0.0571)
= 0.305
About seven flags in ten would be false, against fewer than three in ten on the validation set. Figure 2 shows the curve.
Figure 2: Synthetic Steelyard data. J1 v1's recall and specificity come from the 120-call Spanglish validation set, where precision was 0.714. On a stream with 3% violations the same judge is wrong on about seven flags in ten, so quote precision at each stratum's real prevalence.
Specificity has a staffing cost too. If it held across the 7,696 calls judged a day under the stratified plan, about 7,696 x 6/102 = 453 calls a day would carry a false C07 flag. Steelyard's human audit covers 400 calls a week, about 67 a day over six calling days. A flag nobody has time to read is not a control, which is why the bar has a specificity term and flag volume is a memo tripwire.
Publish the whole confusion matrix, label the class behind each rate, and restate precision for every stratum at its own prevalence. Kappa shifts with prevalence as well 13, so two kappas are comparable only when their sets have a similar balance of classes.
When do you revalidate?
On any change to the judge model, the judge prompt, a feature extractor, the STT, or the language mix of scored traffic. The fast check is a rescore of a frozen 200-call anchor set; a full revalidation on a fresh test set per language group follows when the anchor numbers move.
- Judge model. Use a dated model id; when an alias moves to a new model, you have a new, unvalidated judge.
- Prompt or rubric, including a policy-pack change that redefines a check, such as a new CONC reading.
- Feature extractor. Swapping the crying tagger or the syllable counter changes what J2 is given.
- STT version or language setting. A transcript judge hears the caller through the STT.
- Language mix. More volume on POS-short-nov, which calls Spanish-dominant and English borrowers, moves traffic toward a group J1 has never been tested on.
The anchor set holds 200 calls whose human labels are frozen and which no prompt was tuned on. After any trigger the candidate rescores all 200, and the team checks three things: recall and specificity per class against the registry, the calls whose verdict flipped since the last version, and, by listening, every labeled violation the candidate now lets through. Both versions scored the same calls, so compare them with McNemar's test on the calls where they disagree 16; Wilson intervals are for one proportion, not a difference.
An anchor rescore can flag a change but cannot readmit a judge: 200 calls contain too few violations of one kind for a lower bound above 0.90. A judge whose anchor numbers shift goes back through full validation. To keep the set current without losing comparability, add calls in dated batches and never drop old ones.
How do judges run in production without blocking the call?
Score after hang-up, from a queue; only deterministic guardrails run during the call. Each judge reads the finished recording and trace in one idempotent job with capped retries and a dead-letter queue, and reported violation rates blend judge scores with a random human audit through prediction-powered inference.
Why not live. A judge needs the complete call, and any LLM request inside a turn adds to the turn gap that the latency chapter budgets. Blocking a sentence before Daniela hears it is a different tool, a guardrail on TTS input text, and the compliance chapter owns it. Judges describe what already happened.
The job. At call end the trace is finalized, the recording redacted, and one job per (call_id, judge id, judge version) enqueued, so retries and replays write one row. The result lands in the trace's judge_scores with judge version and prompt hash.
Failure handling. Transient errors retry with backoff a bounded number of times, then land in a dead-letter queue that pages someone as it grows. A missing score stays missing, never a pass; otherwise a judge outage shows up as a compliance gain. Judge workers share provider quotas with Hazel. At Steelyard's invented prices a blueprint judge costs $0.02 per judged call and a raw-audio judge $0.125 (load and cost prices the plan), and production monitoring owns which calls get judged.
PPI for reported rates. Compliance asks for one number, say this month's C07 violation rate. Counting judge flags imports the judge's mistakes; counting audited calls is unbiased but rests on a small sample. Prediction-powered inference (PPI) combines a model's predictions on many items with a small human-labeled sample to give provably valid confidence intervals for quantities such as a mean, without assumptions about the model; more accurate predictions give narrower intervals 17. For a single stratum:
PPI estimate = (mean judge verdict over all judged calls)
+ (mean of human label minus judge verdict over the audited calls)
The second term, the rectifier, measures how wrong the judge is on the calls a person checked. With toy numbers, not Steelyard's: if the judge flags 2.0% of calls and over-flags by 0.6 points on the audited ones, the estimate is 1.4%.
It is valid only under three conditions, and each one is easy to break:
- The audit sample is random within its stratum. Auditing only flagged calls measures precision; it cannot correct the rate, because it never sees the violations the judge missed.
- A stratified sample needs the stratified estimator. Basic PPI assumes the human sample is drawn from the same population as the judged items. Steelyard judges every risk-stratum call (about 5,200 a day) and 2% of the rest (2,496 a day), so it runs stratified PPI within each stratum and weights strata by their population 18.
- A weak judge buys little. The narrower interval comes from judge accuracy, so while J1 is at v1 the human audit remains the main evidence. Confidence-driven inference also chooses which items humans label, while guaranteeing results "no less accurate" than human labels alone 19.
A simulated month of stratified PPI is worked in the voice judge validation recipe.
What gate would have caught it?
Run C09 in code on 100% of calls: fail any call where an agent turn after a recorded hardship statement contains an amount or a payment request. That catches T10's words without a model. The delivery half needs J2, which gates only after it passes validation per language group on the lower bound.
The gate, in order:
- C09 in code. From the tool log and TTS input text: once hardship is stated, flag_hardship and transfer_to_human(hardship) must run, and no later agent turn may carry an amount entity or a payment request. T10 fails on "one hundred dollars". The check needs no validation study because it contains no judgment.
- J2 stays advisory until validated. Its fails go to human audit with the evidence turns attached, the same route J1's flags take.
- J2 admission. Per language group and STT language setting: violation recall and specificity >= 0.90 on the Wilson lower bound, kappa >= 0.60 on the bootstrap lower bound, test read once, and each caller-side extractor validated first.
- A regression case. T10 becomes an F04 scenario card in Spanglish with its delivery features, so every later judge version meets the call the transcript judge passed.
The transcript judge that passed T10 could never meet step 3 for this dimension: with no rate, overlap or crying tag in its input, no prompt finds the delivery failure.
What goes in the decision memo?
A judge admission memo admits or rejects one judge version for one role, gives intervals per language group, and creates a registry entry. Steelyard's memo rejects J1 v1 as a C07 gate, keeps it advisory with flags routed to human audit, and sets out what a v2 has to show.
Decision. Reject J1 v1 as a C07 gate in every language group. It stays advisory: flags go to human audit, the English and Spanish C07 lexicon runs on 100% of calls, and no release waits on J1. A v2 gates in a language group only after about 100 native-labeled violations there, with recall and specificity >= 0.90 and kappa >= 0.60 on their lower bounds, test read once.
Context. T09 (2026-11-18) showed C07 failures are rare and costly. J1 v1 failed Spanglish validation in November 2026 and has never seen a UK call, where C07 also covers pressure to pay within an unreasonably short period (CONC 7.3.10R), like T10's "one hundred dollars, this week".
Options considered. (a) Gate on J1 v1 now: it lets one violation in six through on the point estimate, nearly two in five at the Wilson lower bound. (b) Advisory J1 beside lexicon and audit: chosen. (c) Raw-audio judge: coercion lives in the words, so audio adds cost, not evidence.
Evidence. gold-v7 Spanglish test split, 120 calls, 18 violations: recall 0.833 (Wilson 0.608-0.942), specificity 0.941 (0.878-0.973), kappa 0.725 (bootstrap about 0.52-0.89), agreement 0.925, precision 0.714 (about 0.305 at 3% prevalence). English, Spanish-dominant: untested.
Owner and cadence. Eval platform staff engineer: registry and anchor reruns per trigger. QA lead: bilingual raters. Compliance lead: signs status changes.
Tripwires. Any audited C07 violation J1 passed. Daily J1 flag volume above audit capacity (about 67 calls a day). Any revalidation trigger.
Dated facts. MTalk-Bench (2025), AudioJudge (July 2025), TRACE (EACL 2026), tau-Multilingual (September 2026), CONC (read November 2026).
The registry entry the memo creates:
{
"id": "J1",
"version": "v1",
"dimension": "C07 coercion and consequence misstatement",
"model": "<pinned, dated judge model id>",
"prompt_sha256": "<hash of the frozen prompt>",
"inputs": ["agent-turn transcript", "C07 lexicon hits (English, Spanish)"],
"positive_class": "violation",
"gold_set": "gold-v7",
"languages_validated": [],
"stt_language_settings_validated": [],
"jurisdictions_validated": [],
"dev_calls": "<size of the dev split>",
"results": {
"Spanglish": {"test_calls": 120, "test_violations": 18, "test_read_once": true,
"tpr_violation_recall": 0.833, "tpr_ci95": [0.608, 0.942],
"tnr_specificity": 0.941, "tnr_ci95": [0.878, 0.973],
"kappa": 0.725, "kappa_ci95": [0.52, 0.89], "precision": 0.714,
"verdict": "FAIL"}
},
"admission_rule": "recall and specificity Wilson lower bound >= 0.90; kappa bootstrap lower bound >= 0.60",
"validated_on": "2026-11",
"revalidate_triggers": ["judge model", "judge prompt", "feature extractor", "STT version or language setting",
"language mix", "frozen 200-call anchor set drifts"],
"owner": "<named person>",
"status": "advisory",
"flags_route_to": "human audit"
}
What an auditor asks of each field:
| Field | Holds | Question it answers |
|---|
| id, version | J1, v1 | Which judge scored this call? |
| model, prompt_sha256 | Dated model id, prompt hash | Is today's judge the one that was validated? |
| inputs | Transcript, lexicon hits | Could it see the evidence? |
| positive_class | violation | Which class do TPR and TNR refer to? |
| languages_validated, stt_language_settings_validated, jurisdictions_validated | All empty for J1 v1 | Where, and on which inputs, may it gate? |
| dev_calls, test_calls, test_read_once | Split sizes | Was the test split used for tuning? |
| tpr, tnr, kappa, ci95 | Point and interval per language group | Is the lower bound above the bar? |
| validated_on, revalidate_triggers | Date, triggers | Is the result still current? |
| owner, status, flags_route_to | Person, advisory, human audit | Who answers for it, and where do flags go? |
What to do this week
Write down every judge you run with its dimension, its input, the language groups it was validated on, and its violation-class recall and specificity with intervals. Any judge whose validated-languages field is empty is advisory from today.
- Move every check a rule can express (event order, required strings, amounts after hardship or a dispute, gendered self-reference in Spanish) out of the judge and into code on 100% of calls.
- Give each delivery dimension a blueprint: agent rate, turn gap and overlap from spans, and caller tags from an extractor validated on your own phone audio.
- Before turning on an emotion or distress extractor, check which of your jurisdictions require a notice.
- Recompute every judge's precision at production prevalence and count the flags per day your audit team would have to read.
- Freeze 200 human-labeled anchor calls and trigger a rescore on any change of judge model, prompt, extractor, STT or STT language setting.
- Run judges after hang-up, give each job a (call, judge, version) key, and display unscored calls as unscored.
- Run the voice judge validation recipe on your labels, and read biases and mitigations and pairwise vs direct before you pick a judge format.
NOTE
What will date on this page. MTalk-Bench correlations (arXiv v2, September 2025). AudioJudge per-task and system-level numbers (arXiv v1, July 2025). SpeechJudge-Eval agreement (arXiv v2, 30 November 2025). TRACE SpeakBench accuracies and cost ratio (EACL 2026 Findings, arXiv v2, January 2026). tau-Multilingual's admission gate and per-language results (arXiv v1, 21 September 2026; self-reported). Full-Duplex-Bench-v2 alpha values (arXiv v2, April 2026), VoiceAgentBench agreement (arXiv v3, February 2026) and tau-Voice reviewer agreement (March 2026). Fu and Liu's cross-language kappa (May 2025, text only). PPI and its stratified and confidence-driven variants (2023 to 2025). AI Act Article 50(3) notice duty and the Article 5(1)(f) limits on emotion recognition (Regulation (EU) 2024/1689, read November 2026). FCA CONC 7.3.4R and 7.3.10R (sourcebook as updated July 2026, read November 2026). Judge models change faster than any of these papers; recheck the newest audio judges on per-item numbers, not on their system rankings.