This section teaches how to evaluate a voice agent on real phone calls, for the engineers who build one and the managers and compliance leads who sign off on it. The headline result: on Sierra's tau-Voice benchmark, a reasoning text model completed 85% of 278 English customer-service tasks, while three audio-native voice agents completed 31-51% on clean phone audio [1]. The 17 chapters show where that gap opens on a call and how to measure each part of it.
At a glance
17
chapters, in four groups
2
editions, global and India
12
runnable labs, one per voice recipe
1-3
the chapters to read first
Running example: Hazel, the AI agent that places outbound collections calls in English, Spanish and Spanglish for Steelyard Credit, a fictional US lender. The full background is under "About the running example" below.
What you will be able to do
- Read a voice agent trace, with audio, transcript, tool calls and timings on one clock, and code the failures in real calls.
- Score the words that carry money: entity error rate on amounts, dates and confirmations in English and Spanish, on phone-quality audio.
- Measure latency, turn-taking and barge-in by turn type and language, and check what the agent says aloud.
- Score task success from the CRM record, validate audio judges against human labels, and set release gates with pass^k and intervals.
- Run the checks in production: load and cost, drift and rare-violation monitoring, and US, UK and EU rules written as tests.
Where each chapter looks on a call
Figure: Hazel's call path, top to bottom: audio crosses the phone line, speech-to-text writes it down, the turn detector decides the caller is done, the LLM calls CRM tools, text-to-speech answers, and the result is written to the CRM. Each box names the chapters that measure that stage: blue for the pipeline measures, green for the outcome checks. The dashed box lists the chapters that look at the whole call.
The 17 chapters in four groups
Each chapter opens on one incident from Steelyard's call log, most with a trace id (T01 to T10), then works backward to the metric that would have shown it, how to measure it on phone audio, what it hides, and the gate that would have caught the call. Read chapters 1 to 3 first; by chapter 17 you hold a ten-incident log, and the platform design is checked against it.
- Diagnose, chapters 1-2: what a call trace holds, and how to code real failures.
- Measure the pipeline, chapters 3-9: metric layers, then STT, phone audio, latency, turn-taking and speech output.
- Validate outcomes and judges, chapters 10-13: task success, simulated callers, audio judges and reliability statistics.
- Run it in production, chapters 14-17: load and cost, monitoring, compliance and the platform design.
The chapter list follows the same order, 1 to 17, with a link and a one-line summary for each chapter.
Chapters:
- Diagnose (chapters 1-2)
- Measure the pipeline (chapters 3-9)
- Layered metrics for voice agents. Six layers, one north star, and code, judge or human per check.
- Entity error rate: STT beyond WER. Dollar amounts, dates and confirmations in two languages, canonicalized and gated.
- Normalization and code-switching in multilingual STT. Scorer rules, numeral canonicalization, language settings and Spanglish slices.
- Testing on telephony audio. Carrier codec chains, band limits, packet loss and a condition matrix.
- Measuring voice agent latency. From end-of-turn delay to turn gap, with P95 by turn type.
- Turn-taking, barge-in and recovery. Endpointing curves per language and three failures counted apart.
- Evaluating speech output for agents. Spoken amounts and dates in two locales, before any MOS comparison.
- Validate outcomes and judges (chapters 10-13)
- Run it in production (chapters 14-17)
Two editions
Global edition (this page)
Lender and agent: Steelyard Credit, Tucson; Hazel
Languages: English, Spanish, Spanglish
Money: US dollars
Rules: TCPA, state law, Regulation F as policy; UK and EU
India edition
Lender and agent: Tarazu Finance; Asha
Languages: Hinglish, Hindi, Indian English
Money: Rupees, in lakh and crore
Rules: RBI recovery rules, TRAI caller ID, DPDP
This is the global edition. The India edition runs the same 17 chapters on an Indian lender's Hinglish calls under RBI, TRAI and DPDP rules, with amounts in rupees. The methods, numbers that are computed and the course quizzes are the same in both; the incidents, languages and rules differ. The course lists both editions; read one.
Hands-on materials
TIP
Hands-on materials. The course materials live in the voice-ai folder of the ai-evals-course repository on GitHub: synthetic data, the labs behind the twelve voice recipes in the cookbook (one folder per recipe under voice-ai/labs/), exercises and templates. Run each lab beside the chapter it supports; every recipe page names its folder.
Metric targets and a day-1 starter live in the voice playbook.
Background and sources
Why voice needs its own evaluation (the evidence)
On a phone line a voice agent can lose the score it earned in text: speech crosses a narrowband carrier leg before speech-to-text hears it, and a turn detector guesses from a pause whether the caller is done. Sierra's tau-Voice paper (March 2026) measured the gap: across 278 English customer-service tasks, a reasoning text model completed 85%, while three audio-native voice agents completed 31-51% on clean phone audio and 26-38% once noise, accents, dropped audio and interruptions were added. Of 91 reviewed failures, 79-90% came from the agent, not the simulated caller [1]. Later leaderboard entries reached 67% (April 2026) and above 80% (September-October 2026) on newer versions with longer call limits, so they do not compare with the paper [2] [3]. Test the task on the phone path it will run on.
CAUTION
Vendor-reported. Treat it as a hypothesis to test on your own calls.
About the running example
The running example is Steelyard Credit, a US consumer lender in Tucson, and Hazel, the AI agent that places its outbound early-bucket collections calls. Steelyard Credit is a fictional lender; its numbers are illustrative. Hazel speaks English, Spanish and Spanglish, following the account preference and then the call itself, never a guess from a surname. The mix is common: the 2024 American Community Survey counts about 44.9 million residents age 5 and over who speak Spanish at home [4], and in a 2022 Pew survey published in 2023, 63% of US Latinos said they use Spanglish at least sometimes [5].
Steelyard collects its own debts in its own name, so the FDCPA and Regulation F largely do not bind it; it adopts their conduct rules as company policy, while the TCPA, state law such as California's Rosenthal Act and the ban on unfair, deceptive or abusive practices still apply [6] [7]. Its UK and Spanish books, Steelyard Credit UK Ltd and Steelyard Credit Espana S.L., appear in the compliance chapter and the capstone.
Hazel runs as a cascade of 8 kHz mu-law audio, streaming STT, a turn detector, an LLM with CRM tools and TTS, where a misheard word is a money error. In trace T01 the borrower said "yeah I can do two fifty on the twentieth, rest next month", STT wrote "two fifteen", and Hazel recorded the promise to pay before reading it back. The utterance WER was 8.3%, one word in twelve; the promise was $35.00 short. WER weighs that word like any other in the utterance, which is why the entity chapter scores the amount itself.
The decision memo format
Every chapter closes with Steelyard's decision memo, written for the engineering manager who signs the release gate and the compliance lead who answers to the regulator:
- Decision: what changes, in one sentence.
- Context: the incident and trace id behind it.
- Options considered, and why the others lost.
- Evidence: numbers with intervals and the dataset version.
- Owner and cadence.
- Tripwires that reopen the decision.
- Dated facts, with as-of dates.
Existing courses as of November 2026
As of November 2026 we found no course dedicated to voice agent evaluation end to end. What exists is single lessons, such as one 10-minute evaluation lesson inside DeepLearning.AI's Voice for AI Agents and Applications [8] and a free 30-minute Maven talk by Ravin Kumar of Google DeepMind [9]; Hamel Husain and Shreya Shankar's Maven evals course, whose description does not mention voice, speech or audio [10]; and vendor guides from Hamming, Coval and Cekura.
NOTE
What will date on this page: the tau-Voice leaderboard scores (as of October 2026), the list of existing courses (as of November 2026), and the ACS 2024 and Pew 2023 population figures (replace them when newer releases appear).