This section teaches how to evaluate a voice agent on real phone calls, for the engineers who build one and the managers and compliance leads who sign off on it, using an Indian lender's Hinglish collections calls. The headline result: on Sierra's tau-Voice benchmark, a reasoning text model completed 85% of 278 English service tasks, while three audio-native voice agents completed 31-51% on clean phone audio [1]. The 17 chapters show where that gap opens on a call and how to measure each part of it.
At a glance
17
chapters, in four groups
2
editions, global and India
12
runnable labs, one per voice recipe
1-3
the chapters to read first
Running example: Asha, the collections agent of Tarazu Finance, a fictional Indian lender, on outbound calls in Hinglish, Hindi and Indian English under RBI recovery rules. The full background is under "About the running example" below.
What you will be able to do
- Read a voice agent trace, with audio, transcript, tool calls and timings on one clock, and code the failures in real calls.
- Score the words that carry money: entity error rate on rupee amounts, dates and confirmations in Hinglish, on phone-quality audio.
- Measure latency, turn-taking and barge-in by turn type, and check what the agent says aloud.
- Score task success from the CRM record, validate audio judges against human labels, and set release gates with pass^k and intervals.
- Run the checks in production: load and cost, drift and rare-violation monitoring, and RBI, TRAI and DPDP rules written as tests.
Where each chapter looks on a call
Figure: Asha's call path, top to bottom: audio crosses the phone line, speech-to-text writes it down, the turn detector decides the caller is done, the LLM calls CRM tools, text-to-speech answers, and the result is written to the CRM. Each box names the chapters that measure that stage: blue for the pipeline measures, green for the outcome checks. The dashed box lists the chapters that look at the whole call.
The 17 chapters in four groups
Each chapter opens on one incident from Tarazu's call floor, most with a trace id (T01 to T10), then works backward to the metric that would have shown it, how to measure it on phone audio, what it hides, and the gate that would have caught the call. Read chapters 1 to 3 first; by chapter 17 you hold a ten-incident log, and the platform design is checked against it.
- Diagnose, chapters 1-2: what a call trace holds, and how to code real failures.
- Measure the pipeline, chapters 3-9: metric layers, then STT, phone audio, latency, turn-taking and speech output.
- Validate outcomes and judges, chapters 10-13: task success, simulated callers, audio judges and reliability statistics.
- Run it in production, chapters 14-17: load and cost, monitoring, compliance and the platform design.
The chapter list follows the same order, 1 to 17, with a link and a one-line summary for each chapter.
Chapters:
- Diagnose (chapters 1-2)
- Measure the pipeline (chapters 3-9)
- Layered metrics for voice agents. Six layers, one north star, and code, judge or human per check.
- Entity error rate: STT beyond WER. Amounts, dates and confirmations, canonicalized and gated.
- Normalization and code-switching in Hinglish STT. Script policy, transliterated WER, and scorer changes as re-baselines.
- Testing on telephony audio. Codecs, band limits, packet loss and a condition matrix.
- Measuring voice agent latency. End-of-turn delay to turn gap, with P95 by turn type.
- Turn-taking, barge-in and recovery. Endpointing curves and three failures counted apart.
- Evaluating speech output for agents. Spoken-entity round trips before any MOS comparison.
- Validate outcomes and judges (chapters 10-13)
- Run it in production (chapters 14-17)
Two editions
Global edition
Lender and agent: Steelyard Credit, Tucson; Hazel
Languages: English, Spanish, Spanglish
Money: US dollars
Rules: US, UK and EU rules
India edition (this page)
Lender and agent: Tarazu Finance; Asha
Languages: Hinglish, Hindi, Indian English
Money: Rupees, in lakh and crore
Rules: RBI recovery rules, TRAI caller ID, DPDP
This is the India edition. The global edition runs the same 17 chapters on US, UK and EU calls in English and Spanish; the methods, computed numbers and course quizzes are the same in both.
Hands-on materials
TIP
Hands-on materials. The course materials are also collected in the voice-ai folder of the ai-evals-course repository on GitHub: synthetic data, the labs behind the twelve voice recipes in the cookbook (one folder per recipe under voice-ai/labs/), exercises and templates. Clone it to run each lab next to the chapter it supports; every recipe page names its folder.
Metric targets and a day-1 starter live in the voice playbook.
Background and sources
Why voice needs its own evaluation (the evidence)
A voice agent fails where a text agent never goes: the words cross a narrowband phone line before speech-to-text sees them, the agent guesses from a pause whether the caller has finished, and a wrong reply is heard before any log is written. Sierra's tau-Voice paper (March 2026) measured the cost of the channel. On 278 English service tasks a reasoning text model completed 85%, while three audio-native voice agents completed 31-51% on clean phone audio and 26-38% with noise, accents, dropped audio and interruptions; 79-90% of 91 reviewed failures traced to the agent, not the simulated caller [1]. Sierra's leaderboard later listed submitted results of 67% (April 2026) and above 80% (September-October 2026), on newer versions with longer call limits, so they do not compare with the paper [2] [3]. A score earned on text still says little about the same task over a phone line.
CAUTION
Vendor-reported. Treat it as a hypothesis to test on your own calls.
About the running example
The running example is Tarazu Finance and its collections agent, Asha. Tarazu Finance is a fictional lender; its numbers are illustrative. Asha makes Tarazu's outbound early-bucket collections calls in Hinglish, follows borrowers into Hindi or Indian English, and works under RBI recovery rules, through a cascade of 8 kHz mu-law audio, streaming STT, a turn detector, an LLM with CRM tools and TTS. Here a misheard word is a money error: in trace T01 the borrower said "dhai hazaar" and STT heard "do hazaar", an utterance WER of 8.3% and a promise to pay written Rs 500 short.
The decision memo format
Every chapter ends with Tarazu's decision memo, written for the manager who signs the release gate and the compliance lead who answers for the bot:
- Decision: what changes, in one sentence.
- Context: the incident and trace id behind it.
- Options considered, and why the others lost.
- Evidence: numbers with intervals and the dataset version.
- Owner and cadence.
- Tripwires that reopen the decision.
- Dated facts, with as-of dates.
Existing courses as of November 2026
As of November 2026 we found no course dedicated to voice agent evaluation end to end. What exists is single lessons, such as the one 10-minute evaluation lesson in DeepLearning.AI's Voice for AI Agents and Applications [4] and a free 30-minute Maven talk by Google DeepMind's Ravin Kumar [5]; Hamel Husain and Shreya Shankar's Maven evals course, whose description never mentions voice, speech or audio [6]; and vendor guides from Hamming, Coval and Cekura.
NOTE
What will date on this page: the tau-Voice leaderboard scores (as of October 2026) and the list of existing courses (as of November 2026).