The week Tarazu shipped the build that produced T01, its dashboard showed corpus word error rate (WER) flat at 11% and promises to pay (PTP) per right-party contact (RPC) up 6 points; two weeks later the kept rate fell. Every metric on that dashboard was correct, and none of them measured whether the promise in the CRM was the promise the borrower made. This chapter builds the metric set that would have: six layers from words to compliance, one north star with guardrails, and a rule for which checks run as code, which need a judge and which need a person.
What did the dashboard miss?
It missed a wrong amount. The borrower said "dhai hazaar" (Rs 2,500), speech-to-text (STT) heard "do hazaar" (Rs 2,000), and Asha stored a Rs 2,000 promise with no read-back. WER saw one wrong word, promises per right-party contact counted a promise, and nothing compared the stored amount with what was said.
t_ms ch event / text
14100 agent TTS: "Rahul ji, aapka overdue chaar hazaar do sau pachaas rupaye hai. Aap kab tak bhar paayenge?"
18220 caller vad_start
caller REF: "haan main dhai hazaar de dunga bees tarikh ko, baaki next month"
caller STT: "haan main do hazaar de dunga bees tarikh ko baaki next month"
21940 caller vad_end
22310 agent eot_commit (end-of-turn delay 370 ms)
22990 agent llm first token (TTFT 670 ms)
23010 agent tool record_promise_to_pay(amount_paise=200000, ptp_date=2026-11-20, readback_confirmed=false)
23740 agent TTS: "Theek hai, toh main do hazaar rupaye bees November ke liye note kar rahi hoon."
28120 caller REF: "haan haan theek hai" (a backchannel; he did not register the changed amount)
Diagnosis: the dashboard measured words in and promises out, and the failure lived in between, where an amount became a tool argument.
Each number moved for a reason you can read off the call. WER stayed flat because a misheard "dhai" costs the same as a misheard "ko", one word in twelve (the entity error rate chapter owns that arithmetic). PTP per RPC rose because promises booked before read-back, with "haan haan" taken as agreement, are cheaper to get. The kept rate fell later because Rs 2,000 paid against a Rs 2,500 promise is a broken promise, and the kept check runs only after the promise date. That lag is why a business metric alone cannot gate a release.
What are the layers of a voice agent metric set?
Six: lexical (were the words right), entity (were amounts, dates and yes/no right), semantic (was the meaning kept), task (is the CRM in the right end state), experience (did the call feel right) and compliance (did it break a rule). The first four come from Cresta's STT evaluation; Tarazu adds the last two.
Cresta, a contact-center AI vendor, evaluates speech-to-text in four layers: lexical (WER, CER, SER), entity (accuracy on business-critical tokens such as phone numbers, emails, amounts and dates, plus keyword recall), semantic (whether meaning is preserved, scored mainly by LLM judges calibrated against human ratings) and task (downstream behavior, measured by agent success rate, p95 latency and interruption rate). When comparing systems it applies hard constraints first, then compares on WER, and uses entity, worst-slice and task metrics to break ties 1.
That order fits an STT bake-off, where everything downstream is shared, and not a release gate for a collections agent. Tarazu keeps hard constraints first and reverses the rest: task and entity decide the release, and WER becomes a diagnostic that explains an entity failure after the fact.
Tarazu changes the stack in two places. Experience gets its own layer, because latency and interruption trade against accuracy (next section) and a blended task number hides which way a change moved. Compliance sits on top as a constraint, not a score: a call with a disclosure before verification has failed whatever its other scores, and compliance is never averaged into anything.
| Layer | Tarazu metric | Typical check |
|---|
| Compliance | Policy violation rate; zero C01-C16 failures per call | Code first; lexicon, judge and human audit for C07 |
| Task | Verified PTP per RPC; CRM diff | Code |
| Experience | P95 platform turn gap by turn type; talk-over, false cutoff, missed barge-in | Code on spans and labels; judge J2 for pacing |
| Semantic | Meaning preserved per caller turn | Judge, validated per language |
| Entity | Amount and date entity error rate | Code (canonicalize, exact match) |
| Lexical | WER, CER | Code; diagnostic only |
The CRM-diff method is owned by task success and outbound outcomes, latency terms by measuring voice agent latency, and talk-over and false cutoff by turn-taking and barge-in.
Figure: Synthetic Tarazu data. The metric stack for Asha. Compliance is a constraint on top, task and entity decide a release, experience is measured on its own layer, and lexical WER (dashed) explains entity failures instead of gating.
Why do accuracy and experience pull apart?
Because what makes an agent accurate (waiting longer, reading amounts back, checking before answering) costs time and turns, and what makes it feel fast costs accuracy. ServiceNow's EVA benchmark found the trade-off at launch; its leaderboard now shows one system leading both, so treat it as a design problem.
EVA scores voice agents on EVA-A for accuracy (task completion, faithfulness, speech fidelity) and EVA-X for experience (conversation progression, conciseness, turn-taking). Its March 2026 launch results, 20 systems on 50 English airline scenarios with 3 trials each, showed agents strong on task completion tending to give a worse experience, with no configuration dominating both 2. The EMNLP 2026 Findings paper, 12 systems on 213 scenarios with 5 trials each, found no system above 0.5 on both pass@1 scores and reported the trade-off only among cascade pipelines 3. Since late October 2026, EVA's own leaderboard shows one fully managed cascade harness, Cartesia Managed Agents with GPT 5.5, leading on both axes (0.87 EVA-A, 0.91 EVA-X pass@1), and ServiceNow now attributes the trade-off to system design rather than a hard limit 4.
The part worth copying is how EVA defines a pass. A call passes accuracy only if Task Completion = 1.0 AND Faithfulness >= 0.5 AND Speech Fidelity >= 0.95, and passes experience only if Turn-Taking >= 0.8 AND Conversation Progression >= 0.5 AND Conciseness >= 0.5 3. Each axis is a conjunction of per-call thresholds, not an average, so a good turn-taking score cannot buy back a failed task. The paper also reports a median gap of 0.44 between pass@k and pass^k on EVA-A: succeeding in at least one trial was far more common than succeeding in every trial (see reliability statistics).
Tarazu's AND-gates, per call:
- Accuracy pass: CRM diff equals the gold end state AND every amount and date entity is right AND zero C01-C16 failures.
- Experience pass: every platform turn gap within its turn-type SLO AND no talk-over AND no false cutoff AND no missed barge-in AND no early hang-up.
Report both pass rates per segment and the trade-off shows whenever a change moves them apart. Moving TW's silence budget from 300 to 400 ms after T05 cut false cutoffs and added 100 ms of silence budget to every default-state turn (amount-asked and date-asked states stayed at 600 ms); Tarazu took it. The closed read-back costs one turn per promise and catches T01-type amounts, so Tarazu made it policy (C12). The rule: an accuracy step that protects money or a regulated outcome moves to the compliance layer and stops being traded.
Which number is the north star?
Two numbers. Offline and in canaries, compliant verified promises to pay per right-party contact: a promise counts only if the stored amount and date match what the borrower confirmed and the call broke no rule. Online, kept promises per dial against a no-bot or human holdout.
The first (compliant verified PTP per RPC) is observable the day a build ships; the second (kept PTP per dial) is the business result. "Verified" is checked against the reference transcript, and "compliant" means zero C01-C16 failures on the call.
The denominator is half the metric. A common convention, from OpsDog, computes PTP rate as promises per RPC and kept rate as promises paid per promises made 5 6, but shops differ, so a PTP rate without its denominator cannot be compared. Tarazu's offline number uses RPC because it isolates the conversation: the dialer, connect rate and answering machine detection decide who reaches Asha, and her job starts there. The funnel denominators are owned by task success and outbound outcomes.
Why each word is in the name:
- Verified is what the T01 week lacked: verified PTP scores the Rs 2,000 promise zero, because the reference says "dhai hazaar".
- Compliant stops the agent buying promises with breaches, such as a promise after a threat (T09) or a stated hardship (T10).
- Per dial online puts every upstream stage back in, so an AMD or dialer change cannot flatter the result.
- Against a holdout turns kept PTP into lift: the bot-arm rate minus the holdout rate. Payments move with salary dates and festival weeks, and only a holdout separates the agent from the calendar. The arms are independent, so the difference gets Newcombe's interval. The kept rate (kept PTP per PTP) stays a diagnostic.
Verified PTP needs a reference transcript, which production has only for audited calls, so the offline number runs on gold-v7 nightly and on the audited canary sample (reweighting is in production monitoring). Kept PTP per dial needs only the dialer log and payments table, so it is the online number.
Which guardrails sit beside it?
Guardrails are the metrics a release may not make worse while it moves the north star, each with a direction, a band that triggers action and a denominator. Tarazu's cover compliance, complaints, entities, latency, turn-taking, hang-ups and transfers.
The general distinction is in eval vs monitoring vs guardrails. Bands are Tarazu policy; "not worse than control" means a paired test on shared gold scenarios (McNemar or a paired bootstrap) whose interval for the difference must stay inside the agreed margin.
| Guardrail | Direction | Band (Tarazu policy) | Denominator |
|---|
| Policy violation rate | Down | Zero on the gold set; any production C04 or C05 failure pages on-call | Connected calls |
| Complaint and DNC request rate | Down | Not worse than control | RPC calls |
| Amount and date entity error rate | Down | Not worse than control on gold-v7, speaker-clustered bootstrap | Amount and date mentions |
| P95 platform turn gap by turn type | Down | Inside the SLO: 1,900 ms non-tool and 2,600 ms tool turns at TW's 400 ms budget | Agent turns of that type |
| False-cutoff rate at the campaign budget | Down | Not worse than control; tripwire at Hindi-dominant above 4% for two weeks | Labelled hold pauses |
| Talk-over, missed barge-in and recovery failure rates | Down | Not worse than control, each reported separately | Caller turns during agent speech |
| Early hang-up rate (under 10 s after the opener) | Down | Not worse than control | Human-answered connects |
| Transfer rate | Two-sided | Within a band around control, by reason | RPC calls |
Every guardrail is reported per gated segment (3 language groups by 4 flow groups), because a release that helps Hinglish promise calls and breaks Hindi-dominant dispute calls must fail, and a pooled number hides that. Transfer rate gets a band on both sides: a rise can mean the agent is dodging hard calls, and a fall in hardship transfers can mean C09 is failing, since F04 requires a transfer. Sign-off and overrides follow release gates.
Which checks are code and which need a judge?
Use code when the truth sits in structured state or an exact string: tool order, CRM rows, timestamps, TTS input text. Use a judge when the check needs meaning a rule cannot pin down, and a person when the judge is not validated for that language or the stakes are regulatory.
Code runs first, on every call, and a judge never replaces a code check that can decide the same thing.
| Check type | When | Tarazu examples | Unit cost (illustrative) | Results arrive |
|---|
| Deterministic code and lexicon | Truth in structured state or an exact string | C01, C02, C04 order, C05, C10-C13, C15, C16; CRM diff; entity match; C07 lexicon | Rs 0.002 per call | At call end, 100% |
| Text judge | Meaning over the transcript | J3 pushed past a dispute (C08); J1 coercion beyond the lexicon (advisory) | Rs 0.15 per call per dimension | Async, stratified sample |
| Blueprint judge | Delivery as well as words | J2 pacing and empathy after hardship | Rs 0.40 per call | Async, stratified sample |
| Human audit | Unvalidated judge, regulatory stakes, calibration | J1 flags, disputes, verified PTP on canary | Rs 62.50 per 150 s call | Weekly queue |
Unit costs come from Tarazu's invented cost model, not vendor prices. A human audit costs 31,250 times a deterministic check, and a text judge 75 times; deterministic checks on all 1,30,000 daily connects cost Rs 260 a day [C]. Monthly costs per sampling policy are in evaluating under load, and its cost.
Three incidents show where each type earns its place:
- T03, dues stated to the borrower's wife, is a code check. Ordered events and TTS text decide C04 exactly, on every call; a judge would only add a miss rate.
- T09, "ghar pe log aayenge", needed all three. The lexicon had "police" and "jail" but not "ghar aa". A judge catches paraphrase, but J1 v1 failed its Hinglish validation (recall 0.833, Wilson lower bound 0.608 against a 0.90 bar), so its flags go to human audit and never gate. Sampling for rare paths is owned by production monitoring.
- T10, the money request after "meri naukri chali gayi", splits in two. The C09 part, a PTP request after stated hardship, is a tool-order check in code and runs first. The pacing part (5.9 syllables per second, started 180 ms after she stopped, over a sob) needs judge J2, which stays advisory until it passes per-language validation, owned by audio judges.
Write judge rubrics as narrow binary questions, not 1-to-5 scores (rubric design; dimensions in the conversational quality playbook).
How do you score one call at every layer?
Score each layer separately and keep every result. T01 fails entity, semantic, task and compliance and passes every experience check; a blended score would average that into something that looks fine.
| Layer | Metric on T01 | Result |
|---|
| Lexical | Utterance WER | 1/12 = 8.3% [C] |
| Entity | Amount entity accuracy | 0/1: "dhai hazaar" is 250000 paise, STT "do hazaar" is 200000 |
| Semantic | Meaning preserved on the caller turn | No: the promised amount changed |
| Task | CRM diff against the gold end state | Fail: 200000 paise and readback_confirmed=false; gold is 250000 and true |
| Experience | End-of-turn delay; LLM TTFT; TTS TTFB; platform turn gap | 370 ms (SLO 450 then); 670 ms (900 on tool turns); 190 ms (250); 1,800 ms (tool-turn SLO 300 + 2,200 = 2,500 ms at the old budget) [C]: all pass |
| Compliance | C12 read-back before write | Fail: record_promise_to_pay at 23,010 ms with no read-back |
Entity, task and C12 are exact code checks, so gate there; the semantic judge also catches it, at judge cost and judge error. The experience pass is information: the call felt fine, which is why he said "haan haan theek hai". WER cannot gate it, since 8.3% is below the 11% corpus mean. And C12 would have absorbed the STT error: a closed read-back ("Toh dhai hazaar rupaye, bees November tak. Sahi hai?") makes the borrower hear the wrong number before anything is written.
The production version is the metric registry, abbreviated:
| Metric | Layer | Unit | Denominator | Check type | Cadence | Owner |
|---|
| Compliant verified PTP per RPC | Task + compliance | Rate | RPC calls | Code on reference transcripts | Nightly sim; canary audit | Eval platform |
| Kept PTP per dial vs holdout | Business | Rate difference | Dials, both arms | Code (payments join) | Weekly, after the kept check | Collections analytics |
| Policy violation rate | Compliance | Rate | Connected calls | Code and lexicon on 100%; judges sampled | Daily | Compliance lead |
| Amount and date entity error rate | Entity | Rate | Entity mentions in gold-v7 | Code | Every release | STT owner |
| P95 platform turn gap by turn type | Experience | ms | Agent turns of the type | Code on spans | Daily | Latency owner |
| Transfer rate | Guardrail | Rate | RPC calls | Code (tool log) | Daily, by reason | Collections operations |
One owner and one denominator per row; a metric two teams own gets argued about instead of acted on.
How can the north star be gamed?
Three ways, each with a tripwire: cheap promises the borrower will not keep, an upstream classifier changing who counts as a contact, and hard calls handed to humans so the bot's numbers stay clean. An agent tuned against a metric finds these without anyone intending it.
Cheap promises. Raw PTP per RPC rewards any stored promise, so the agent learns to accept vague dates and treat backchannels as agreement. "Verified" closes most of it; what remains is confirmed, unrealistic promises. Tripwire: kept rate by weekly promise cohort, per segment, against the holdout's cohorts, plus the faster signal of raw PTP per RPC rising while the verified share falls.
Denominator drift through AMD. In T07 an operator announcement counted as a human connect and inflated the denominator for PTP per connect; the opposite drift, people classified as machines, drops calls from the RPC denominator, and when the dropped calls are the hard ones it flatters PTP per RPC. Tripwire: the online number is per dial, which AMD cannot move, and an offline-online gap triggers the AMD check in task success and outbound outcomes.
Transfers as an escape. A transferred angry or dispute call costs the bot little on the north star and removes its chance to fail C07 or C08. Rewarded on violation rate, the agent learns to transfer anything ambiguous. Tripwire: the two-sided transfer band by reason, plus human-queue load per segment.
A fourth route is organizational: if the team that ships the agent writes the references, "verified" drifts toward what STT heard, so references come from people who never see the agent's output.
How do you read a vendor's collections case study?
As a funnel with missing rows: ask for every rate's denominator, a holdout arm, the kept or cure rate behind any promise rate, and the regulatory regime. Two published case studies show the gaps.
CAUTION
Vendor-reported. Treat it as a hypothesis to test on your own calls.
Gnani.ai, a voice AI vendor, says in a self-reported case study (published July 2025, checked November 2026) that a "$6B Leading Indian Bank" collected Rs 300 Cr+ in three months over 12.5M+ calls with "81% Connectivity", defined on the page as 81% of unique customers reached per dial 7. The page gives no holdout, PTP, kept-promise or cure figures, so the number is gross collections, not lift, and Gnani's site navigation describes the same story as "55% of dues recovered across diverse loan categories", a different metric.
Skit.ai, a voice AI vendor, reports in a self-published case study (published April 2026, checked November 2026) that SameDay Auto Finance, a Dallas-based US auto lender working mostly early-delinquency accounts, saw its PTP rate go from 5.7% to 11.5%, its collection rate rise 43% and collection call cost fall 75%, with 68% connectivity at 2 attempts per account per day, after a four-phase deployment over a year 8. The page does not state the PTP denominator or any kept-promise rate, and the deployment runs under US rules, not Indian ones.
Before either number reaches a deck, ask Gnani what the same accounts would have paid with no bot, which of "81% connectivity" and "55% of dues recovered" is the claim, and how many of the calls were right-party contacts. Ask Skit whether PTP is per dial, connect or RPC, how many promises were kept, and how much of the 43% came from the bot rather than a year of other changes.
What gate would have caught it?
The amount entity layer plus C12 on the gold set. A release candidate must not raise amount or date entity error rate on the gold-v7 reference utterances, and the nightly audio scenarios must show zero promise writes before a confirmed read-back. Either one blocks the T01 build.
Tarazu's gate for any change to STT, the prompt or the tools:
- Entity layer on gold-v7. The 1,200 reference-transcribed utterances from 180 speakers are scored against control. Utterances from one speaker are correlated, so the interval is a speaker-clustered bootstrap, never Wilson; worse than control beyond the margin in any language group fails.
- C12 on every promise scenario. The nightly run has 240 scenarios, 20 per gated segment, so 60 in the promise flow group (F01 and F09) across three language groups. record_promise_to_pay must follow a closed read-back and an explicit yes, and the stored values must equal the confirmed ones. Any failure blocks.
- Task layer on the same scenarios. CRM diff with the verified-PTP definition, per segment.
- WER reported, not gated. It goes in the release notes to explain the entity result.
Zero C12 failures across the 60 promise-group scenarios bounds the failure rate at 4.87% (one-sided 95%; rule of three gives 5%), and one segment's 20 scenarios at 13.9% [C]. Zero in a night means "not common", and the backstop is the same C12 check on 100% of real calls, which turns a rare failure into a counted one. Candidate against control on shared scenarios is a paired comparison (McNemar or a paired bootstrap); one segment's pass rate gets a Wilson interval.
What goes in the decision memo?
The memo records the metric set as a decision: which number leads, which guard it, who owns each, and what reopens it. Tarazu's north-star memo:
Decision. North star: compliant verified PTP per RPC offline and in canaries; kept PTP per dial against a holdout online. Raw PTP per RPC becomes a diagnostic. WER leaves the release gate; amount and date entity error rate and C12 join it.
Context. In the T01 week, corpus WER held at 11% and PTP per RPC rose 6 points; the kept rate fell two weeks later (illustrative).
Options considered. (a) Raw PTP per RPC with a WER gate: what failed. (b) Kept rate as the offline gate: weeks late. (c) The two-number north star: chosen.
Evidence. gold-v7 (600 scenario cards, 1,200 reference utterances); T01 scored at every layer. Speaker-clustered bootstrap for entity error rate, McNemar for candidate versus control, Newcombe for bot versus holdout. A clean night on the 60 promise scenarios bounds C12 failure at 4.87% (one-sided 95%), so the 100% production C12 check stays.
Owner and cadence. Eval platform staff engineer owns definitions and the gate; collections analytics owns the holdout and weekly kept readout. Nightly 240 scenarios; canary at 5% of traffic.
Tripwires. Raw PTP per RPC up while the verified share falls; cohort kept rate below holdout; transfer rate outside its band; Hindi-dominant false cutoff above 4% for two weeks; any production C04 or C05.
Dated facts. Cresta layers (May 2026); EVA paper (September 2026) and leaderboard (late October 2026); OpsDog conventions (checked November 2026).
What to do this week
Write down your north star with its full denominator, then score your last five incidents at every layer. A failure on a layer you do not gate on names your next gate.
- Add a verified flag to every stored promise, computed on the gold set and the audited sample.
- Write per-call AND-gates for accuracy and experience, and stop blending them.
- Move WER to the release notes; gate on amount and date entity error rate with a speaker-clustered interval.
- Mark every compliance check code, judge or human, and rewrite as code any judge-only check code could decide.
- Give transfer rate a two-sided band by reason.
NOTE
What will date on this page. EVA's leaderboard result for Cartesia Managed Agents with GPT 5.5 (0.87 EVA-A, 0.91 EVA-X pass@1; updated late October 2026, checked 2026-11-20); new entries may change who leads. EVA-Bench paper figures (arXiv v3, 9 September 2026, Findings of EMNLP 2026) and the March 2026 launch results. Cresta's layers and comparison order (blog of 29 May 2026, checked 2026-11-20). OpsDog's PTP and kept-rate conventions (checked 2026-11-20). Gnani's case study (published 25 July 2025, modified 19 November 2026) and Skit.ai's (published 7 April 2026, modified 30 June 2026), both checked 2026-11-20.