On trace T07, answering machine detection (AMD) labelled a carrier's "number busy" announcement as a human, Asha played her opener to a recording for 11 seconds, and the attempt went into the books as a human connect. On trace T01, the promise write returned "ok" and the call closed with a promise in the CRM, but the stored amount was Rs 500 short of what the borrower said and it was written before any read-back.
What happened on the calls?
T07 never reached a person, yet counted as one. T01 reached the right person and wrote a wrong record. Neither shows up if you score what the agent said or what its tools returned.
T07, 2026-11-10, 10:12, persona P09, flow F08:
t_ms ch event / text
900 caller carrier announcement: "Aap jis number se sampark karna chahte hain woh abhi vyast hai, kripya thodi der baad call karein."
2380 agent amd=human (confidence 0.71)
2600 agent TTS opener plays to the announcement (11 s)
13900 agent no reply detected; agent repeats "Hello, kya aap mujhe sun pa rahe hain?"
21000 agent end_call(no_response); mark_contact not set; attempt counted as a human connect
Diagnosis: an AMD tuned on US-style voicemail greetings called an Indian operator announcement human, at a cost of 21 s of trunk time, an attempt counted against C11, and an inflated denominator for PTP per connect.
T01, 2026-11-14, persona P01, flow F01. The trace stores the CRM rows before and after the call. Diffed against the scenario's gold end state, the call fails on two fields:
table ptps (crm_before: no rows; one row added)
field expected actual
amount_paise 250000 200000 MISMATCH (Rs 500 short)
ptp_date 2026-11-20 2026-11-20 ok
readback_confirmed true false MISMATCH (written before read-back)
tool log: record_promise_to_pay -> ok
result: task FAIL; spoken checks scored separately
Diagnosis: speech-to-text (STT) heard "dhai hazaar" as "do hazaar", the promise was written before a closed read-back, and "haan haan theek hai" was a backchannel, not a yes. The entity side is in entity error rate; this page is about the record.
How do you score task success on a voice call?
Score the state of the world after the call, not the words. Diff the CRM before and after the call against the scenario's gold end state: rows that must be written, rows that may be, and rows that must not change. Spoken obligations, such as naming the grievance officer, are separate checks with their own results.
The idea comes from tau-bench, which scores each task with a rule-based check: the final database must match the annotated goal state and, where a task annotates required outputs, the agent's replies must contain them as substrings 1. tau-Voice keeps tau2-bench's final-database comparison 2 but checks the agent's spoken communications with an LLM evaluator instead of string matching, so its task scoring is not fully deterministic, even though the paper's metrics section describes success as deterministically evaluated 3. The tau-bench recipe shows final-state scoring on a text agent; the voice version adds the spoken checks and a real CRM.
Tarazu keeps the two halves apart. The CRM diff is deterministic code. Spoken checks run on the TTS input text or agent-channel STT, and some need a judge; fold a judge's variance into the task number and a diff regression becomes impossible to separate from judge noise. Report both, per call:
- Task pass, spoken fail: the PTP row is right, but Asha never named the grievance officer (C06). A CRM-only scorer passes it.
- Task fail, spoken pass: a correct read-back, a clear yes, and a write that never landed. A transcript-only scorer passes it.
The north star requires both rather than averaging them; that choice is made in layered metrics for voice agents.
Do not score the tool's return value. T01's record_promise_to_pay returned "ok" and wrote the wrong row. Vapi's testing guide warns that "A tool reporting success does not independently prove that an external system changed" 4. Vapi says this about what a simulation judge can see; applying it to promises is this course's step. The tool log tells you which call to blame; the CRM row is the evidence.
Four details decide whether a CRM diff can be trusted:
- Snapshot after the writes settle. If writes go through a queue, read the after-state from the change log keyed by
attempt_id, or a correct call scores as a missing write.
- Ignore ids and timestamps by name, never by a wildcard that also hides a wrong
ptp_date.
- Match rows as sets and fail on extras. A timeout and a retry can write two
ptps rows behind one "ok".
- Some fields are predicates. A callback time is right if it sits inside the calling window (08:00-19:00 local, MFI 09:00-18:00) or the borrower asked for that time.
Outcome versus path scoring is covered in trajectory vs outcome evaluation and argument checks in tool call evaluation. The path still matters where a flow requires tools in order, such as check_payment_status first on an "already paid" claim.
What does the gold end state look like for each flow?
Each flow gets three lists: what must be written, what may be written, and what must not change. Any write not on the first two lists fails the call. Branches that depend on data, such as whether a payment exists, read the before snapshot, not the transcript.
| Flow | Must write | May write | Must not change |
|---|
| F01 Happy path | One ptps row with the amount and date the borrower confirmed, readback_confirmed=true; payment link sent; grievance SMS in sms_log | Nothing else | grievances, hardship, dnc, recovery_hold |
| F02 Already paid | If payments has it: grievance SMS, disposition paid. If not: grievances row (already_paid), recovery_hold=true, grievance SMS | Nothing else | ptps in both branches; recovery_hold in the found branch |
| F03 Dispute | grievances row (amount_wrong), recovery_hold=true, grievance SMS | Nothing else | ptps |
| F04 Hardship | hardship row, grievance SMS, transfer to the hardship queue | Nothing else | ptps |
| F05 Family member | Disposition third_party, rpc=false | A neutral callbacks row inside the window | ptps, contacts.verified_at |
| F06 Guarantor | contacts.verified_at on the guarantor's row, grievance SMS | callbacks or a transfer; ptps only with his explicit agreement | Nothing extra; the order check fails dues fetched before verification |
| F07 Voicemail | amd_label voicemail, disposition voicemail, grievance SMS | Nothing else | ptps, rpc=true |
| F08 IVR or announcement | amd_label set to the true class (operator announcement or ivr), disposition ivr, rpc=false | Nothing else | sms_log, ptps |
| F09 Callback request | callbacks row, requested_by=borrower, inside the window unless he asked otherwise; grievance SMS if verified | Nothing else | ptps |
| F10 Complaint threat | Grievance SMS | dnc with scope bot, or a transfer to the grievance queue | ptps |
Three rows cause most scorer bugs. F02 has two gold states chosen by the payments table before the call; a spec that reads the transcript ("she said she paid") passes an agent that logs a dispute for a payment already in the ledger. F06 allows a promise row, and whether the guarantor agreed explicitly is a spoken check on the reference transcript. F05 and F08 must write rpc=false, because one wrong flag moves every per-RPC and per-connect rate above it. The recording and retention checks on call_attempts belong to safety, privacy and compliance checks.
On a simulated call the gold state comes from the scenario card. On a real call it comes from the reference transcript of an audited call: the amount and date the borrower confirmed after an explicit yes. That is what "verified PTP" means. The collections task checker recipe implements the diff for all ten flows and runs it on seeded calls, including T01, T02 and T07.
What are RPC, PTP and kept rate, and what are their denominators?
A right-party contact (RPC) is a call that reaches someone allowed to discuss the account. A promise to pay (PTP) is a recorded commitment to pay an amount by a date. The kept rate is promises paid over promises made. Denominators vary by shop, so a rate quoted without one cannot be compared with anything.
A common convention, from OpsDog, computes PTP rate as promises per RPC 5 and kept rate as promises paid per promises made 6. The RPC rate itself is RPCs over dials in some shops and RPCs over connected calls in others 7 5. None of this is a standard. Write the denominator into the metric name (ptp_per_rpc, kept_per_ptp, kept_ptp_per_dial) and never ship a dashboard tile called "PTP rate".
Who counts as a right party is a choice too. OpsDog's definition includes co-borrowers and trusted parties such as legal counsel. RBI's recovery directions for NBFCs, effective 1 January 2027, allow loan dues to be discussed only with the borrower or guarantor 8. So Pooja Verma answering Rahul's phone (P05) is a human answer, not an RPC. No cited source says whether those directions bind an AI agent; Tarazu applies them to Asha as policy. As context only, Exotel, a voice-platform vendor, reads RBI's 2022 Digital Lending Guidelines as binding AI voice bots the same way they bind human agents, which is Exotel's interpretation, not RBI wording 9.
Tarazu's definitions:
- RPC: the borrower or a guarantor, with
verify_identity returning verified=true. A borrower who will not verify is a claimed right party, since dues cannot be discussed with them.
- PTP: a
ptps row; a verified PTP's stored amount and date equal what the borrower confirmed on the reference transcript.
- Kept rate: kept PTP per PTP, set by the kept check after the promise date.
Tarazu's bot works the 1-30 and 31-90 days past due (DPD) buckets. Report every rate by DPD bucket as a monitored slice: early and late buckets promise and pay at different rates, so a campaign mix shift moves the pooled rate with no change in the agent.
Where does each eval layer sit in the funnel?
Each step of the outbound funnel has its own eval and each rate its own denominator: dials, connects, human answers, RPCs, promises, payments. An eval that watches one step cannot see a failure at another.
At Tarazu's daily volumes (26 calling days a month): 2,50,000 dials; 52% connect, so 1,30,000 calls answered by anything; 88% of those are a human, so 1,14,400; 70% of human answers are a right party, so 80,080 RPCs; 38% of RPCs promise, about 30,430 PTPs; 62% of promises are kept, about 18,870. So kept PTP per dial is about 7.5%, RPC per dial 32.0% and PTP per connect 23.4%.
Figure: Synthetic Tarazu data. The daily outbound funnel, each rate's denominator, and the eval that guards each step. Only the last three steps are the conversation; the first three decide who the conversation is with.
| Step | What can go wrong | Eval | Owner chapter |
|---|
| Dial -> connect | Call outside the window, over the attempt cap, wrong CLI | Deterministic dialer checks (C01, C11, C16) | Compliance checks |
| Connect -> human | Machine taken for a person, or a person for a machine | AMD confusion matrix with costs | This page |
| Human -> RPC | Dues to a family member; guarantor not verified | Party handling and the order check before any dues (C04, C05) | Compliance checks |
| RPC -> PTP | Wrong amount heard, cut off mid-date, loop on a clarification | Entity error rate, read-back order (C12), turn-taking and latency guardrails | Entity error rate, turn-taking |
| PTP -> record | Wrong row, missing row, duplicate row | CRM diff; verified-PTP match rate | This page |
| Record -> kept | Promise made but not paid | Kept check; kept PTP per dial against a holdout | This page |
The rule: a rate's denominator should start at the first step the thing you evaluate controls. PTP per RPC is the conversation. PTP per connect mixes in AMD. Kept PTP per dial mixes in everything, which makes it the online business number, not a release gate.
How do you evaluate answering machine detection?
Treat AMD as six classes, not two: human, voicemail, operator announcement, caller tune, interactive voice response (IVR) and silence. Score it as a confusion matrix on labelled openings from your own carriers, and price each error: dropping a borrower and talking to a recording cost different things.
CAUTION
Vendor-reported. Treat it as a hypothesis to test on your own calls.
LiveKit reports a macro F1 of 97.0% (micro F1 and accuracy 94.7%, human-class F1 95.7%) for its AMD, using its default Gemini 3.1 Flash-Lite plus Cartesia Ink-Whisper pairing on an undisclosed internal dataset (May 2026); the dataset's languages are not named, so the figure says nothing specific about Hinglish calls 10. Twilio's documentation says AMD will not always return the right answer, with the example of a two-second voicemail greeting read as a human; its AnsweredBy values are human, machine_start, machine_end_beep, machine_end_silence, machine_end_other, fax or unknown 11. Its FAQ calls DetectMessageEnd close to 100% accurate in US destinations with default settings, and says accuracy may be reduced for international dialing because voicemail tones differ by country 12. Twilio publishes no measured accuracy for India, and none of its labels is an operator announcement, a caller tune or an IVR. Map your vendor's labels onto the six classes before you measure anything.
The matrix below is the synthetic stand-in for gold-v7's 200 labelled openings from the collections task checker recipe (seed 20261120), current AMD setting A. Rows are the true class, columns the predicted class:
| True \ predicted | Human | Voicemail | Announcement | Caller tune | IVR | Silence | Recall [Wilson 95%] |
|---|
| Human (80) | 77 | 1 | 0 | 0 | 0 | 2 | 0.96 [0.90, 0.99] |
| Voicemail (40) | 6 | 31 | 1 | 0 | 2 | 0 | 0.78 [0.62, 0.88] |
| Operator announcement (25) | 8 | 2 | 14 | 0 | 1 | 0 | 0.56 [0.37, 0.73] |
| Caller tune (20) | 1 | 2 | 0 | 14 | 0 | 3 | 0.70 [0.48, 0.85] |
| IVR (20) | 3 | 0 | 2 | 0 | 15 | 0 | 0.75 [0.53, 0.89] |
| Silence (15) | 2 | 1 | 0 | 0 | 0 | 12 | 0.80 [0.55, 0.93] |
The off-diagonal cells have different prices, so keep them apart instead of reporting one accuracy:
| Error | Count (setting A) | What it costs |
|---|
| Human taken for a machine (false machine) | 3 of 80 | The borrower is dropped before the opener. At Tarazu's funnel rates each one is 0.70 expected RPCs, 0.266 expected promises and 0.165 expected kept promises lost, plus a redial against the attempt policy |
| Machine taken for a human (false human) | 20 of 120 | T07's 21 s of trunk time each, a connected-call slot under C11, an inflated connect count; on a voicemail, the opener is recorded with the lender's name and the borrower's first name, and the neutral voicemail message is never left |
| Machine taken for another machine | 14 of 120 | A voicemail read as an IVR leaves no message; an announcement read as a voicemail gets a message and a grievance SMS it did not need |
The operator-announcement row is T07 eight times over: 8 of 25, a false-human rate of 32%, Wilson [17%, 52%]. And 25 openings cannot sign anything off: zero false humans in 25 still leaves an exact one-sided 95% upper bound of 11.3%. Showing a rate below 5% takes 59 announcements with zero errors on that exact bound (73 on a two-sided 95% Wilson bound), and 204 with a two-sided Wilson bound when 4 of them (2%) are errors. Use a 200-opening matrix to rank classes and choose what to label next; gate on a stratum big enough to bound.
Announcement wording and caller tunes differ by operator, so carrier is a monitored slice on the matrix even when it is too thin to gate.
A worked example shows why the denominator matters. The recipe projects one day under setting A and a stricter setting B that demands more evidence before it says human, with the same agent, prompt and model:
| Setting | Calls Asha handled | Machines engaged | RPC | PTP | PTP per handled call | PTP per RPC | RPC per dial |
|---|
| A (current) | 1,12,710 | 2,600 | 77,077 | 29,289 | 26.0% | 38% | 30.8% |
| B (stricter human) | 1,08,420 | 1,170 | 75,075 | 28,528 | 26.3% | 38% | 30.0% |
A dashboard that counts a connect as a call AMD passed to Asha, the denominator T07 inflated, shows PTP per connect up 0.3 points under B, and a team watching it would call B a better agent. B drops more real borrowers before the opener: 2,002 RPCs and 761 promises a day, about 470 kept promises, to save 8.3 trunk hours. The denominator shrank faster than the numerator.
PTP per RPC stays at 38%, which correctly says the conversation did not change, and is blind to the borrowers B never reached; if the dropped humans were harder calls it would even rise. RPC per dial moves for the right reason, 30.8% to 30.0%. Judge an AMD change on the costed matrix and RPC per dial, the agent on PTP per RPC, and neither on PTP per handled connect. B's break-even is one lost RPC per 15 trunk seconds, a bad price for a borrower conversation.
What is special about voicemail?
A voicemail is a recording someone else may hear, so it must say nothing about the debt and still carry the grievance details. Tarazu treats it as a recovery-related communication: it speaks the grievance officer's name and phone, the SMS carries the rest, and it never says an amount, product, DPD or "overdue".
Two messages left on P08's carrier voicemail:
compliant TTS: "Namaste, yeh Tarazu Finance se ek sandesh hai. Kripya hamein 1600-XXX-XXXX par call karein. Kisi shikayat ke liye hamare grievance officer Ms Neha Kulkarni se 020-4000-0000 par sampark karein; poori jaankari SMS mein bheji ja rahi hai. Dhanyavaad."
tools: send_grievance_sms -> mark_contact(voicemail) (passes C04 and C06)
non-compliant TTS: "Rahul ji, aapka EMI overdue hai, kripya chaar hazaar do sau pachaas rupaye jaldi bharein." (C04 fail)
C04 forbids any product, amount, DPD or "overdue" before verify_identity returns verified, and nobody verifies on a voicemail. C06 requires the grievance redressal officer's name, email, phone and address in all recovery-related communications 8; treating a voicemail as one is Tarazu's reading, not RBI wording. Asha speaks the name and phone and send_grievance_sms delivers all four. The compliant message names no borrower, since whoever checks a shared phone hears it.
What to check on every call AMD labels voicemail:
- Content, on the TTS input text: lender name, callback number, grievance officer name and phone present; no amount, date of dues, product, DPD or "overdue"; no borrower name.
- Records, in the diff: a grievance SMS in
sms_log, mark_contact(voicemail), no ptps row, rpc=false.
- Timing, on the recording's agent channel: the message starts after the beep. P08's beep comes after 9 s; a message started during the greeting loses its first half, which may be the half with the grievance details.
- Class, on the opening: a voicemail that AMD called human gets the opener, not the neutral message, and fails the flow though no dues rule broke.
The C04 and C06 check code lives in safety, privacy and compliance checks; this page owns which records a voicemail call must leave.
How do you know a promise was good?
Three tests, in order of how long they take. At call time, a promise is verified if the stored amount and date match what the borrower confirmed after an explicit yes. After the promise date plus grace, it is kept if payments cover it. Months later, the program is good only if kept PTP per dial beats a holdout.
Pin the kept check's rules before anyone reads a kept rate:
- When it runs. At
ptp_date plus a grace period covering the gap between payments.paid_on and posted_at; run too early and late postings read as broken. The grace length is a collections decision, versioned with the checker.
- What counts as kept. Payments with
paid_on between the promise and ptp_date summing to at least amount_paise, on any channel. A partial payment is broken; report the share paid beside the rate.
- Re-promises. A moved date is a new promise. Mark the old one superseded, not broken, or one borrower fails twice.
- Payments with no promise. P02 paid before the call: the found branch of F02, not a kept PTP.
The agent controls the kept rate's denominator. Book fewer, safer promises and the kept rate rises while collections fall; book every vague "haan" and PTP per RPC rises while the kept rate falls, the T01-week pattern in layered metrics. Kept rate is a diagnostic, not the north star.
The business result needs a holdout, because payments move with salary dates and festival weeks and a bot-arm rate alone cannot separate the agent from the calendar:
- Randomize accounts, not calls, or one borrower lands in both arms.
- Stratify by campaign and DPD bucket.
- Name the holdout's treatment. A human-agent holdout makes kept PTP per dial comparable; a no-call holdout has no promises, so compare payment per account.
- Fix the outcome window in advance, to the last
ptp_date plus grace, or unexpired promises count as broken.
- Use Newcombe's interval for the difference of two independent arms. Arm sizes are in reliability statistics for voice agents.
The holdout runs for weeks and says whether the program pays. The verified-PTP check runs tonight and says whether this release writes the promises borrowers made.
What gate would have caught them?
One gate per incident. For T07, an AMD gate blocks any AMD build, setting or carrier change whose upper bound on false humans among operator announcements, or false machines among humans, exceeds the signed threshold. For T01, a verified-PTP match-rate gate on gold-v7, backed by the C12 order check on every call.
gate: outbound-task
amd:
openings: gold-v7 openings, plus a growing operator-announcement stratum by carrier
block_if:
- wilson_upper(false_human | operator_announcement) > signed_threshold
- wilson_upper(false_machine | human) > signed_threshold
- mcnemar(candidate vs incumbent, same openings) shows more false machines
report_always: [six_class_matrix, cost_table, rpc_per_dial_projection, per_carrier_matrix]
task:
scenarios: gold-v7, 600 cards, nightly 240 at k=4
scorer: crm-diff (pinned by hash), spoken checks reported separately
block_if:
- verified_ptp_match_rate wilson_upper_of_misses > signed_threshold # per gated segment
- any unexpected write in F03, F04, F07, F08 # dispute, hardship, machines
online: C12 order check on 100% of calls; verified-PTP audit on the human-audited sample
The verified-PTP match rate, promises whose stored amount and date equal the confirmed values over promises written, is a single proportion per segment, so Wilson applies. Candidate versus incumbent on the same scenarios is paired: McNemar or a paired bootstrap.
T07 is the operator-announcement row: 8 of 25 fails any reasonable threshold. T01 fails the diff on two fields and C12 on order, and C12 catches it on the live call. Online, add a label-free monitor for T07's signature: a call labelled human with no caller reply to the opener and no mark_contact.
What goes in the decision memo?
The memo is the AMD operating point, priced in borrowers and trunk seconds, not accuracy points.
Decision. Keep AMD setting A's global threshold. Do not ship setting B. Add an operator-announcement class trained on Indian carrier audio and gate it on its own row.
Context. T07: an operator announcement labelled human at confidence 0.71 heard the opener for 11 s and counted as a human connect.
Options considered. (1) Setting B, stricter on human: 1,430 fewer machines engaged a day, 2,002 fewer RPCs. (2) Keep A unchanged: T07 recurs. (3) A plus an announcement class and a labelled stratum: chosen.
Evidence. Synthetic stand-in for gold-v7's 200 openings (recipe seed 20261120). A: false machine 3 of 80, false human 20 of 120, announcement false human 8 of 25 (32%, Wilson [17%, 52%]). B: false machine 5 of 80, false human 9 of 120. Break-even for B: one lost RPC per 15 trunk seconds.
Owner and cadence. Telephony owner runs AMD; the eval platform staff engineer owns the gate; the collections lead signs thresholds. Rerun on every AMD or carrier change and monthly on fresh openings.
Tripwires. RPC per dial falls while PTP per RPC holds; handled-call count changes with flat dials; a new carrier announcement; any call labelled human with no reply and no mark_contact.
Dated facts. LiveKit AMD figures (May 2026); Twilio AMD docs (modified 19 August 2026) and FAQ (checked 2026-11-20); RBI/2026-27/230, effective 1 January 2027.
What to do this week
Start with the two measurements most teams skip: a labelled AMD matrix on your own carriers, and a field-level diff of what each call left in the CRM. Both run on calls you already have.
- Label 100 call openings per carrier in the six classes and build the matrix for your current AMD setting before touching its threshold.
- Write must, may and must-not lists for your top three flows, and make unlisted writes fail.
- Rename every promise metric to carry its denominator, and put RPC per dial next to PTP per RPC.
- Check five of last week's promises by hand against their reference transcripts.
- Write down the kept check's grace period, partial-payment and re-promise rules, and get them signed.
NOTE
What will date on this page. LiveKit's AMD figures are from its May 2026 blog post. Twilio's AMD documentation was modified 19 August 2026 and its FAQ is undated (both checked 2026-11-20); AnsweredBy values and the US accuracy claim may change. Vapi's testing guide is undated (checked 2026-11-20). OpsDog's PTP and kept-rate conventions and Call Centre Helper's RPC definition were checked 2026-11-20. RBI/2026-27/230 takes effect on 1 January 2027; recheck for later amendments. The tau-Voice scoring description is from the March 2026 paper. Recipe numbers are tied to seed 20261120.