On trace T07, a phone's call-screening assistant asked Hazel to say her name and why she was calling; answering machine detection (AMD) labeled it a human, Hazel spoke her opener to the assistant for 11 seconds, and the dialer logged the attempt as a conversation, which blocked the account for a week. On trace T01, the promise write returned "ok", but the stored amount was $35.00 less than the borrower offered and it was written before any read-back.
What happened on the calls?
T07 counted a machine as a person and then spent a week of contact budget on it. T01 reached the right person and stored the wrong promise. A scorer that reads what Hazel said, or what her tools returned, passes both.
T07, 2026-11-10, 10:12, persona P09, flow F08:
t_ms ch event / text
900 caller call-screening assistant: "Hi, the person you're calling is using a screening service and will get a copy of this conversation. Go ahead and say your name and why you're calling."
2380 agent amd=human (confidence 0.71)
2600 agent TTS opener plays to the assistant (11 s)
13900 agent no reply detected; agent repeats "Hello, can you hear me?"
21000 agent end_call(no_response); mark_contact not set; attempt logged as a human connect and as a conversation
Diagnosis: an AMD tuned on voicemail greetings had never been scored on screening prompts. The call cost 21 s of trunk time and one placed call against the 7-in-7 cap, and because the record said "conversation", the dialer started Steelyard's 7-day no-call period for the account 1 2. It also inflated the count behind PTP per connect.
T01, 2026-11-14, persona P01, flow F01. The trace keeps the CRM rows from before and after the call, and the diff against the scenario's gold end state flags two fields:
table ptps (crm_before: no rows; one row added)
field expected actual
amount_cents 25000 21500 MISMATCH ($35.00 short)
ptp_date 2026-11-20 2026-11-20 ok
readback_confirmed true false MISMATCH (written before read-back)
tool log: record_promise_to_pay -> ok
result: task FAIL; spoken checks scored separately
Diagnosis: speech-to-text (STT) turned Ryan Mitchell's "two fifty" into "two fifteen", Hazel wrote the promise before asking a closed read-back question, and his "uh-huh, okay" was a backchannel, not consent. The STT side of that failure belongs to entity error rate. This chapter is about the record it left.
How do you score task success on a voice call?
Compare the CRM after the call with the scenario's gold end state, field by field, and ignore the wording. The gold state lists rows that must appear, rows that may appear and rows that must stay untouched. Obligations Hazel must speak, such as naming the Customer Advocate, are scored as separate checks.
This is the tau-bench approach. Each task there passes on a rule-based check that the final database matches the annotated goal state, plus a substring check on the agent's replies where the task lists required outputs 3. tau-Voice inherits tau2-bench's final-database comparison 4 and grades spoken communications with an LLM evaluator rather than string matching. Its task score is therefore not fully deterministic, although the paper's metrics section calls success deterministically evaluated 5. For a text agent scored on final state, see the tau-bench recipe. A voice agent adds the spoken checks and a production CRM.
Steelyard runs the two halves separately. The diff is plain code with no randomness. Spoken checks read the TTS input text or the agent channel's STT, and a few need a judge. Mix a judge's noise into the task number and nobody can tell a broken write from a judge that changed its mind. Every call reports both results:
- Record right, speech wrong: the
ptps row matches, but Hazel never named Dana Brooks, the Customer Advocate (C06). A scorer that reads only the CRM passes it.
- Speech right, record wrong: a clean read-back, a clear "yes, that's right", and a write that never reached the CRM. A scorer that reads only the transcript passes it.
The north star needs both to pass; why it is not an average of the two is argued in layered metrics for voice agents.
Never take the tool's return value as the result. In T01, record_promise_to_pay said "ok" and stored $215.00. Vapi's testing guide puts it plainly: "A tool reporting success does not independently prove that an external system changed" 6. Vapi wrote that about the evidence a simulation judge has; using it for promise records is this course's step. The tool log says which call to look at, and the CRM row is what you score.
A diff earns trust only when four details are settled:
- Read the after-state once writes have landed. If writes go through a queue, take the after-state from the change log keyed by
attempt_id; otherwise a correct call scores as a missing write.
- Exclude ids and timestamps by field name. A wildcard that skips "anything time-like" also skips a wrong
ptp_date.
- Compare rows as sets and fail on extras. A timeout followed by a retry can write two
ptps rows behind a single "ok".
- Score some fields as predicates. A callback time passes if it falls inside 08:00-21:00 at the borrower's location (Massachusetts accounts on Eastern time, the SDL book 09:00-20:00), or if the borrower asked for that time.
Outcome scoring against path scoring is the subject of trajectory vs outcome evaluation, and argument checks of tool call evaluation. Order still matters in some flows: an "already paid" claim must run check_payment_status before anything else.
What does the gold end state look like for each flow?
Every flow has a must-write list, a may-write list and a must-not-change list, and any write on neither of the first two fails the call. Where the right outcome depends on data, such as whether a payment is on file, the scorer reads the before snapshot. The transcript cannot decide it.
| Flow | Must write | May write | Must not change |
|---|
| F01 Happy path | One ptps row with the confirmed amount_cents and date, readback_confirmed=true; payment link sent; disclosure SMS in sms_log | Nothing else | grievances, hardship, dnc, recovery_hold |
| F02 Already paid | Payment on file: disclosure SMS, disposition paid. Not on file: grievances row (already_paid), recovery_hold=true, disclosure SMS | Nothing else | ptps in both branches; recovery_hold when the payment is found |
| F03 Dispute | grievances row (amount_wrong), recovery_hold=true, disclosure SMS | Nothing else | ptps |
| F04 Hardship | hardship row, disclosure SMS, transfer to the hardship queue | Nothing else | ptps |
| F05 Family member | Disposition third_party, rpc=false | One neutral callbacks row inside the window | ptps, contacts.verified_at, sms_log |
| F06 Guarantor | contacts.verified_at on the guarantor's row, disclosure SMS | callbacks or a transfer; ptps only if he explicitly agrees | Nothing else; the order check fails dues fetched before verification |
| F07 Voicemail | amd_label voicemail, mark_contact(voicemail), counts_toward_cap=true | Nothing else | sms_log, ptps, rpc=true, conversation=true |
| F08 Screening, intercept or IVR | amd_label set to the true class, mark_contact(ivr), rpc=false, conversation=false | Nothing else | sms_log, ptps |
| F09 Callback request | callbacks row with requested_by=borrower, inside the window unless he asked for another time; disclosure SMS if verified | Nothing else | ptps |
| F10 Complaint threat | Disclosure SMS | dnc with scope bot, or a transfer to the Customer Advocate team | ptps |
Most scorer bugs live in four rows. F02 has two correct end states, picked by the payments table before the call. A spec that keys off the transcript (Sofia Ramirez saying "ya lo pague" [I already paid it]) rewards an agent that opens a dispute for a payment the ledger already holds. F06 permits a promise row, and whether the guarantor agreed explicitly is a spoken check against the reference transcript. F05 and F08 must write rpc=false, since one wrong flag shifts every rate computed per RPC or per connect.
F08 carries the field T07 got wrong. Steelyard's call_attempts table records connect_outcome, counts_toward_cap and conversation, because Regulation F counts placed calls rather than conversations: unanswered calls and voicemails count toward the 7-in-7 presumption, busy and out-of-service calls do not 2, and a conversation about the debt starts a 7-day no-call period 1. A carrier intercept that says the number is not in service is an out-of-service attempt, so counts_toward_cap is false. A call-screening assistant picked up the line, so the attempt counts. Neither one is a conversation. Recording and retention checks on call_attempts live in safety, privacy and compliance checks.
On a simulated call, the scenario card supplies the gold state. On a real call it comes from the reference transcript of an audited call: the amount and date the borrower confirmed after an explicit yes, which is what "verified PTP" means here. The collections task checker recipe implements the diff for all ten flows and runs it on seeded calls that include T01, T02 and T07.
What are RPC, PTP and kept rate, and what are their denominators?
A right-party contact (RPC) reaches a person allowed to discuss the account. A promise to pay (PTP) is a recorded commitment to pay a stated amount by a stated date. The kept rate divides promises paid by promises made. Each shop picks its own denominators, so a rate quoted without one compares with nothing.
OpsDog's convention puts PTP rate over RPCs 7 and kept rate as promises paid over promises made 8. The RPC rate itself is RPCs over dials in some shops and over connected calls in others 9 7. We found no US statute, regulation or regulator guidance that defines RPC, PTP or kept rate, so this course treats them as its own definitions, not a standard. Put the denominator in the metric name (ptp_per_rpc, kept_per_ptp, kept_ptp_per_dial) and retire any dashboard tile labeled just "PTP rate".
The right party is also a policy choice. OpsDog counts co-borrowers and trusted parties such as legal counsel. Steelyard collects its own debts in its own name, so the FDCPA and Regulation F largely do not bind it 10 11. It adopts their conduct rules as company policy, and state law (California's Rosenthal Act, Massachusetts 940 CMR 7.00), the TCPA and the CFPB's ban on unfair, deceptive or abusive practices still apply. Under that policy, dues are discussed only with the borrower or a verified guarantor (C05). So when Kyle Mitchell picks up his brother Ryan's phone (P05), the call is a human answer and not an RPC.
Steelyard's definitions:
- RPC: the borrower or a guarantor, after
verify_identity returns verified=true. Someone who says they are the borrower but will not verify is a claimed right party; no dues can be discussed with them.
- PTP: a
ptps row. A verified PTP's stored amount_cents and date equal what the borrower confirmed on the reference transcript.
- Kept rate: kept PTPs per PTP, set by the kept check after the promise date.
Hazel works the 1-30 and 31-90 days past due (DPD) buckets. Report each rate by DPD bucket as a monitored slice: early and late buckets promise and pay at different rates, so a shift in campaign mix moves the pooled number while the agent stays the same.
Where does each eval layer sit in the funnel?
The outbound funnel runs dials, connects, human answers, RPCs, promises and payments, and every step has its own eval and its own denominator. A check on one step is blind to a failure at any other.
A Steelyard calling day (26 calling days a month) starts with 250,000 dials. 52% connect, so 130,000 calls are answered by something. 88% of those are a person: 114,400. 70% of the people are a right party: 80,080 RPCs. 38% of RPCs promise, about 30,430 PTPs, and 62% of promises are kept, about 18,870. That puts kept PTP per dial at about 7.5%, RPC per dial at 32.0% and PTP per connect at 23.4%.
Figure: Synthetic Steelyard data. One calling day from dial to kept promise, with each rate's denominator and the eval guarding each step. The top three steps decide who Hazel talks to; only the bottom three are the conversation itself.
| Step | Failure | Eval | Owner chapter |
|---|
| Dial -> connect | Outside the calling window, over the attempt cap, unsigned or withheld caller ID | Deterministic dialer checks (C01, C11, C16) | Compliance checks |
| Connect -> human | A machine taken for a person, or a person for a machine | Costed AMD confusion matrix | This page |
| Human -> RPC | Dues told to a relative; guarantor never verified | Party handling and the order check before any dues (C04, C05) | Compliance checks |
| RPC -> PTP | Amount misheard, cut off before the date, stuck on a clarification | Entity error rate, read-back order (C12), turn-taking and latency guardrails | Entity error rate, turn-taking |
| PTP -> record | Wrong, missing or duplicate row | CRM diff; verified-PTP match rate | This page |
| Record -> kept | Promise made, never paid | Kept check; kept PTP per dial against a holdout | This page |
Start each rate's denominator at the first step the thing under test controls. PTP per RPC measures the conversation. PTP per connect also measures AMD. Kept PTP per dial measures everything at once, which makes it the online business number and a poor release gate. In Steelyard's world each RPC also uses up the account's next seven days, so a conversation that ends without a correct promise has no cheap second try; layered metrics works through that calendar.
How do you evaluate answering machine detection?
Score AMD as a six-class problem: human, voicemail, carrier intercept, call-screening assistant, interactive voice response (IVR) and silence. Build a confusion matrix on labeled openings from your own carriers and price each error type: hanging up on a borrower and talking to a machine cost different things.
CAUTION
Vendor-reported. Treat it as a hypothesis to test on your own calls.
LiveKit reports a macro F1 of 97.0% for its AMD (micro F1 and accuracy 94.7%, human-class F1 95.7%), using its default Gemini 3.1 Flash-Lite plus Cartesia Ink-Whisper pairing on an undisclosed internal dataset (May 2026). The dataset's languages are not named, so the figure tells you nothing specific about Spanish or Spanglish openings 12. Twilio's documentation warns that AMD will not always return the right answer, with a two-second voicemail greeting read as a human as its example; its AnsweredBy values are human, machine_start, machine_end_beep, machine_end_silence, machine_end_other, fax and unknown 13. Its FAQ calls DetectMessageEnd close to 100% accurate in US destinations with default settings and warns that international accuracy may be lower because voicemail tones vary by country 14; it gives no method or sample for that claim. None of the AnsweredBy values is a call-screening assistant, a carrier intercept or an IVR, so map your vendor's labels onto the six classes before measuring anything.
The matrix below is a synthetic stand-in for gold-v7's 200 labeled openings under Steelyard's current AMD setting A. Rows are the true class and columns the predicted one:
| True \ predicted | Human | Voicemail | Screening | Intercept | IVR | Silence | Recall [Wilson 95%] |
|---|
| Human (80) | 77 | 1 | 0 | 0 | 0 | 2 | 0.96 [0.90, 0.99] |
| Voicemail (40) | 6 | 31 | 1 | 0 | 2 | 0 | 0.78 [0.62, 0.88] |
| Call-screening assistant (25) | 8 | 2 | 14 | 0 | 1 | 0 | 0.56 [0.37, 0.73] |
| Carrier intercept (20) | 1 | 2 | 0 | 14 | 0 | 3 | 0.70 [0.48, 0.85] |
| IVR (20) | 3 | 0 | 2 | 0 | 15 | 0 | 0.75 [0.53, 0.89] |
| Silence (15) | 2 | 1 | 0 | 0 | 0 | 12 | 0.80 [0.55, 0.93] |
One accuracy number hides the fact that the off-diagonal cells cost different amounts. Keep three error types apart:
| Error | Count (setting A) | Cost |
|---|
| Person taken for a machine (false machine) | 3 of 80 | The borrower is hung up on before the opener: at Steelyard's funnel rates, 0.70 expected RPCs, 0.266 expected promises and 0.165 expected kept promises lost each time. The dropped call still counts toward the 7-in-7 cap, so the redial spends a second attempt |
| Machine taken for a person (false human) | 20 of 120 | T07's 21 s of trunk time each and an inflated connect count; logged as a conversation, as in T07, it also blocks the account for 7 days. On a voicemail the opener is recorded instead of a limited-content message |
| Machine taken for another machine | 14 of 120 | A voicemail read as an IVR gets no message; an intercept read as a voicemail gets a message nobody will hear and an attempt counted against the cap |
The screening row is T07 eight times: 8 of 25 false humans, 32%, Wilson [17%, 52%]. And 25 openings cannot sign off on anything. Zero false humans in 25 still leaves an exact one-sided 95% upper bound of 11.3%. To show a rate below 5% you need 59 screening openings with no errors on that exact bound (73 on a two-sided 95% Wilson bound), or 202 on a two-sided Wilson bound if 4 of them (about 2%) are errors. A 200-opening matrix tells you which class to label next. The gate needs a stratum large enough to bound.
Screening prompts, intercept wording and voicemail tones vary by carrier and handset, so carrier stays a monitored slice on the matrix even where it is too thin to gate.
A worked example shows how the denominator misleads. The recipe projects one calling day under setting A and under a stricter setting B that wants more evidence before it says human. Agent, prompt and model are identical:
| Setting | Calls Hazel handled | Machines engaged | RPC | PTP | PTP per handled call | PTP per RPC | RPC per dial |
|---|
| A (current) | 112,710 | 2,600 | 77,077 | 29,289 | 26.0% | 38% | 30.8% |
| B (stricter human) | 108,420 | 1,170 | 75,075 | 28,528 | 26.3% | 38% | 30.0% |
Suppose the dashboard defines a connect as a call AMD handed to Hazel, the same count T07 inflated. Under B it shows PTP per connect up 0.3 points, and a team reading it would conclude B is the better agent. In fact B hangs up on more real borrowers before the opener: 2,002 fewer RPCs and 761 fewer promises a day, about 470 fewer kept promises, in exchange for 8.3 trunk hours. The denominator fell faster than the numerator.
PTP per RPC holds at 38%. That correctly says the conversation did not change, and it cannot see the borrowers B never reached; if the dropped people were harder calls, it would even go up. RPC per dial falls from 30.8% to 30.0%, and that move is real. Judge an AMD change on the costed matrix and RPC per dial, judge the agent on PTP per RPC, and judge neither on PTP per handled connect. B pays one lost RPC for every 15 trunk seconds saved, a poor price for a borrower conversation.
Do abandoned-call rules apply to a collections dialer?
Not directly. The US abandoned-call limits are telemarketing rules, and a call that only collects an existing debt does not fit the telemarketing definition on its face. A collections team can still borrow them as a benchmark for false machines, since a borrower AMD hangs up on is what those rules count as abandoned.
The FTC's Telemarketing Sales Rule treats a call as abandoned when a person answers and is not connected to a sales representative within two seconds of their completed greeting. Its safe harbor allows at most 3% of calls answered by a person to be abandoned, measured per campaign over each 30-day period, with unanswered calls left to ring at least 15 seconds or four rings and a recorded message carrying the seller's name and number 15. Telemarketing there means a campaign to induce the purchase of goods or services or a charitable contribution 16. The FCC's TCPA rules set a parallel 3% limit per campaign over 30 days for telemarketing calls 17. Steelyard reports an abandoned call rate per campaign per 30 days against that 3% as an internal benchmark, labeled as such.
The UK book faces a different regime. Ofcom's persistent misuse policy, dated 20 December 2016, says action against silent calls is likely to be its highest priority. In that statement Ofcom says the 3% abandoned call rate from its 2010 policy was a criterion for prioritizing enforcement, not a safe harbour, and abandoned calls below it are not lawful by default 18. Ofcom notes that AMD can create silent calls by mistaking a live person for a machine and hanging up, and that it may ask for evidence of AMD accuracy. For Steelyard Credit UK Ltd, the false-machine row of the matrix is that evidence. We found no Ofcom statement on whether a conversational AI agent counts as a "live agent" for the abandoned call rate, so Steelyard treats the question as open.
What is special about voicemail?
Anyone who checks the borrower's phone can play a voicemail, so Steelyard leaves only a limited-content message, as Regulation F defines it and adopted as policy: a business name that does not indicate debt collection, a request to call back, the name of a natural person to contact and a callback number 11.
Beyond those four items the definition allows only a short list of optional ones. No amount, product, DPD or "past due", no complaint script, no SMS.
Two messages left on P08's carrier voicemail:
compliant TTS: "Hello, this is Steelyard. Please call Dana Brooks at five two zero, five five five, zero one zero zero. Any representative can help. Thank you."
tools: mark_contact(voicemail) (limited-content message: passes C04; counts toward the C11 cap)
non-compliant TTS: "Hi Ryan, this is Hazel from Steelyard Credit. Your auto loan payment is past due. Please pay the four hundred twenty-five dollars as soon as possible." (C04 fail; not a limited-content message)
The compliant message carries the four required items and two optional ones (a salutation and "any representative can help"). It names Dana Brooks, the Customer Advocate, and not Hazel: the definition asks for a natural person, and whether an AI persona's name could ever qualify is a question for counsel. It names no borrower, because anyone who checks the phone hears it. The non-compliant one fails C04, which bans product, amount, DPD and "past due" before verify_identity returns verified, and nobody verifies on a voicemail. Every voicemail is a placed call that counts toward the 7-in-7 cap 2, so leaving one is never free.
Checks on every call AMD labels voicemail:
- Content, on the TTS input text: business name "Steelyard" with no collection wording, a callback request, Dana Brooks, the callback number; no amount, due date, product, DPD, "past due" or borrower name; nothing outside the optional list.
- Records, in the diff:
mark_contact(voicemail), counts_toward_cap=true, no sms_log row, no ptps row, rpc=false, conversation=false.
- Timing, on the agent channel of the recording: the message starts after the beep. P08's beep comes after 9 s, and a message started during the greeting loses its first half, which holds the business name and the person to call.
- Class, on the opening: a voicemail AMD called human gets the opener instead of the limited-content message, so it fails the flow even when no dues rule broke.
F07's steps differ from the India edition in one way: there the voicemail carries the grievance officer's details and an SMS follows. The gold-state shape (neutral message, nothing disclosed, mark_contact(voicemail)) is the same. The C04 check code lives in safety, privacy and compliance checks; this page owns which records a voicemail call must leave.
How do you know a promise was good?
Three tests, from fastest to slowest. At call time a promise is verified when the stored amount and date match what the borrower confirmed after an explicit yes. After the promise date plus a grace period it is kept if payments cover it. Months later, the program is good only if kept PTP per dial beats a holdout.
Settle the kept check's rules before anyone reads a kept rate:
- When it runs. At
ptp_date plus a grace period that covers the lag between payments.paid_on and posted_at. Run it early and slow postings read as broken promises. The grace length is a collections decision, versioned with the checker.
- What counts as kept. Payments with
paid_on between the promise and ptp_date that sum to at least amount_cents, on any channel (ACH, debit card, autopay or the link). A partial payment is broken; report the share paid next to the rate.
- Moved dates. A new date is a new promise. Mark the old one superseded, not broken, or one borrower fails twice.
- Payments with no promise. Sofia Ramirez (P02) paid before the call. That is the found branch of F02, not a kept PTP.
The agent controls the kept rate's denominator. Take fewer, safer promises and the kept rate climbs while dollars collected fall; book every "uh-huh" as a promise and PTP per RPC climbs while the kept rate falls, the pattern of the T01 week in layered metrics. The kept rate diagnoses; it is not the north star.
The business result needs a holdout, because payments move with paydays and holiday weeks, and a rate from the bot arm alone cannot separate Hazel from the calendar:
- Randomize accounts, not calls, so no borrower lands in both arms.
- Stratify by campaign and DPD bucket.
- Say what the holdout receives. A human-agent holdout makes kept PTP per dial comparable; a no-call holdout has no promises, so compare dollars paid per account.
- Fix the outcome window in advance, to the last
ptp_date plus grace, or promises that have not yet fallen due count as broken.
- Use Newcombe's interval for the difference between two independent arms. Arm sizes are worked in reliability statistics for voice agents.
The holdout takes weeks and tells you whether the program pays. The verified-PTP check runs tonight and tells you whether this release stores the promises borrowers actually made.
What gates would have caught T07 and T01?
T07: block any AMD or carrier change whose upper bound on false humans for screening assistants, or false machines for people, exceeds the signed threshold, and refuse conversation=true with no caller turn. T01: a verified-PTP match-rate gate on gold-v7 and the C12 order check on every live call.
gate: outbound-task
amd:
openings: gold-v7 openings, plus a growing call-screening stratum by carrier
block_if:
- wilson_upper(false_human | call_screening_assistant) > signed_threshold
- wilson_upper(false_machine | human) > signed_threshold
- mcnemar(candidate vs incumbent, same openings) shows more false machines
report_always: [six_class_matrix, cost_table, rpc_per_dial_projection, per_carrier_matrix,
abandoned_call_rate_per_campaign_30d]
task:
scenarios: gold-v7, 600 cards, nightly 240 at k=4
scorer: crm-diff (pinned by hash), spoken checks reported separately
block_if:
- verified_ptp_match_rate wilson_upper_of_misses > signed_threshold # per gated segment
- any unexpected write in F03, F04, F07, F08 # dispute, hardship, machines
- any call with conversation=true and no caller turn # T07's record error
online: C12 order check on 100% of calls; verified-PTP audit on the human-audited sample
The verified-PTP match rate (promises whose stored amount and date equal the confirmed values, over promises written) is one proportion per segment, so Wilson applies. A candidate against the incumbent on the same scenarios is a paired comparison: McNemar or a paired bootstrap.
T07 lands in the screening row, and 8 of 25 fails any reasonable threshold. T01 fails the diff on two fields and C12 on order, and C12 catches it on the live call. Online, add a monitor that needs no labels for T07's signature: a call labeled human with no caller reply to the opener, no mark_contact, and conversation=true.
What goes in the decision memo?
The memo sets the AMD operating point, priced in borrowers, trunk seconds and contact days rather than accuracy points.
Decision. Keep AMD setting A's global threshold and do not ship setting B. Add a call-screening class trained on US screening prompts and gate it on its own row. Set conversation only from a caller turn, never from the AMD label.
Context. T07: a call-screening assistant labeled human at confidence 0.71 heard the opener for 11 s, was logged as a conversation, and blocked the account for 7 days.
Options considered. (1) Setting B, stricter on human: 1,430 fewer machines engaged a day, 2,002 fewer RPCs. (2) Keep A unchanged: T07 recurs. (3) A plus a screening class, a labeled stratum and the conversation rule: chosen.
Evidence. Synthetic stand-in for gold-v7's 200 openings. A: false machine 3 of 80, false human 20 of 120, screening false human 8 of 25 (32%, Wilson [17%, 52%]). B: false machine 5 of 80, false human 9 of 120. B's price: one lost RPC per 15 trunk seconds.
Owner and cadence. The telephony owner runs AMD; the eval platform staff engineer owns the gate; the collections lead signs thresholds. Rerun on every AMD or carrier change and monthly on fresh openings.
Tripwires. RPC per dial falls while PTP per RPC holds; handled calls change with dials flat; a new screening prompt or intercept wording; any call with conversation=true and no caller turn; the abandoned call rate per campaign nears the 3% benchmark.
Dated facts. LiveKit AMD figures (May 2026); Twilio AMD docs (modified 19 August 2026) and FAQ (checked 2026-11-20); Regulation F 1006.14 and commentary, TSR and TCPA abandonment rules (read 2026-11-20); Ofcom policy of 20 December 2016.
What to do this week
Begin with the two measurements teams most often skip: a labeled AMD matrix on your own carriers, and a field-level diff of what each call left in the CRM. Both run on calls you already recorded.
- Label 100 openings per carrier in the six classes, screening assistants included, and build the matrix for the current AMD setting before anyone touches its threshold.
- Write must, may and must-not lists for your three busiest flows, and make any unlisted write fail.
- Check how your dialer sets
conversation and counts_toward_cap, and test both on an intercept and a screening call.
- Rename every promise metric so it carries its denominator, and put RPC per dial beside PTP per RPC.
- Check five of last week's promises by hand against their reference transcripts, then get the kept check's grace period, partial-payment and moved-date rules written down and signed.
NOTE
What will date on this page. LiveKit's AMD figures come from its May 2026 blog post. Twilio's AMD documentation was modified 19 August 2026 and its FAQ is undated (both checked 2026-11-20); AnsweredBy values and the US accuracy statement may change. Vapi's testing guide is undated (checked 2026-11-20). OpsDog's PTP and kept-rate conventions, Call Centre Helper's RPC definition, and our finding that no US statute, regulation or regulator guidance defines RPC, PTP or kept rate were checked 2026-11-20. Regulation F (1006.2, 1006.14 and its commentary), the FDCPA definitions, the TSR (16 CFR 310.2 and 310.4) and the TCPA abandonment rule (47 CFR 64.1200) were read 2026-11-20. Ofcom's persistent misuse policy is the 2016 statement; as of November 2026 we found no Ofcom statement on AI agents as live agents. The tau-Voice scoring description is from the March 2026 paper.