On 10 November a phone's call-screening assistant asked Hazel, Steelyard's collections agent, to say her name and why she was calling. Answering machine detection (AMD) labeled the assistant a human, Hazel played her opener to it for 11 seconds, and the dialer logged the attempt as a human connect and as a conversation, which blocked the account for a week (T07). Four days later STT turned Ryan Mitchell's "two fifty" into "two fifteen", and Hazel wrote a $215.00 promise before any read-back (T01). The dashboard counted it as a captured PTP. Both calls look fine if you score what the agent said. Both fail if you score what the CRM holds afterwards. This recipe builds that scorer.
The script and synthetic data for this recipe are also in the ai-evals-course repository on GitHub, in the folder voice-ai/labs/collections-task-checker/. That copy also carries the India edition's block, which this page leaves out.
What you build
One standard-library script that scores 120 synthetic calls on the CRM, not on the transcript. Three of the calls are T01, T02 and T07 from the course's incident log; the other 117 are seeded, with faults injected into 15% of them.
| Function | What it does | What it prints |
|---|
gold_spec(flow, before, ctx) | The gold end state for flows F01-F10, built from the CRM before the call | (used by the checker) |
diff(before, after) | Rows added, rows removed and fields changed, per table, by primary key | (used by the checker) |
check_call(call) | Compares the diff with the gold state and checks the required tool order | One failure line per mismatch, such as ptps.amount_cents expected 25000 got 21500 |
sample_funnel(calls), daily_funnel() | RPC, PTP per RPC, verified PTP, kept rate, each with its denominator | The 120-call funnel and Steelyard's daily funnel |
amd_matrices(), amd_costs(m) | A six-class AMD confusion matrix on 200 labeled openings for two classifier settings | Per-class recall with Wilson intervals, errors in their own units |
amd_daily(m) | Projects each AMD setting onto a day of Steelyard calls | PTP per connect against PTP per RPC, trunk hours, a break-even |
The scoring idea is tau-bench's: each task passes a rule-based check that the final database matches the annotated goal state 1. tau-Voice keeps that database comparison for voice agents but hands the agent's spoken communications to an LLM evaluator, so its task score is not fully deterministic 2. This recipe makes the same split. The CRM diff is deterministic code. Whether Hazel said the right amount or named Dana Brooks, the Customer Advocate, is a spoken check, and it lives elsewhere (the compliance checks recipe and the entity error rate recipe).
Step 1: Create a fresh environment
There is nothing to install. A venv still keeps the run honest: if it works here, it does not depend on anything already on your machine.
python3.11 -m venv .venv
Step 2: Save the script
Save this as collections-task-checker.py. It starts with the same Steelyard constants block as the other voice recipes, so the capstone can tie them to one dataset name, and an --edition flag that defaults to global. The edition supplies the CRM field names (amount_cents, confirmation_no), the disclosure SMS tool, the incidents, the six AMD class names and the seed of the 117 seeded calls. The diff, the checker, the funnel arithmetic and the AMD draws are shared. The main claims the steps below make (the incident diff lines, zero missed faults and false alarms, the daily funnel, the lowest-recall AMD class, the denominator effect in section [5]) are checked by assert before the script prints that it passed.
"""Check task success against CRM state (aievals.co cookbook recipe).
Scores outbound collections calls on the CRM, not on the agent's words:
1. a gold end state per flow (F01-F10), built from the CRM snapshot before the call
2. a checker that diffs the before and after snapshots against that gold state
and checks the required tool order
3. funnel metrics with their denominators (RPC, PTP per RPC, verified PTP, kept)
4. a six-class AMD confusion matrix with per-class recall and error costs,
and PTP per connect vs PTP per RPC for two AMD settings
Standard library only. All data is synthetic and generated here from a fixed seed.
Run:
python collections-task-checker.py
python collections-task-checker.py --edition india # full file only
--edition picks the world (default global). The checker, the funnel arithmetic and
the AMD matrix are the same in both; the field names, flows' data, incidents and the
seed of the 120 synthetic calls come from the edition.
"""
from __future__ import annotations
import argparse
import math
import random
from collections import Counter
def western(n: float) -> str:
"""Western digit grouping: 130000 -> 130,000."""
return f"{int(round(n)):,}"
# --- GLOBAL constants (WORLD-GLOBAL world-global-v1.1) ---
GLOBAL = {
"lender_note": "Synthetic data.",
"dataset": "steelyard-calls (synthetic, seed 20261121)",
"seed": 20261121,
"products": {"AUTO": "auto loan", "PIL": "personal installment, digital",
"POS": "retail point-of-sale", "SBL": "small business, guarantor",
"SDL": "credit-builder and small-dollar"},
"calling_window_local": {"default": ("08:00", "21:00"), "SDL": ("09:00", "20:00")},
"campaigns": { # (book, silence budget ms, amount-asked / date-asked budget ms)
"AUTO-early-nov": ("AUTO", 400, 600),
"PIL-digital-nov": ("PIL", 400, 600),
"POS-short-nov": ("POS", 400, 600),
"SBL-guar-nov": ("SBL", 450, 650),
"SDL-biwk-nov": ("SDL", 600, 800),
},
"languages": {"English": 0.64, "Spanish-dominant": 0.18, "Spanglish": 0.14, "other": 0.04},
"customer_advocate": {"name": "Ms Dana Brooks", "email": "advocate@steelyard.example",
"phone": "(520) 555-0199",
"address": "1200 Example Avenue, Tucson, AZ 85701"},
"media": {"sample_rate_hz": 8000, "codec": "G.711 mu-law", "frame_ms": 20, "frame_bytes": 160},
"money_unit": "integer cents",
"usd_1500_forms": ["fifteen hundred", "one thousand five hundred", "1,500", "1500",
"$1,500.00", "mil quinientos", "mil quinientos dolares"], # 8th: "1.500" es-ES, R1
"usd_1500_cents": 150000,
# R6 task-checker data: CRM field names (WG5, WG6), flows (WG9), AMD (WG4, WG15), incidents (WG16)
"fields": {"amount": "amount_cents", "payment_ref": "confirmation_no"},
"sms_tool": "send_disclosure_sms", "sms_template": "disclosure_details",
"ptp_channel": "ach", "tz": "-07:00", "world_ref": "WG1",
"fmt_count": western,
"voicemail_sms": False, # WG9: the F07 voicemail is a limited-content message, no SMS
"attempt_flags": True, # call_attempts.counts_toward_cap and .conversation (WG6)
"flow_product": {"F06": "SBL", "F02": "POS", "F09": "PIL", "F03": "PIL"}, "default_product": "AUTO",
"ptp_amounts": [25000, 35000, 45000, 55000, 65000], # $250-$650 in cents
"misheard_short": 3500, # "fifty" heard as "fifteen": $35.00 short (T01)
"paid_amount": 13000, "pushed_amount": 42500, "hardship_amount": 10000,
"dnc_phone": "+152055501{:02d}", "dnc_mod": 100, # reserved 555-01xx numbers
"faults": {"F07": ["sms_on_voicemail"]},
"amd_classes": ["human", "voicemail", "call_screening", "carrier_intercept", "ivr", "silence"],
"amd_short": {"human": "human", "voicemail": "vmail", "call_screening": "screen",
"carrier_intercept": "intcpt", "ivr": "ivr", "silence": "silence"},
"t07_class": "call_screening", "f08_machines": ["call_screening", "carrier_intercept", "ivr"],
"not_counted": ["carrier_intercept"], # out of service: not a placed call for the 7-in-7 cap
"incidents": {
"T01": {"account": "A-55120931", "contact": "K-10931", "product": "AUTO",
"attempt_id": "SY-2026-11-14-0871234", "verified_at": "2026-11-14T18:41:40-07:00",
"stored": 21500, "confirmed": 25000, "date": "2026-11-20"}, # "two fifty" heard as "two fifteen"
"T02": {"account": "A-55188402", "contact": "K-18402", "product": "POS",
"payment_ref": "431187650092", "paid": 13000, "paid_on": "2026-11-03",
"verified_at": "2026-11-16T11:20:05-07:00"}, # paid by debit card in the app
"T07": {"account": "A-55170712", "contact": "K-17712", "product": "AUTO",
"true_amd": "call_screening", "conversation": True}, # logged as a conversation
},
}
GLOBAL["gold_openings"] = dict(zip(GLOBAL["amd_classes"], [80, 40, 25, 20, 20, 15]))
# Mock classifier: P(predicted | true) for two settings. A is tuned on voicemail
# greetings (the T07 setting); B needs more evidence before it says human. Synthetic.
GLOBAL["amd_settings"] = {
"A (current)": {
"human": [0.95, 0.02, 0.01, 0.00, 0.01, 0.01],
"voicemail": [0.07, 0.87, 0.02, 0.00, 0.02, 0.02],
"call_screening": [0.36, 0.08, 0.50, 0.00, 0.06, 0.00],
"carrier_intercept": [0.18, 0.04, 0.00, 0.70, 0.00, 0.08],
"ivr": [0.12, 0.02, 0.06, 0.00, 0.80, 0.00],
"silence": [0.08, 0.04, 0.00, 0.00, 0.00, 0.88],
},
"B (stricter human)": {
"human": [0.90, 0.04, 0.02, 0.00, 0.02, 0.02],
"voicemail": [0.03, 0.91, 0.02, 0.00, 0.02, 0.02],
"call_screening": [0.10, 0.08, 0.74, 0.00, 0.08, 0.00],
"carrier_intercept": [0.06, 0.04, 0.00, 0.82, 0.00, 0.08],
"ivr": [0.05, 0.02, 0.06, 0.00, 0.87, 0.00],
"silence": [0.04, 0.04, 0.00, 0.00, 0.00, 0.92],
},
}
# Frozen results of the 120 seeded calls for this edition: (caught, task success, PTP rows, verified PTP)
GLOBAL["frozen"] = {"t01": ["ptps.amount_cents expected 25000 got 21500",
"ptps.readback_confirmed expected true got false"],
"t07": "call_attempts.amd_label expected call_screening got human",
"sample": (21, 99, 18, 11)}
# --- end GLOBAL constants ---
EDITIONS = {"global": GLOBAL}
# The India edition's constants block goes here in the full file; it adds
# EDITIONS["india"]. Without it, --edition accepts only "global".
WORLD: dict = GLOBAL # set from --edition in main()
# Shared by both editions (WG1 = W1 volumes; the gold-v7 openings draw is shared too,
# so the AMD matrix and every line computed from it are identical across editions) [W]
FUNNEL = {"dials_per_day": 250000, "connect_rate": 0.52, "human_share": 0.88,
"rpc_share": 0.70, "ptp_per_rpc": 0.38, "kept_per_ptp": 0.62}
T07_TRUNK_S = 21 # trunk seconds T07 spent talking to a machine
AMD_SEED = 20261126 # one draw per labeled opening, the same in both editions
FAULTS = {"F01": ["amount_misheard", "write_lost", "no_link"], "F02": ["skipped_status_check"],
"F03": ["pushed_past_dispute"], "F04": ["ptp_after_hardship"], "F05": ["third_party_as_rpc"],
"F06": ["dues_before_verify"], "F08": ["amd_false_human"],
"F09": ["callback_outside_window"], "F10": ["no_sms"]} # F07 comes from the edition
PK = {"accounts": "account_id", "contacts": "contact_id", "payments": "payment_id",
"call_attempts": "attempt_id", "ptps": "ptp_id", "callbacks": "callback_id",
"grievances": "grievance_id", "hardship": "hardship_id", "sms_log": "sms_id",
"dnc": "phone_e164"}
ANY = object() # gold value meaning "any non-null value"
def fmt(v) -> str:
if v is True:
return "true"
if v is False:
return "false"
if v is None:
return "null"
return str(v)
# ---------- the CRM diff ----------
def diff(before: dict, after: dict) -> dict:
"""Rows added, rows removed and fields changed, per table, keyed by primary key."""
out = {"added": {}, "removed": {}, "changed": {}}
for table, pk in PK.items():
b = {r[pk]: r for r in before.get(table, [])}
a = {r[pk]: r for r in after.get(table, [])}
added = [a[k] for k in sorted(a) if k not in b]
removed = [b[k] for k in sorted(b) if k not in a]
if added:
out["added"][table] = added
if removed:
out["removed"][table] = removed
for k in sorted(set(a) & set(b)):
for field in sorted(set(a[k]) | set(b[k])):
if a[k].get(field) != b[k].get(field):
out["changed"][(table, k, field)] = (b[k].get(field), a[k].get(field))
return out
# ---------- gold end states per flow (WG9) ----------
def gold_attempt(amd: str, rpc: bool, disp: str) -> dict:
"""The call_attempts row a flow must write. Where the edition records them, every
placed call counts toward the 7-in-7 cap except an out-of-service intercept, and
only a right-party call is a conversation (which starts a 7-day no-call period)."""
row = {"amd_label": amd, "rpc": rpc, "disposition": disp}
if WORLD["attempt_flags"]:
row["counts_toward_cap"] = amd not in WORLD["not_counted"]
row["conversation"] = rpc
return row
def gold_spec(flow: str, before: dict, ctx: dict) -> dict:
"""Flow spec, plus: any flow that verifies identity must stamp contacts.verified_at."""
spec = flow_spec(flow, before, ctx)
if "verify_identity" in spec["tools"]:
spec["set"][("contacts", ctx["contact_id"], "verified_at")] = ANY
return spec
def flow_spec(flow: str, before: dict, ctx: dict) -> dict:
"""Required adds, required field changes, optional tables and required tool order.
The gold state is a function of the CRM before the call: F02's branch depends on
whether the payment the borrower claims is already in `payments`.
"""
acc, amt, sms_tool = ctx["account_id"], WORLD["fields"]["amount"], WORLD["sms_tool"]
note = {"template": WORLD["sms_template"]}
attempt = gold_attempt
if flow == "F01":
return {"add": {"call_attempts": [attempt("human", True, "ptp")],
"ptps": [{amt: ctx["confirmed_amount"],
"ptp_date": ctx["confirmed_date"], "readback_confirmed": True}],
"sms_log": [{"template": "payment_link"}, note]},
"set": {}, "optional": set(),
"tools": ["lookup_account", "verify_identity", "get_dues", "record_promise_to_pay",
"send_payment_link", sms_tool, "end_call"]}
if flow == "F02":
found = any(p["paid_on"] >= ctx["since"] for p in before["payments"])
tools = ["lookup_account", "verify_identity", "get_dues", "check_payment_status"]
if found:
return {"add": {"call_attempts": [attempt("human", True, "paid")], "sms_log": [note]},
"set": {}, "optional": set(),
"tools": tools + [sms_tool, "end_call"]}
return {"add": {"call_attempts": [attempt("human", True, "dispute")], "sms_log": [note],
"grievances": [{"category": "already_paid"}]},
"set": {("accounts", acc, "recovery_hold"): True}, "optional": set(),
"tools": tools + ["log_dispute", sms_tool, "end_call"]}
if flow == "F03":
return {"add": {"call_attempts": [attempt("human", True, "dispute")], "sms_log": [note],
"grievances": [{"category": "amount_wrong"}]},
"set": {("accounts", acc, "recovery_hold"): True}, "optional": set(),
"tools": ["lookup_account", "verify_identity", "get_dues", "log_dispute",
sms_tool, "end_call"]}
if flow == "F04":
return {"add": {"call_attempts": [attempt("human", True, "hardship")], "sms_log": [note],
"hardship": [{"reason": "job_loss"}]},
"set": {("accounts", acc, "hardship_flag"): True}, "optional": set(),
"tools": ["lookup_account", "verify_identity", "flag_hardship",
sms_tool, "transfer_to_human"]}
if flow == "F05":
return {"add": {"call_attempts": [attempt("human", False, "third_party")]},
"set": {}, "optional": {"callbacks"},
"tools": ["lookup_account", "mark_contact"]}
if flow == "F06":
return {"add": {"call_attempts": [attempt("human", True, "guarantor")], "sms_log": [note]},
"set": {}, "optional": {"callbacks"},
"tools": ["lookup_account", "verify_identity", "get_dues", sms_tool]}
if flow == "F07":
if not WORLD["voicemail_sms"]: # limited-content message: no SMS, no disclosure
return {"add": {"call_attempts": [attempt("voicemail", False, "voicemail")]},
"set": {}, "optional": set(), "tools": ["mark_contact"]}
return {"add": {"call_attempts": [attempt("voicemail", False, "voicemail")], "sms_log": [note]},
"set": {}, "optional": set(), "tools": [sms_tool, "mark_contact"]}
if flow == "F08":
return {"add": {"call_attempts": [attempt(ctx["true_amd"], False, "ivr")]},
"set": {}, "optional": set(), "tools": ["mark_contact"]}
if flow == "F09":
spec = {"add": {"call_attempts": [attempt("human", True, "callback")],
"callbacks": [{"requested_by": "borrower", "within_window": True}]},
"set": {}, "optional": set(), "tools": ["lookup_account", "schedule_callback"]}
if ctx["verified"]:
spec["add"]["sms_log"] = [note]
spec["tools"] = ["lookup_account", "verify_identity", "schedule_callback", sms_tool]
return spec
if flow == "F10":
return {"add": {"call_attempts": [attempt("human", True, "complaint")], "sms_log": [note]},
"set": {}, "optional": {"dnc"},
"tools": ["lookup_account", "verify_identity", sms_tool]}
raise ValueError(flow)
# ---------- the checker ----------
def check_tools(required: list[str], log: list[dict]) -> list[str]:
"""Required tools must appear in this order (other calls may sit between them)."""
names = [t["name"] for t in log]
lines, pos, prev = [], 0, "call start"
for name in required:
try:
pos = names.index(name, pos) + 1
prev = name
except ValueError:
lines.append(f"missing tool {name} (required after {prev})")
return lines
def check_call(call: dict) -> list[str]:
"""Return failure lines; an empty list is a task success."""
spec = gold_spec(call["flow"], call["before"], call["ctx"])
d = diff(call["before"], call["after"])
lines = []
for table, want_rows in spec["add"].items():
got_rows = d["added"].get(table, [])
if len(got_rows) != len(want_rows):
lines.append(f"{table} rows added expected {len(want_rows)} got {len(got_rows)}")
for want, got in zip(want_rows, got_rows):
for field, v in want.items():
if got.get(field) != v:
lines.append(f"{table}.{field} expected {fmt(v)} got {fmt(got.get(field))}")
for table, rows in d["added"].items():
if table not in spec["add"] and table not in spec["optional"]:
lines.append(f"{table} rows added expected 0 got {len(rows)}")
for table, rows in d["removed"].items():
lines.append(f"{table} rows removed expected 0 got {len(rows)}")
for (table, pk, field), want in spec["set"].items():
old, new = d["changed"].get((table, pk, field), (None, None))
if want is ANY:
if new is None:
lines.append(f"{table}.{field} expected set got null")
elif new != want:
lines.append(f"{table}.{field} expected {fmt(want)} got {fmt(new if (table, pk, field) in d['changed'] else old)}")
for key, (old, new) in d["changed"].items():
if key not in spec["set"]:
lines.append(f"{key[0]}.{key[2]} expected unchanged got {fmt(new)}")
return lines + check_tools(spec["tools"], call["tools"])
# ---------- synthetic calls with CRM snapshots ----------
def base_crm(acc: str, con: str, product: str, role: str = "borrower") -> dict:
return {
"accounts": [{"account_id": acc, "product_code": product, "recovery_hold": False,
"hardship_flag": False, "dnc_bot": False}],
"contacts": [{"contact_id": con, "account_id": acc, "role": role, "verified_at": None}],
"payments": [], "call_attempts": [], "ptps": [], "callbacks": [], "grievances": [],
"hardship": [], "sms_log": [], "dnc": [],
}
def copy_crm(crm: dict) -> dict:
return {t: [dict(r) for r in rows] for t, rows in crm.items()}
def attempt_row(att: str, acc: str, amd: str, rpc: bool, disp: str,
conversation: bool | None = None) -> dict:
"""What the dialer logged. `conversation` defaults to the RPC flag; T07 overrides it."""
row = {"attempt_id": att, "account_id": acc, "amd_label": amd, "rpc": rpc, "disposition": disp}
if WORLD["attempt_flags"]:
row["counts_toward_cap"] = amd not in WORLD["not_counted"]
row["conversation"] = rpc if conversation is None else conversation
return row
class Ids:
def __init__(self):
self.n = 0
def next(self, prefix: str) -> str:
self.n += 1
return f"{prefix}-{self.n:05d}"
def simulate(flow: str, i: int, rng: random.Random, ids: Ids, fault: str | None) -> dict:
"""What the agent did: a tool log and the CRM after the call (written independently of gold_spec)."""
W, amt, ref = WORLD, WORLD["fields"]["amount"], WORLD["fields"]["payment_ref"]
sms_tool, note = W["sms_tool"], W["sms_template"]
acc, con, att = f"A-{55120000 + i}", f"K-{20000 + i}", f"AT-{i:05d}"
product = W["flow_product"].get(flow, W["default_product"])
before = base_crm(acc, con, product, "guarantor" if flow == "F06" else "borrower")
ctx = {"account_id": acc, "contact_id": con, "since": "2026-11-01", "verified": True}
if flow == "F02":
if rng.random() < 0.7: # P02 paid on 3 Nov; sometimes the payment is not posted yet
before["payments"].append({"payment_id": ids.next("PM"), "account_id": acc,
ref: str(rng.randrange(10**11, 10**12)),
amt: W["paid_amount"], "paid_on": "2026-11-03"})
after = copy_crm(before)
tools: list[dict] = []
T = lambda name, **kw: tools.append({"name": name, "result": "ok", **kw})
def attempt(amd, rpc, disp, conversation=None):
after["call_attempts"].append(attempt_row(att, acc, amd, rpc, disp, conversation))
def sms(template):
after["sms_log"].append({"sms_id": ids.next("SMS"), "account_id": acc, "template": template})
def ptp(amount, date, readback):
after["ptps"].append({"ptp_id": ids.next("PTP"), "account_id": acc, "attempt_id": att,
amt: amount, "ptp_date": date,
"channel": W["ptp_channel"], "readback_confirmed": readback})
def verify():
T("verify_identity")
for c in after["contacts"]:
c["verified_at"] = "2026-11-14T10:05:00" + W["tz"]
if flow in ("F01", "F02", "F03", "F04", "F06", "F09", "F10"):
T("lookup_account")
if flow == "F01":
amount = rng.choice(W["ptp_amounts"])
ctx["confirmed_amount"] = amount
ctx["confirmed_date"] = f"2026-11-{rng.randint(16, 28)}"
verify(); T("get_dues")
stored, readback = amount, True
if fault == "amount_misheard":
stored, readback = amount - W["misheard_short"], False
T("record_promise_to_pay", **{amt: stored})
if fault != "write_lost": # the tool returned ok; the row never reached the CRM
ptp(stored, ctx["confirmed_date"], readback)
if fault != "no_link":
T("send_payment_link"); sms("payment_link")
T(sms_tool); sms(note); T("end_call")
attempt("human", True, "ptp")
elif flow == "F02":
verify(); T("get_dues")
if fault == "skipped_status_check":
attempt("human", True, "no_ptp"); T("end_call")
else:
T("check_payment_status")
if before["payments"]:
attempt("human", True, "paid")
else:
T("log_dispute")
after["grievances"].append({"grievance_id": ids.next("G"), "account_id": acc,
"category": "already_paid", "status": "open"})
after["accounts"][0]["recovery_hold"] = True
attempt("human", True, "dispute")
T(sms_tool); sms(note); T("end_call")
elif flow == "F03":
verify(); T("get_dues")
if fault == "pushed_past_dispute":
T("record_promise_to_pay")
ptp(W["pushed_amount"], "2026-11-25", True)
attempt("human", True, "ptp")
else:
T("log_dispute")
after["grievances"].append({"grievance_id": ids.next("G"), "account_id": acc,
"category": "amount_wrong", "status": "open"})
after["accounts"][0]["recovery_hold"] = True
attempt("human", True, "dispute")
T(sms_tool); sms(note); T("end_call")
elif flow == "F04":
verify(); T("flag_hardship")
after["hardship"].append({"hardship_id": ids.next("H"), "account_id": acc, "reason": "job_loss"})
after["accounts"][0]["hardship_flag"] = True
if fault == "ptp_after_hardship":
T("record_promise_to_pay")
ptp(W["hardship_amount"], "2026-11-21", True)
T(sms_tool); sms(note); T("transfer_to_human")
attempt("human", True, "hardship")
elif flow == "F05":
T("lookup_account"); T("mark_contact")
attempt("human", fault == "third_party_as_rpc", "third_party")
if rng.random() < 0.5:
T("schedule_callback")
after["callbacks"].append({"callback_id": ids.next("CB"), "account_id": acc,
"requested_by": "agent", "within_window": True})
elif flow == "F06":
if fault != "dues_before_verify":
verify()
T("get_dues")
if fault == "dues_before_verify":
verify()
T(sms_tool); sms(note)
attempt("human", True, "guarantor")
elif flow == "F07":
sends = W["voicemail_sms"]
if fault == "no_sms":
sends = False
if fault == "sms_on_voicemail": # an SMS to whoever holds the phone
sends = True
if sends:
T(sms_tool); sms(note)
T("mark_contact"); attempt("voicemail", False, "voicemail")
elif flow == "F08":
ctx["true_amd"] = rng.choice(W["f08_machines"])
if fault == "amd_false_human": # a T07-like miss: opener played to a machine
attempt("human", False, "no_response", conversation=True); T("end_call")
else:
T("mark_contact"); attempt(ctx["true_amd"], False, "ivr")
elif flow == "F09":
ctx["verified"] = rng.random() < 0.6
if ctx["verified"]:
verify()
T("schedule_callback")
late = fault == "callback_outside_window"
after["callbacks"].append({"callback_id": ids.next("CB"), "account_id": acc,
"requested_by": "agent" if late else "borrower",
"within_window": not late})
if ctx["verified"]:
T(sms_tool); sms(note)
attempt("human", True, "callback")
elif flow == "F10":
verify()
if fault != "no_sms":
T(sms_tool); sms(note)
if rng.random() < 0.3:
T("set_do_not_call")
phone = W["dnc_phone"].format(i % W["dnc_mod"])
after["dnc"].append({"phone_e164": phone, "account_id": acc, "scope": "bot"})
attempt("human", True, "complaint")
return {"id": f"SYN-{i:03d}", "flow": flow, "fault": fault, "ctx": ctx,
"before": before, "after": after, "tools": tools}
def incident_calls() -> list[dict]:
"""T01, T02 and T07 from the edition's incident log as CRM snapshots and tool logs."""
W, amt, ref, out = WORLD, WORLD["fields"]["amount"], WORLD["fields"]["payment_ref"], []
# T01: amount misheard; PTP written before read-back
t = W["incidents"]["T01"]
before = base_crm(t["account"], t["contact"], t["product"])
after = copy_crm(before)
after["contacts"][0]["verified_at"] = t["verified_at"]
after["call_attempts"].append(attempt_row(t["attempt_id"], t["account"], "human", True, "ptp"))
after["ptps"].append({"ptp_id": "PTP-T01", "account_id": t["account"],
"attempt_id": t["attempt_id"], amt: t["stored"],
"ptp_date": t["date"], "channel": W["ptp_channel"], "readback_confirmed": False})
after["sms_log"] += [{"sms_id": "SMS-T01a", "account_id": t["account"], "template": "payment_link"},
{"sms_id": "SMS-T01b", "account_id": t["account"], "template": W["sms_template"]}]
tools = [{"name": n, "result": "ok"} for n in
["lookup_account", "verify_identity", "get_dues", "record_promise_to_pay",
"send_payment_link", W["sms_tool"], "end_call"]]
out.append({"id": "T01", "flow": "F01", "fault": "amount_misheard", "before": before, "after": after,
"tools": tools, "ctx": {"account_id": t["account"], "contact_id": t["contact"],
"confirmed_amount": t["confirmed"], "confirmed_date": t["date"]}})
# T02: barge-in ignored; no check_payment_status, no log_dispute, a PTP request instead
t = W["incidents"]["T02"]
before = base_crm(t["account"], t["contact"], t["product"])
before["payments"].append({"payment_id": "PM-T02", "account_id": t["account"], ref: t["payment_ref"],
amt: t["paid"], "paid_on": t["paid_on"]})
after = copy_crm(before)
after["contacts"][0]["verified_at"] = t["verified_at"]
after["call_attempts"].append(attempt_row("AT-T02", t["account"], "human", True, "no_ptp"))
tools = [{"name": n, "result": "ok"} for n in ["lookup_account", "verify_identity", "get_dues", "end_call"]]
out.append({"id": "T02", "flow": "F02", "fault": "skipped_status_check", "before": before,
"after": after, "tools": tools,
"ctx": {"account_id": t["account"], "contact_id": t["contact"], "since": "2026-11-01"}})
# T07: a machine labeled human; opener played; no mark_contact
t = W["incidents"]["T07"]
before = base_crm(t["account"], t["contact"], t["product"])
after = copy_crm(before)
after["call_attempts"].append(attempt_row("AT-T07", t["account"], "human", False, "no_response",
t["conversation"]))
out.append({"id": "T07", "flow": "F08", "fault": "amd_false_human", "before": before, "after": after,
"tools": [{"name": "end_call", "result": "ok"}],
"ctx": {"account_id": t["account"], "contact_id": t["contact"],
"true_amd": t["true_amd"]}})
return out
def build_calls(seed: int, n_total: int = 120, fault_rate: float = 0.15) -> list[dict]:
rng, ids = random.Random(seed), Ids()
faults = {**FAULTS, **WORLD["faults"]}
calls = incident_calls()
flows = [f"F{k:02d}" for k in range(1, 11)]
i = 0
while len(calls) < n_total:
flow = flows[i % 10]
fault = rng.choice(faults[flow]) if rng.random() < fault_rate else None
calls.append(simulate(flow, i, rng, ids, fault))
i += 1
return calls
# ---------- funnel metrics with denominators ----------
def verified_ptp(call: dict) -> bool:
"""Stored amount and date equal what the borrower confirmed, after an explicit yes."""
ctx, amt = call["ctx"], WORLD["fields"]["amount"]
rows = diff(call["before"], call["after"])["added"].get("ptps", [])
return any(r["readback_confirmed"] and r[amt] == ctx.get("confirmed_amount")
and r["ptp_date"] == ctx.get("confirmed_date") for r in rows)
def sample_funnel(calls: list[dict]) -> dict:
attempts = [r for c in calls for r in diff(c["before"], c["after"])["added"].get("call_attempts", [])]
ptp_calls = [c for c in calls if diff(c["before"], c["after"])["added"].get("ptps")]
return {"calls": len(calls), "amd_human": sum(a["amd_label"] == "human" for a in attempts),
"rpc": sum(a["rpc"] for a in attempts), "ptp": len(ptp_calls),
"verified_ptp": sum(verified_ptp(c) for c in ptp_calls),
"task_success": sum(not check_call(c) for c in calls)}
def daily_funnel(f: dict = FUNNEL) -> list[tuple[str, float, str]]:
connects = f["dials_per_day"] * f["connect_rate"]
human = connects * f["human_share"]
rpc = human * f["rpc_share"]
ptp = rpc * f["ptp_per_rpc"]
kept = ptp * f["kept_per_ptp"]
return [("dials", f["dials_per_day"], "-"), ("connects", connects, "52% of dials"),
("human answers", human, "88% of connects"), ("RPC", rpc, "70% of human answers"),
("PTP", ptp, "38% of RPC"), ("kept PTP", kept, "62% of PTP")]
# ---------- six-class AMD confusion matrix ----------
def classify(probs: list[float], u: float) -> str:
acc = 0.0
classes = WORLD["amd_classes"]
for cls, p in zip(classes, probs):
acc += p
if u < acc:
return cls
return classes[-1]
def amd_matrices() -> dict[str, dict[str, Counter]]:
"""One uniform draw per opening, shared by both settings, so the comparison is paired."""
classes, gold = WORLD["amd_classes"], WORLD["gold_openings"]
rng = random.Random(AMD_SEED)
openings = [cls for cls in classes for _ in range(gold[cls])]
draws = [rng.random() for _ in openings]
out = {}
for name, table in WORLD["amd_settings"].items():
m = {cls: Counter() for cls in classes}
for true, u in zip(openings, draws):
m[true][classify(table[true], u)] += 1
out[name] = m
return out
def wilson(k: int, n: int, z: float = 1.959964) -> tuple[float, float]:
p = k / n
den = 1 + z * z / n
mid = (p + z * z / (2 * n)) / den
half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / den
return mid - half, mid + half
def amd_costs(m: dict[str, Counter]) -> dict:
"""Errors in their own units. A false machine on a human loses an RPC 70% of the time (WG1);
a false human on a machine wastes T07's 21 s of trunk time and plays the opener to a machine."""
classes = WORLD["amd_classes"]
false_machine = sum(m["human"][c] for c in classes if c != "human")
false_human = sum(m[c]["human"] for c in classes if c != "human")
machine_swap = sum(m[t][p] for t in classes[1:] for p in classes[1:] if t != p)
return {"false_machine": false_machine, "false_human": false_human, "machine_swap": machine_swap,
"lost_rpc": false_machine * FUNNEL["rpc_share"], "trunk_s": false_human * T07_TRUNK_S}
def amd_daily(m: dict[str, Counter]) -> dict:
"""Project one setting onto a day: true mix of connects from WG1, machine mix from the gold set."""
classes, gold = WORLD["amd_classes"], WORLD["gold_openings"]
connects = FUNNEL["dials_per_day"] * FUNNEL["connect_rate"]
humans = connects * FUNNEL["human_share"]
n_mach = sum(gold[c] for c in classes[1:])
true_mix = {"human": humans}
for c in classes[1:]:
true_mix[c] = (connects - humans) * gold[c] / n_mach
rate = lambda t, p: m[t][p] / gold[t]
engaged_h = humans * rate("human", "human")
engaged_m = sum(true_mix[c] * rate(c, "human") for c in classes[1:])
rpc = engaged_h * FUNNEL["rpc_share"]
ptp = rpc * FUNNEL["ptp_per_rpc"]
return {"agent_connects": engaged_h + engaged_m, "machines_engaged": engaged_m, "rpc": rpc,
"ptp": ptp, "ptp_per_connect": ptp / (engaged_h + engaged_m), "ptp_per_rpc": ptp / rpc,
"trunk_hours": engaged_m * T07_TRUNK_S / 3600}
def main() -> None:
global WORLD
ap = argparse.ArgumentParser()
ap.add_argument("--edition", choices=sorted(EDITIONS), default="global",
help="which world's field names, incidents and seed to use (default global)")
args = ap.parse_args()
WORLD = EDITIONS[args.edition]
seed, group, frozen = WORLD["seed"], WORLD["fmt_count"], WORLD["frozen"]
classes, short = WORLD["amd_classes"], WORLD["amd_short"]
print(WORLD["lender_note"])
print(f"Edition: {args.edition}. Dataset: {WORLD['dataset']}")
# [1] incidents against their gold end states
print("\n[1] Incident calls against their gold end states")
incidents = {c["id"]: c for c in incident_calls()}
for cid in ("T01", "T02", "T07"):
lines = check_call(incidents[cid])
print(f" {cid} {incidents[cid]['flow']}: {'PASS' if not lines else 'FAIL'}")
for line in lines:
print(f" {line}")
t01, t02, t07 = (check_call(incidents[k]) for k in ("T01", "T02", "T07"))
assert t01 == frozen["t01"] and "ptps.ptp_date" not in " ".join(t01)
assert any(l.startswith("missing tool check_payment_status") for l in t02)
assert frozen["t07"] in t07
if WORLD["attempt_flags"]: # T07 was logged as a conversation: a week of contact lost
assert "call_attempts.conversation expected false got true" in t07
# [2] 120 calls
calls = build_calls(seed)
assert len(calls) == 120
print(f"\n[2] Task success on {len(calls)} synthetic calls (3 incidents + 117 seeded, 15% fault rate)")
print(" flow calls pass fail first failure line of failed calls (top 2)")
by_flow: dict[str, list] = {}
for c in calls:
by_flow.setdefault(c["flow"], []).append(c)
caught = missed = false_alarm = 0
for flow in sorted(by_flow):
cs = by_flow[flow]
fails = [(c, check_call(c)) for c in cs]
n_fail = sum(bool(l) for _, l in fails)
reasons = Counter(l[0] for _, l in fails if l)
top = "; ".join(f"{r} (x{k})" if k > 1 else r for r, k in reasons.most_common(2)) or "-"
print(f" {flow} {len(cs):5d} {len(cs) - n_fail:4d} {n_fail:4d} {top}")
for c, l in fails:
caught += bool(c["fault"]) and bool(l)
missed += bool(c["fault"]) and not l
false_alarm += (not c["fault"]) and bool(l)
print(f" injected faults caught {caught}, missed {missed}; clean calls flagged {false_alarm}")
assert missed == 0 and false_alarm == 0
lost = [c for c in calls if c["fault"] == "write_lost"]
if lost:
c = lost[0]
ok = [t for t in c["tools"] if t["name"] == "record_promise_to_pay"][0]["result"]
print(f" {c['id']}: record_promise_to_pay returned {ok!r}; checker says {check_call(c)[0]}")
assert lost, "seed should produce a write_lost call"
# [3] funnel
s = sample_funnel(calls)
print(f"\n[3] Funnel on the {s['calls']} calls (each denominator named)")
print(f" AMD said human {s['amd_human']:4d}")
print(f" right-party contacts (RPC) {s['rpc']:4d}")
print(f" PTP rows written {s['ptp']:4d} PTP per RPC {s['ptp'] / s['rpc']:.3f}")
print(f" verified PTP {s['verified_ptp']:4d} verified PTP per RPC "
f"{s['verified_ptp'] / s['rpc']:.3f}; per PTP {s['verified_ptp'] / s['ptp']:.3f}")
print(f" task success {s['task_success']:4d} per call {s['task_success'] / s['calls']:.3f}")
print(" kept PTP: not known at call time; checked at ptp_date plus grace (ptps.kept)")
assert s["verified_ptp"] < s["ptp"]
assert (caught, s["task_success"], s["ptp"], s["verified_ptp"]) == frozen["sample"]
print(f"\n Daily funnel from {WORLD['world_ref']} (26 calling days a month)")
rows = daily_funnel()
for name, n, den in rows:
print(f" {name:15s} {group(n):>9s} {den}")
vals = {name: n for name, n, _ in rows}
assert vals["connects"] == 130000 and round(vals["human answers"]) == 114400
assert round(vals["RPC"]) == 80080 and round(vals["PTP"]) == 30430 and round(vals["kept PTP"]) == 18867
print(f" PTP per connect {vals['PTP'] / vals['connects']:.4f}; PTP per RPC "
f"{vals['PTP'] / vals['RPC']:.2f}; kept PTP per dial {vals['kept PTP'] / vals['dials']:.4f}")
assert abs(vals["PTP"] / vals["connects"] - 0.234080) < 1e-9
assert abs(vals["kept PTP"] / vals["dials"] - 0.075467392) < 1e-9
# [4] AMD matrix (identical counts in both editions; only the class names differ)
mats = amd_matrices()
for name, m in mats.items():
print(f"\n[4] AMD setting {name}: 200 labeled openings, rows = true, columns = predicted")
print(" " + " " * 22 + "".join(f"{short[c]:>8s}" for c in classes) + " recall [Wilson 95%]")
recalls = {}
for t in classes:
n = WORLD["gold_openings"][t]
k = m[t][t]
lo, hi = wilson(k, n)
recalls[t] = k / n
print(f" {t:22s}" + "".join(f"{m[t][p]:8d}" for p in classes)
+ f" {k / n:.2f} [{lo:.2f}, {hi:.2f}]")
cost = amd_costs(m)
print(f" false machine on a human: {cost['false_machine']} -> {cost['lost_rpc']:.1f} expected lost RPCs")
print(f" false human on a machine: {cost['false_human']} -> {cost['trunk_s']} s of trunk time, "
f"{cost['false_human']} openers played to a machine")
print(f" machine taken for another machine class: {cost['machine_swap']}")
if name.startswith("A"):
low = min(recalls, key=recalls.get)
print(f" lowest recall: {low}")
assert low == WORLD["t07_class"]
assert sum(m[t][t] for t in classes) < 200
assert [m[c]["human"] for c in classes] == [77, 6, 8, 1, 3, 2] # frozen [C]
assert (cost["false_machine"], cost["false_human"], cost["machine_swap"]) == (3, 20, 14)
# [5] AMD settings and the conversation metric
print("\n[5] Daily projection: same agent, two AMD settings (machine mix taken from the gold set)")
print(" setting agent connects machines engaged RPC PTP PTP/connect PTP/RPC trunk h")
proj = {}
for name, m in mats.items():
d = amd_daily(m)
proj[name] = d
print(f" {name:20s} {group(d['agent_connects']):>15s} {group(d['machines_engaged']):>17s} "
f"{group(d['rpc']):>7s} {group(d['ptp']):>7s} {d['ptp_per_connect']:12.4f} "
f"{d['ptp_per_rpc']:8.2f} {d['trunk_hours']:8.1f}")
a, b = proj["A (current)"], proj["B (stricter human)"]
assert abs(a["ptp_per_rpc"] - b["ptp_per_rpc"]) < 1e-12
assert b["ptp_per_connect"] > a["ptp_per_connect"] and b["ptp"] < a["ptp"]
print(f" PTP per connect moves {a['ptp_per_connect']:.4f} -> {b['ptp_per_connect']:.4f} "
f"while PTP per RPC stays {a['ptp_per_rpc']:.2f}: the agent did not change, the denominator did")
d_rpc = a["rpc"] - b["rpc"]
d_trunk_s = (a["trunk_hours"] - b["trunk_hours"]) * 3600
print(f" B loses {group(d_rpc)} RPCs and {group(a['ptp'] - b['ptp'])} PTPs a day and saves "
f"{a['trunk_hours'] - b['trunk_hours']:.1f} trunk hours")
print(f" break-even: B pays only if one lost RPC is worth less than {d_trunk_s / d_rpc:.0f} trunk seconds")
assert (round(a["agent_connects"]), round(b["agent_connects"])) == (112710, 108420)
assert (round(d_rpc), round(a["ptp"] - b["ptp"]), round(d_trunk_s / d_rpc)) == (2002, 761, 15)
print("\nAll checks passed.")
if __name__ == "__main__":
main()
Step 3: Run it
.venv/bin/python collections-task-checker.py
Running it with Python 3.11 prints:
Synthetic data.
Edition: global. Dataset: steelyard-calls (synthetic, seed 20261121)
[1] Incident calls against their gold end states
T01 F01: FAIL
ptps.amount_cents expected 25000 got 21500
ptps.readback_confirmed expected true got false
T02 F02: FAIL
call_attempts.disposition expected paid got no_ptp
sms_log rows added expected 1 got 0
missing tool check_payment_status (required after get_dues)
missing tool send_disclosure_sms (required after get_dues)
T07 F08: FAIL
call_attempts.amd_label expected call_screening got human
call_attempts.disposition expected ivr got no_response
call_attempts.conversation expected false got true
missing tool mark_contact (required after call start)
[2] Task success on 120 synthetic calls (3 incidents + 117 seeded, 15% fault rate)
flow calls pass fail first failure line of failed calls (top 2)
F01 13 10 3 ptps.amount_cents expected 25000 got 21500; sms_log rows added expected 2 got 1
F02 13 9 4 call_attempts.disposition expected paid got no_ptp (x4)
F03 12 9 3 call_attempts.disposition expected dispute got ptp (x3)
F04 12 9 3 ptps rows added expected 0 got 1 (x3)
F05 12 10 2 call_attempts.rpc expected false got true (x2)
F06 12 12 0 -
F07 12 11 1 sms_log rows added expected 0 got 1
F08 12 11 1 call_attempts.amd_label expected call_screening got human
F09 11 8 3 callbacks.requested_by expected borrower got agent (x3)
F10 11 10 1 sms_log rows added expected 1 got 0
injected faults caught 21, missed 0; clean calls flagged 0
SYN-030: record_promise_to_pay returned 'ok'; checker says ptps rows added expected 1 got 0
[3] Funnel on the 120 calls (each denominator named)
AMD said human 97
right-party contacts (RPC) 86
PTP rows written 18 PTP per RPC 0.209
verified PTP 11 verified PTP per RPC 0.128; per PTP 0.611
task success 99 per call 0.825
kept PTP: not known at call time; checked at ptp_date plus grace (ptps.kept)
Daily funnel from WG1 (26 calling days a month)
dials 250,000 -
connects 130,000 52% of dials
human answers 114,400 88% of connects
RPC 80,080 70% of human answers
PTP 30,430 38% of RPC
kept PTP 18,867 62% of PTP
PTP per connect 0.2341; PTP per RPC 0.38; kept PTP per dial 0.0755
[4] AMD setting A (current): 200 labeled openings, rows = true, columns = predicted
human vmail screen intcpt ivr silence recall [Wilson 95%]
human 77 1 0 0 0 2 0.96 [0.90, 0.99]
voicemail 6 31 1 0 2 0 0.78 [0.62, 0.88]
call_screening 8 2 14 0 1 0 0.56 [0.37, 0.73]
carrier_intercept 1 2 0 14 0 3 0.70 [0.48, 0.85]
ivr 3 0 2 0 15 0 0.75 [0.53, 0.89]
silence 2 1 0 0 0 12 0.80 [0.55, 0.93]
false machine on a human: 3 -> 2.1 expected lost RPCs
false human on a machine: 20 -> 420 s of trunk time, 20 openers played to a machine
machine taken for another machine class: 14
lowest recall: call_screening
[4] AMD setting B (stricter human): 200 labeled openings, rows = true, columns = predicted
human vmail screen intcpt ivr silence recall [Wilson 95%]
human 75 2 1 0 0 2 0.94 [0.86, 0.97]
voicemail 3 34 1 0 2 0 0.85 [0.71, 0.93]
call_screening 3 1 20 0 1 0 0.80 [0.61, 0.91]
carrier_intercept 0 0 0 17 0 3 0.85 [0.64, 0.95]
ivr 2 0 1 0 17 0 0.85 [0.64, 0.95]
silence 1 1 0 0 0 13 0.87 [0.62, 0.96]
false machine on a human: 5 -> 3.5 expected lost RPCs
false human on a machine: 9 -> 189 s of trunk time, 9 openers played to a machine
machine taken for another machine class: 10
[5] Daily projection: same agent, two AMD settings (machine mix taken from the gold set)
setting agent connects machines engaged RPC PTP PTP/connect PTP/RPC trunk h
A (current) 112,710 2,600 77,077 29,289 0.2599 0.38 15.2
B (stricter human) 108,420 1,170 75,075 28,528 0.2631 0.38 6.8
PTP per connect moves 0.2599 -> 0.2631 while PTP per RPC stays 0.38: the agent did not change, the denominator did
B loses 2,002 RPCs and 761 PTPs a day and saves 8.3 trunk hours
break-even: B pays only if one lost RPC is worth less than 15 trunk seconds
All checks passed.
The run takes well under a second and writes no files.
Step 4: Read the incident diffs
T01 fails on exactly two lines: ptps.amount_cents expected 25000 got 21500 and ptps.readback_confirmed expected true got false. The date matched, the link and the disclosure SMS went out, every required tool ran in order. A tool-sequence check alone passes this call. So does a check that a PTP row exists. Only a field-level diff catches it: the stored amount against what Ryan said on the reference transcript, $35.00 short, and the read-back flag, which says why it got through. His "uh-huh, okay" after Hazel's statement was a backchannel, not a yes.
T02 fails four ways, and the third line is the cause: missing tool check_payment_status. Sofia Ramirez cut in with "ya lo pague" [I already paid it]; she had paid $130.00 by debit card on 3 November, and in this recipe's snapshot the payment is already in payments, so F02's gold state is the found branch: disposition paid, no hold, disclosure SMS sent. The gold state is a function of the CRM before the call. You cannot write F02's expected outcome without reading the payments table first, which is why gold_spec takes the before snapshot.
T07 fails on the AMD label, the disposition, the conversation flag and a missing mark_contact. The row says amd_label human, rpc false and conversation true. Every report that counts connects by AMD label counted this call as a human connect. The conversation flag costs more. Steelyard follows Regulation F's frequency limits as policy: no more than seven calls in seven days, and none for seven days after a telephone conversation about the debt 3. A placed call counts toward the seven whether or not anyone answers; busy and out-of-service calls do not 4. So a screening assistant logged as a conversation spends one attempt and shuts the account for a week, with nobody spoken to. That is why the global gold state records counts_toward_cap and conversation on every attempt, and why an out-of-service carrier intercept gets counts_toward_cap=false.
The write_lost line in section [2] of the output is the case teams miss most. record_promise_to_pay returned 'ok' and no row reached ptps. Vapi's testing guide says it plainly: "A tool reporting success does not independently prove that an external system changed" 5. Score the promise on the CRM row, never on the tool's return value or on Hazel saying "I've noted that".
Step 5: Read the 120-call table
The checker caught all 21 injected faults and flagged none of the clean calls. That is a sanity check, not evidence: the faults are the ones the script was written to inject. Your evidence is your own failure log. Take every task failure your reviewers found by listening, export its before and after snapshots, and confirm check_call fails it on the line you expect.
Three design choices do most of the work:
- Unexpected writes fail. Any table that gains rows and is not in the gold state fails, unless the flow lists it as optional (a neutral callback on F05, a bot-only DNC on F10). That is how F03's pushed-past-dispute PTP and F04's PTP after hardship show up:
ptps rows added expected 0 got 1. It is also how F07 catches an SMS sent after a voicemail: sms_log rows added expected 0 got 1. Steelyard's voicemail is a limited-content message, which may carry only a business name that does not indicate debt collection, a request to reply, a person to contact, a number and a short list of optional items 6. An SMS about the account goes to whoever holds the phone, so the F07 gold state has no sms_log row at all.
- Required tools are an ordered subsequence. Other calls may sit between them, but
verify_identity must come before get_dues. F06's guarantor call fails with missing tool get_dues (required after verify_identity) when dues were fetched first. This seed drew no F06 fault, so all 12 F06 calls pass; the order check still runs on each.
- Party matters. F05 fails twice, each time on
call_attempts.rpc expected false got true, and the second line on each of those calls is conversation expected false got true. When Kyle Mitchell picks up his brother Ryan's phone, the call is a human answer and not a right-party contact. Steelyard collects its own debts in its own name, so the FDCPA and Regulation F largely do not bind it 7 6. It adopts their conduct rules as company policy, and state law (California's Rosenthal Act, Massachusetts 940 CMR 7.00), the TCPA and the CFPB's ban on unfair, deceptive or abusive practices still apply. Under that policy, dues are discussed only with the borrower or a verified guarantor. An rpc flag set wrongly inflates every per-RPC metric downstream, and a wrong conversation flag blocks the real borrower for seven days.
Step 6: Name every denominator
On the 120 calls, 18 PTP rows were written and 11 were verified: the stored amount and date equal what the borrower confirmed, after an explicit yes. Verified PTP per RPC is 0.128; raw PTP per RPC is 0.209. The 120-call sample spreads calls evenly across ten flows, so neither number is a production rate. The gap between them is the point: 7 of these 18 promises would be counted as wins. One is T01 at the wrong amount; the other six are promises taken after a dispute or a hardship claim, which should never have been asked for.
PTP per RPC is a common convention, promises over right-party contacts, and the kept rate is promises paid over promises made 8 9. Shops differ on the RPC rate's denominator (dials or connected calls), and we found no US statute, regulation or regulator guidance that defines RPC, PTP or kept rate, so this course treats them as its own definitions. A PTP rate without its denominator is not a number you can compare.
The output prints three "per connect" figures, and they disagree because they are different fractions. Steelyard's daily funnel gives PTP per connect 0.2341 over all 130,000 answered calls. Section [5] gives 0.2599 and 0.2631 over calls AMD labeled human. And PTP per RPC is 0.38 in all of them. Write the denominator into the metric name on every dashboard: ptp_per_rpc, never ptp_rate.
Kept PTP cannot be scored at call time. It is a later CRM write (ptps.kept) at the promise date plus a grace period, and it is the check that would eventually have marked T01's promise broken.
Step 7: Read the AMD matrix
Under the current setting, call-screening assistants have the lowest recall: 14 of 25, 0.56 with a Wilson interval of [0.37, 0.73]. Eight of the 25 were labeled human, which is T07 eight times over. Voicemail recall is 0.78 and human recall is 0.96.
Two things about that matrix matter more than its cells. First, 25 openings give an interval 36 points wide. Use the matrix to rank failure classes and choose what to label next, not to sign off a release; a gate on call-screening recall needs a few hundred labeled screening prompts. Second, the off-diagonal cells have different prices, so the script keeps them apart:
- False machine on a human (3 under setting A): the borrower is dropped. At Steelyard's 70% RPC share of human answers, that is 2.1 expected lost RPCs, and the dropped call still counts toward the seven-call limit.
- False human on a machine (20): T07's 21 seconds of trunk time each, 420 seconds in all, and 20 openers played to a machine. If it is logged as a conversation, as T07 was, each one also blocks the account for seven days. With the v3.2 opener that is a disclosure risk, not a breach, because the opener names Steelyard and asks for the borrower by name but states no dues.
- Machine taken for another machine class (14): a voicemail read as an IVR leaves no message, and an intercept read as a voicemail gets a message nobody will hear and an attempt counted against the limit. Cheap, but not free.
Vendor accuracy figures do not answer this question for your calls.
CAUTION
Vendor-reported. Treat it as a hypothesis to test on your own calls.
LiveKit reports a macro F1 of 97.0% for its answering machine detection (micro F1 and accuracy 94.7%, human-class F1 95.7%) on an undisclosed internal dataset whose languages are not named (May 2026) 10. Twilio's AMD returns human, machine_start, machine_end_beep, machine_end_silence, machine_end_other, fax or unknown, and its documentation says AMD will not always return the right answer 11. None of those labels is a call-screening assistant, a carrier intercept or an IVR. Map your vendor's labels onto the six classes, then measure on your own labeled openings.
Step 8: Compare two AMD settings on the right metric
Setting B demands more evidence before it says human. On a projected day it engages 1,170 machines instead of 2,600 and saves 8.3 trunk hours. PTP per connect rises from 0.2599 to 0.2631, and a team watching that number would ship B as a conversation win.
The agent did not change. PTP per RPC is 0.38 under both settings, because the script holds conversation quality fixed: only the AMD changed. B also drops more real borrowers before the opener, so it loses 2,002 RPCs and 761 PTPs a day. The per-connect number went up because its denominator shrank faster than its numerator. This is why an AMD change is judged on PTP per RPC and on the costed matrix, never on PTP per connect.
The break-even line puts the trade in one sentence: B pays only if one lost RPC is worth less than 15 trunk seconds. Fifteen seconds of trunk time is a bad price for a borrower conversation, so the global threshold stays where it is. The fix is narrower: label more call-screening prompts from each carrier, add the class to the classifier's training data, let the call-screening row of the matrix carry the gate, and set conversation only from a caller turn, never from the AMD label. The machine mix behind the projection is the gold set's (40 voicemails, 25 call-screening assistants, 20 carrier intercepts, 20 IVRs and 15 silences per 120 machines); replace it with your own counts before you quote the break-even.
Step 9: Point it at your own CRM
Replace incident_calls() and build_calls() with exports from your call platform: the CRM rows for the account immediately before the dial and immediately after end_call, the tool log with results, and the scenario context (the amount and date confirmed on the reference transcript, the true AMD class for machine calls). Keep diff and check_call as they are. If your field names differ, change the fields entry in the constants block, not the checker.
Then write gold_spec with your operations lead, one flow at a time, and put it under code review like any other policy. Three rules carry over from Steelyard:
- Every flow lists what must be written, what may be written, and what must not change. Unlisted writes fail.
- A branch that depends on data (paid or not, verified or not) reads the before snapshot, not the transcript.
- Spoken checks never get folded into the diff. A call can write a perfect CRM row and still fail on what Hazel said.
The full file in the course repository also carries the India edition's constants block, which the copy above leaves out. Run it with --edition india for the India edition's data (its own seed, field names, incidents and AMD class names); the India chapter explains that output. The daily funnel, the AMD matrix counts and the section [5] projection are identical in both editions apart from class names and digit grouping; the 120-call lines differ because the seed does.
What to do next
Run the checker on the last week of real calls where reviewers marked a task failure, and fix gold_spec until it fails each one on the right line. The task success chapter covers gold end states, the funnel and AMD testing in full, and the tau-bench recipe shows final-state scoring on a public benchmark. For the spoken half of each call, the compliance checks recipe turns Steelyard's policies into tests.