The point of this recipe is to leave you with a runnable, calibrated, binary judge wired to a fifty-trace labeled set in less time than it takes to argue about which platform to buy. It uses the Anthropic Python SDK and writes plain Python; you can swap the model client for OpenAI or any provider with a messages API without changing the shape 1. Every step here mirrors the first two rungs of Hamel's three-level ladder: code-graded assertions plus an LLM-graded binary 2.
The thirty minutes break down as: five minutes to scope the feature, ten minutes to label fifty traces in a CSV, ten minutes to write and run the judge prompt, five minutes to read the confusion matrix and the disagreements.
What you build
A single Python module that:
- Reads fifty labeled traces from a CSV.
- Runs a binary judge (PASS or FAIL with a one-sentence critique) on each.
- Reports true positive rate, true negative rate, and the agreement number.
- Writes a JSON file of disagreements you can paste back into the labeling step.
Step 1: scope to one feature, one rubric
Pick one user-visible behavior with a clear correctness question. Not "is the bot helpful." Yes "does the bot honor the dietary restriction stated three turns ago." Start with a binary pass/fail question rather than a 1-to-5 scale; Hamel's judge guide makes the same call 3. Keep the question falsifiable in one sentence, or the rubric will drift as you label: Shankar et al. call this criteria drift, because grading outputs changes what you think the criteria are 4.
For this walkthrough we use a meal-planning assistant. The rubric is one sentence: Does the response respect every dietary restriction the user stated earlier in the conversation?
Step 2: build a fifty-trace labeled set
If you have production traces, sample fifty across the dimensions you care about (input length, presence of restriction, multi-turn). If you do not, seed them by generating a small set with the model itself, then hand-label.
Save them as traces.csv with four columns: id, query, response, label. The label column is PASS or FAIL filled in by you (or one domain expert; ideally not the engineer writing the prompt).
id,query,response,label
1,"I am vegan. Suggest a quick dinner.","Stir-fried tofu with broccoli, garlic, ginger over jasmine rice.",PASS
2,"I keep kosher and avoid dairy. Suggest a quick dinner.","Roast salmon with lemon and a side of buttery mashed potatoes.",FAIL
3,"I have a peanut allergy. Suggest a snack.","Apple slices with almond butter.",PASS
...
Keep enough of each class to measure it. TPR and TNR are conditional rates, so oversampling failures does not bias them; it only makes the failure-side number less noisy. A set matched to a ten percent failure rate holds about five failures in fifty traces, and one misjudged trace then moves TNR by twenty points. This demo uses about ten failures and forty passes. Past the starter set, aim for 30 to 50 of each class in both dev and test 5. Agreement does depend on prevalence, which is one more reason not to headline it.
Step 3: write the judge
The shape is the four-part judge prompt: task framing, rubric, response under review, output format. We turn off up-front thinking, request JSON output, and ask for a one-sentence critique alongside the label so we can read disagreements later. We do not set temperature: Claude Sonnet 5.5 returns a 400 for a non-default value, and the current anthropic Python SDK has no temperature argument on messages.create, so passing one raises a TypeError 6. The judge is therefore not strictly deterministic. Run the labeled set twice and count flipped labels before you trust a one-point change.
"""Binary judge for the dietary-restriction rubric.
Run:
pip install anthropic pandas scikit-learn
export ANTHROPIC_API_KEY=...
python judge.py
"""
from __future__ import annotations
import json
import os
from dataclasses import dataclass
from typing import Literal
import pandas as pd
from anthropic import Anthropic
from sklearn.metrics import confusion_matrix
CLIENT = Anthropic() # reads ANTHROPIC_API_KEY
MODEL = "claude-sonnet-5-5"
RUBRIC = (
"Does the response respect every dietary restriction the user stated "
"earlier in the conversation? Restrictions include allergies, kosher, "
"halal, vegetarian, vegan, and keto. The response FAILS if it suggests "
"any ingredient that violates a stated restriction."
)
JUDGE_PROMPT = """You are grading a meal-planning assistant against one rubric.
RUBRIC:
{rubric}
USER QUERY:
{query}
ASSISTANT RESPONSE:
{response}
Reply with a JSON object: {{"label": "PASS" or "FAIL", "critique": "..."}}.
The critique must be one sentence. If the response violates the rubric, the
critique must name the specific ingredient or behavior that violates it.
"""
@dataclass
class Score:
label: Literal["PASS", "FAIL"]
critique: str
def judge(query: str, response: str) -> Score:
msg = CLIENT.messages.create(
model=MODEL,
max_tokens=200,
# Sonnet 5.5 thinks by default and rejects a non-default temperature.
# "between_tools" turns up-front thinking off, so the reply is plain text.
thinking={"type": "between_tools"},
messages=[
{
"role": "user",
"content": JUDGE_PROMPT.format(
rubric=RUBRIC, query=query, response=response
),
}
],
)
text = next(b.text for b in msg.content if b.type == "text").strip()
# Strip leading code fences if the model wraps the JSON.
if text.startswith("```"):
text = text.split("```", 2)[1]
if text.startswith("json"):
text = text[len("json") :].strip()
data = json.loads(text)
label = data["label"].upper()
if label not in ("PASS", "FAIL"):
raise ValueError(f"unexpected label: {label}")
return Score(label=label, critique=data["critique"])
def main() -> None:
df = pd.read_csv("traces.csv")
df["pred"] = ""
df["critique"] = ""
for i, row in df.iterrows():
s = judge(row["query"], row["response"])
df.at[i, "pred"] = s.label
df.at[i, "critique"] = s.critique
print(f"[{row['id']}] human={row['label']} judge={s.label}")
# Confusion matrix and rates.
y_true = df["label"].tolist()
y_pred = df["pred"].tolist()
tn, fp, fn, tp = confusion_matrix(
y_true, y_pred, labels=["FAIL", "PASS"]
).ravel()
tpr = tp / (tp + fn) if (tp + fn) else float("nan")
tnr = tn / (tn + fp) if (tn + fp) else float("nan")
agreement = (tp + tn) / len(df)
print(f"\nTPR (PASS recall): {tpr:.3f}")
print(f"TNR (FAIL recall): {tnr:.3f}")
print(f"Agreement : {agreement:.3f}")
print(f"Confusion: TP={tp} FP={fp} FN={fn} TN={tn}")
# Save disagreements for the next iteration of the rubric.
disagree = df[df["label"] != df["pred"]]
disagree.to_json("disagreements.json", orient="records", indent=2)
print(f"\nWrote {len(disagree)} disagreements to disagreements.json")
if __name__ == "__main__":
main()
claude-sonnet-4-5 is the wrong default now: Anthropic deprecated it on 2026-09-30 and retires it on 2026-11-30, with claude-sonnet-5-5 as the documented replacement 6. Sonnet 5.5 runs adaptive thinking when a request has no thinking field, and thinking tokens count against max_tokens, which is why the call sets between_tools, the documented way to turn up-front thinking off; with no tools in the request the response is text only 7. Checked against the SDK call signature and Anthropic's docs on 2026-10-01; not run against the live API.
If you prefer OpenAI, the swap is two lines: replace the client and the message-shape:
from openai import OpenAI
CLIENT = OpenAI() # reads OPENAI_API_KEY
MODEL = "gpt-4o-2024-08-06"
def judge(query: str, response: str) -> Score:
msg = CLIENT.chat.completions.create(
model=MODEL,
temperature=0,
response_format={"type": "json_object"},
messages=[
{
"role": "user",
"content": JUDGE_PROMPT.format(
rubric=RUBRIC, query=query, response=response
),
}
],
)
data = json.loads(msg.choices[0].message.content)
...
gpt-4o-2024-08-06 is still a listed snapshot on OpenAI's GPT-4o model page as of 2026-10-01 8. If you move the judge to the GPT-6 family, remove temperature too: OpenAI's guide says to drop it, along with top_p, whenever reasoning effort is not none 9.
Step 4: read the confusion matrix, not the agreement
When the script finishes, you will see something like:
TPR (PASS recall): 0.951
TNR (FAIL recall): 0.444
Agreement : 0.860
Confusion: TP=39 FP=5 FN=2 TN=4
The agreement number is 0.86 and tells you almost nothing. The TPR is 0.95 (the judge catches most real passes) but the TNR is 0.44 (the judge misses more than half of the real failures). That is the usual first-run shape: a rubric that has not yet been tightened against the failure modes the human labeler actually flagged. It is also why Hamel's judge guide says to report the two rates separately instead of raw agreement 3.
Naming note: this recipe treats PASS as the positive class, so TPR is PASS recall. Hamel's current FAQ defines TPR as the share of real failures the evaluator catches, which is this page's TNR 5. Check which class a number calls positive before comparing it with someone else's.
The fix is in disagreements.json. Read it. Find the pattern. Push it into the rubric.
Step 5: tighten the rubric and re-run
Open disagreements.json. Cluster the rows by what the judge missed. In the seed run above, the FAIL cases that the judge labeled PASS were almost all responses that named a vegan-friendly main dish but included a non-vegan garnish or sauce. The rubric did not mention garnishes.
One sentence added to the rubric:
RUBRIC = (
"Does the response respect every dietary restriction the user stated "
"earlier in the conversation? Restrictions include allergies, kosher, "
"halal, vegetarian, vegan, and keto. The response FAILS if any "
"ingredient (including garnishes, sauces, sides, or seasoning) "
"violates a stated restriction."
)
Re-run. On the labeled set above, TNR moves from 0.44 to 0.89. Total budget so far: one prompt revision, no new examples.
Figure: LLM judge calibration on the 50-trace dev set: the 0.86 agreement score and 0.95 TPR (PASS recall) look healthy, but TNR (FAIL recall) starts at 0.44, meaning the judge misses over half of real failures. One rubric sentence covering garnishes, sauces, and sides lifts TNR from 0.44 to 0.89, just under the 0.90 ship bar.
You will iterate three to five times before TPR and TNR both clear 0.90 on this dev set. Each iteration:
- Read the disagreements.
- Find the pattern the judge missed.
- Add one sentence to the rubric.
- Re-run.
Stop when both rates clear 0.90. Then hold out fifteen examples as a test set and read once. If the test numbers are within five points of the dev numbers, the loop is working. If they drift more than five points, your fifty examples are not representative; collect more 4. Fifteen examples is a smoke test of the loop, not a ship gate: Hamel's FAQ suggests, when possible, 30 to 50 Pass and 30 to 50 Fail examples in each of dev and test 5.
What changes when you have more data
This recipe is the minimum viable artifact. At fifty examples, your error bars are wide and you cannot reliably distinguish a 0.90 from a 0.88. At 200 examples the Wilson interval is roughly plus or minus four points; that is the threshold most teams ship behind. Hamel's FAQ now says to plan 100 to 200 labeled examples per failure mode 5. The statistics chapter covers the math.
The same recipe scales to thousands of examples without changing shape. The hand-labeling step gets compressed into a labeling tool (Label Studio, a Streamlit viewer, or Argilla, whose README now says it gets bug fixes but no new features 10). The judge call gets parallelized. The disagreement file becomes a queue routed to a domain expert. The rubric becomes a versioned artifact stored next to the prompt.
What this does not do
It does not measure pairwise preference (covered in pairwise vs direct). It does not handle multi-turn state (covered in the multi-turn cookbook). It does not check tool-call correctness (covered in agent eval with τ-bench). It does not run in CI; for the CI wiring see the production CI integration page.
What it does is replace "let me click around the demo" with a number you can defend, in less than an afternoon. Most teams find that the first pass surfaces failure modes nobody had named, which is exactly the value of writing the rubric down 11.
TIP
Save your traces.csv, the disagreements files, and the final rubric in version control. The rubric is now a contract; the next person to change the prompt has to re-run this script and beat the test numbers.
What to do next
Once this is calibrated, head to LLM-judge calibration for the rate-driven iteration loop in more detail. Then move to adding error bars once your labeled set crosses 200 examples. If your product is RAG-shaped, jump straight to RAG eval with Ragas instead.