A judge prompt is a classifier specification written in English. The structure we use has four parts: a role with explicit definitions, the binary criteria, two or three few-shot examples lifted from your train split, and a strict output schema, the shape of the judge prompt in Hamel Husain's write-up 1. The bar it has to clear is TPR and TNR both above 0.90 on the held-out test split. Skip any of the four and you spend the next week debugging confusion.
Why these four parts
Each part fixes one failure mode that shows up if you skip it.
- Role and definitions kill ambiguity. "Vegan" means different things across product domains; you write the operative definition once so the judge stops guessing.
- Criteria force binary thinking. "PASS if X; FAIL if Y." No 1-to-5 scale, no "rate the helpfulness from 1 to 10." Binary lowers required sample size, raises inter-rater agreement, and forces you to be specific about what wrong actually means 2.
- Few-shot examples carry the boundary cases your prose cannot articulate. Lift them from your training split, never the dev or test split. The train split exists for exactly this purpose.
- Output format locks the schema you parse downstream. Use JSON with a fixed key order so a single string match validates the response.
The fourth part is non-negotiable. Free-form judge output gets parsed by string matching, and string matching silently misclassifies edge cases. A strict schema makes the parser noisy when the judge is noisy. Separately, Shankar et al. name "criteria drift": people need criteria to grade outputs, but grading outputs changes their criteria 3. Plan for it by versioning the rubric and re-labeling when the definition of pass moves.
Figure: The four-part LLM judge prompt template: (1) role and domain definitions, (2) binary PASS/FAIL criteria, (3) two to three few-shot examples lifted from the train split, and (4) a strict JSON output schema. Each part blocks one specific failure mode, from the judge guessing what domain terms mean to silent parser misclassification of free-form output.
The template
Below is the canonical four-part structure, lifted from the dietary-restriction example and generalized. Drop your domain definitions, criteria, and three train-split examples into the placeholders.
# Part 1. Role and domain definitions.
You are an expert {DOMAIN_ROLE} evaluating whether responses
properly adhere to {DOMAIN_CRITERION}.
DEFINITIONS:
- {TERM_1}: {EXACT_OPERATIVE_DEFINITION_1}
- {TERM_2}: {EXACT_OPERATIVE_DEFINITION_2}
- {TERM_3}: {EXACT_OPERATIVE_DEFINITION_3}
[... all terms the rubric depends on ...]
# Part 2. Evaluation criteria, binary.
EVALUATION CRITERIA:
- PASS: The response clearly adheres to {DOMAIN_CRITERION},
including both explicit content AND implicit method.
- FAIL: The response contains any element that violates
{DOMAIN_CRITERION}, even if the rest of the response is correct.
- When in doubt, FAIL and explain why.
# Part 3. Few-shot examples from the train split.
Example 1 (PASS):
Query: "{TRAIN_EXAMPLE_1_QUERY}"
Response: {TRAIN_EXAMPLE_1_RESPONSE}
Explanation: {WHY_THIS_PASSES}
Label: PASS
Example 2 (FAIL):
Query: "{TRAIN_EXAMPLE_2_QUERY}"
Response: {TRAIN_EXAMPLE_2_RESPONSE}
Explanation: {WHY_THIS_FAILS}
Label: FAIL
Example 3 (FAIL, edge case):
Query: "{TRAIN_EXAMPLE_3_QUERY}"
Response: {TRAIN_EXAMPLE_3_RESPONSE}
Explanation: {WHY_THIS_LOOKS_LIKE_PASS_BUT_FAILS}
Label: FAIL
# Part 4. Output schema, strict.
Now evaluate the following:
Query: {{query}}
{DOMAIN_CRITERION}: {{criterion}}
Response: {{response}}
Return JSON, no preamble, no postamble:
{{
"label": "PASS" or "FAIL",
"critique": "specific element(s) that drove the decision"
}}
The placeholders in single braces are template variables you replace at authoring time. The placeholders in double braces are runtime substitutions you do per trace.
Key order is a choice. Hamel's Honeycomb prompt asks for the critique first and the pass/fail outcome second 1. With a non-reasoning judge, that order lets the critique do the reasoning before the label is committed. With a reasoning model the order matters less, and label first keeps the parse trivial. Pick one and keep it fixed.
A worked example: dietary restrictions
The dietary-restriction judge from the seed study guide is a clean example because the domain has explicit, codifiable definitions and a real downstream consequence (someone with celiac disease cannot eat the gluten you missed).
You are an expert nutritionist and dietary specialist evaluating
whether recipe responses properly adhere to specified dietary
restrictions.
DIETARY RESTRICTION DEFINITIONS:
- Vegan: No animal products (meat, dairy, eggs, honey).
- Vegetarian: No meat or fish; dairy and eggs allowed.
- Gluten-free: No wheat, barley, rye, or other gluten-containing grains.
- Dairy-free: No milk, cheese, butter, yogurt.
- Keto: Very low carb (<20g net carbs), high fat, moderate protein.
- Paleo: No grains, legumes, dairy, refined sugar, processed foods.
- Raw vegan: No animal products AND no foods heated above 118 F (48 C).
EVALUATION CRITERIA:
- PASS: Recipe clearly adheres to the dietary restriction in both
ingredients AND preparation method.
- FAIL: Recipe contains ingredients or methods that violate the
dietary restriction. When ambiguous, FAIL and explain why.
Example 1 (PASS):
Query: "Gluten-free pizza dough that actually tastes good"
Response: [Recipe using a gluten-free flour blend, baking powder,
olive oil, honey, apple cider vinegar.]
Explanation: All ingredients are gluten-free. Method introduces
no gluten contamination.
Label: PASS
Example 2 (FAIL):
Query: "Raw vegan Mediterranean quinoa salad"
Response: [Recipe with cooked quinoa, vegetables, olive oil, lemon.]
Explanation: Raw vegan forbids foods heated above 118 F. Boiling
quinoa exceeds that limit.
Label: FAIL
Example 3 (FAIL, edge case):
Query: "Quick keto breakfast"
Response: [Recipe using almond flour pancakes with maple syrup.]
Explanation: Maple syrup adds roughly 13g net carbs per tablespoon,
blowing the <20g keto budget on a single serving.
Label: FAIL
Now evaluate the following:
Query: {{query}}
Dietary Restriction: {{restriction}}
Recipe Response: {{response}}
Return JSON, no preamble:
{{
"label": "PASS" or "FAIL",
"critique": "specific ingredient(s) or method that drove the call"
}}
The structure is the lesson; the domain is interchangeable. Customer support gets a refund-policy version. Code review gets a style-guide version. RAG faithfulness gets a "is this claim supported by the retrieved context" version.
Invocation
The judge is one API call per trace. Pin the model ID and the effort setting, ask for JSON, and parse strictly. The boilerplate below follows the Claude API docs for claude-opus-5-5. Claude Opus 4.7 and later reject non-default temperature, top_p, and top_k with a 400 error, and a response can begin with thinking blocks, so the code reads the text block by type instead of by position 4.
import json
import os
from anthropic import Anthropic
from string import Template
client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
JUDGE_TEMPLATE = Template(open("dietary_judge.txt").read())
# The template file uses $query, $restriction, $response.
def run_judge(query: str, restriction: str, response: str) -> dict:
prompt = JUDGE_TEMPLATE.substitute(
query=query, restriction=restriction, response=response
)
result = client.messages.create(
model="claude-opus-5-5",
max_tokens=4096, # thinking tokens count against this limit
output_config={"effort": "medium"}, # pin it; effort changes judge behavior
messages=[{"role": "user", "content": prompt}],
)
if result.stop_reason == "max_tokens":
raise ValueError("judge output truncated; raise max_tokens")
# A response can start with thinking blocks; read the text block by type.
raw = "".join(b.text for b in result.content if b.type == "text").strip()
# Strict parse. Bail loudly on schema drift.
try:
parsed = json.loads(raw)
except json.JSONDecodeError as e:
raise ValueError(f"judge returned non-JSON: {raw[:200]}") from e
if parsed.get("label") not in {"PASS", "FAIL"}:
raise ValueError(f"judge label invalid: {parsed!r}")
return parsed
if __name__ == "__main__":
out = run_judge(
query="Gluten-free pasta carbonara",
restriction="gluten-free",
response="Use store-bought brown rice spaghetti, pancetta, eggs, "
"pecorino, black pepper. Cook pasta per package directions.",
)
print(out)
# Expected: {"label": "PASS", "critique": "Brown rice spaghetti..."}
Three implementation notes that save real time:
- Cut sampling variation where the API lets you. A judge is a classifier; you do not want sampled variation. On APIs that accept
temperature, set it to zero. Claude Opus 4.7 and later do not accept it, and Anthropic notes that temperature = 0 never guaranteed identical outputs on earlier models 4. So measure the flips instead of assuming determinism: run the judge twice over your dev split and count label changes. If you need a confidence signal, read the label token's logprob where the API exposes it, or report the vote split across repeated runs.
- Strict parser. A judge that returns plain text "PASS" because it forgot the JSON wrapper has to fail loud, not silently. The parser is your contract. Where the provider offers schema-constrained decoding, use it as well:
output_config.format with a json_schema is generally available on current Claude models 5. It removes malformed JSON, not wrong labels, and a refusal can still come back in a shape that does not match the schema, so the parser stays.
- One judge model per rubric. Do not mix Claude and GPT calls on the same rubric without re-calibrating; cross-family agreement is not free. Every Claude model ID is a pinned snapshot 6, so the ID is your pin, and a newer model is a new judge that gets its own calibration run. Check the defaults when you upgrade: Claude Opus 5.5 runs at medium effort when you omit the setting, one level below Claude Opus 5 4.
Few-shot from the train split, never dev or test
The one mistake that ruins a calibration: pulling few-shot examples from the dev or test set. If the judge has seen the dev examples in its prompt, the dev metrics are not a fair signal anymore. The split Hamel and Shankar recommend is 10 to 20 percent train and 40 to 45 percent each for dev and test, with 30 to 50 PASS and 30 to 50 FAIL examples in both dev and test so every split has both classes 2.
The train split exists for one reason: pick few-shot examples. The dev split is the iteration target; the test split is the unbiased number you cite to your team. The calibration page walks the split discipline end to end.
CITATION
Hamel Husain, "Creating an LLM-as-a-Judge That Drives Business Results." The Honeycomb query-evaluator case study, a judge prompt built from domain definitions and few-shot critiques, the train/dev/test discipline, the binary-not-Likert rule, and TPR and TNR over raw agreement. The canonical practitioner reference; the post was last modified 2026-09-01. hamel.dev/blog/posts/llm-judge
What you do not need
You do not need a chain-of-thought block before the label. Liu et al. report CoT helps on G-Eval-style scoring 7, but the cost in tokens and latency rarely pays back on a well-scoped binary rubric. If a rubric is complex enough to need CoT, the rubric is doing too much; split it.
Current reasoning models blur this line. Claude Opus 5.5 always thinks adaptively before it answers, and you steer how much with the effort setting instead of a prompt block 4. On JudgeBench, reasoning effort moved o3-mini from 70.6 to 80.9 percent 8. For a hard-correctness rubric, effort is the setting to sweep during calibration. For a well-scoped binary rubric, start at the lowest level that holds TPR and TNR on the dev split.
You do not need a custom rubric per persona. Build one rubric per business criterion, not per user segment.
You do not need a Likert scale. We covered this elsewhere; binary plus critique is the Hamel and Shankar default because binary labels are more consistent across annotators and need smaller samples to detect a difference 2. Anthropic's eval guide shows both binary and 1-to-5 graders 9, so expect to see Likert in vendor docs and keep the binary default anyway.
What is next
Once the prompt is written, the judge is not deployed; it is a draft. The calibration page shows the train/dev/test loop and the sklearn snippet for TPR and TNR. The biases page shows the position-swap and self-preference fixes you bake into the harness.