A rubric is the falsifiable definition of pass. Most eval programs that fail in the first month fail because the rubric was a vibe, not a definition. The discipline is small (five rules) and the payoff is large.
Binary, not Likert
Use binary PASS or FAIL with a written critique, not a 1-to-5 or 1-to-10 Likert scale. Hamel's line: "1-5 scales are often a sign of a bad eval process" because they collapse the rubric author's job (define what wrong looks like) into the rater's job (pick a middling number) 1. Binary wins on three measurable axes.
Sample size. A binary classifier reaches a given CI width at roughly one fourth the N of a Likert estimator; Bernoulli variance at p=0.5 is fixed and small (0.25), Likert variance at the mean is larger.
Inter-rater agreement. Binary forces both raters to land on the same side of a clean cutoff; Likert lets them slide. A rubric that hit kappa 0.85 on binary often hits kappa 0.55 on the same content in 1-5 form.
Specificity. Writing PASS and FAIL forces the author to specify what failure looks like; writing 1-5 lets the author punt the boundary cases ("a 3 is when it's mostly OK"). The judge then defaults to the middle and the calibration runs aground.
The exception that proves the rule. If a downstream system consumes a continuous score built from preference data (a reward model trained on pairwise preferences, a ranking model), the upstream eval can collect Likert. If the reward is rubric-scored rather than preference-scored, the 2026 layer below applies and the criteria should stay decomposed binaries. For an eval program whose output is a release-gate decision, the decision is binary and the rubric should be too.
Multi-dimensional means multiple binaries
A customer-support reply has to be faithful to the policy, concise, and on-brand. The right rubric is three binaries, not one 1-to-5 2. Each dimension gets its own PASS or FAIL criterion, few-shot examples, and confusion matrix. The system-level pass rate is a product (or weighted product) of the per-dimension rates.
| Pattern | What it scores | When to use |
|---|
| Single binary | one dimension | one dimension dominates |
| Multiple binaries | N independent dimensions | recommended default |
| 1-5 Likert | scalar quality | almost never |
| 1-5 per dimension | N scalars | inherits Likert problems N times |
| Pairwise preference | overall preference | see pairwise |
The three-binary pattern also makes failure attribution clean. A drop in faithfulness points at retrieval, a drop in conciseness points at the prompt, a drop in tone points at the system message. A scalar collapses all three and the on-call cannot triage.
Figure: Rubric design for LLM evals: decompose a single 1-5 Likert quality scale into three binary PASS/FAIL criteria (faithful to policy, concise, on-brand), each with its own critique and failure attribution. Binary rubrics reach inter-rater kappa 0.85 where the same content scores 0.55 on Likert, and need roughly one fourth the sample size for the same confidence interval.
Criteria drift
Shankar et al. document a specific failure mode: the human's notion of pass shifts as they look at more outputs 3. The rubric on day one does not match the rubric on day thirty. The judge calibrated against day-one labels under-performs on day-thirty traces, and the program looks like the judge degraded when the truth signal drifted.
Three mitigations. Lock the rubric in a versioned file tied to the calibration set; every recalibration re-reads the rubric verbatim. Have the principal expert blind-label fifty fresh traces against the live rubric every quarter; disagreement above ten percent triggers a rubric revision. Treat rubric versions as artifacts; a judge calibrated against rubric v1 has TPR and TNR scoped to v1.
Write the critique alongside the label
A judge that returns only PASS or FAIL is debug-hostile. A judge that returns a one-sentence critique externalizes the criterion, speeds up calibration sessions (the expert reads the critique, not the full trace), and trains junior reviewers by example 4. The marginal cost is a few hundred tokens per judge call.
The critique should cite the specific element that drove the call ("the recipe uses cooked quinoa; raw vegan forbids cooking above 118 F"), not a generic restatement of the verdict.
Pin the schema
Sclar et al. show meaning-preserving format changes can move LLaMA accuracy by up to seventy-six points 5. For rubric design: "PASS or FAIL" versus "Yes or No" versus "1 or 0" are three different rubrics from the judge's perspective. Pick one and pin it. A strict JSON output schema does double duty: it makes the parser fail loudly on drift and pins the option labels through prompt edits.
The five-minute falsifiability check
Before committing a rubric:
- Write PASS in one sentence.
- Write FAIL in one sentence.
- Write three boundary traces by hand: one obvious PASS, one obvious FAIL, one edge case.
- Have the principal expert label all three independently.
- If all three labels match the author's, the rubric is shippable. If any disagree, the rubric is not yet falsifiable.
IFEval is a clean example of falsifiable rubrics at scale: every criterion is verifiable by a deterministic script, with zero rater judgment required 6. The same discipline applies inside an LLM-judge rubric.
The 2026 layer: your rubric is now a reward spec
Everything above is about rubrics as an eval instrument. The shift over 2025 and 2026 is that the same rubric became a training signal, which raises the stakes on getting it right. Reinforcement learning from verifiable rewards worked where an answer could be checked by a script; rubrics extended it to open-ended domains by turning "what does good look like" into a graded reward. Rubrics as Rewards used instance-specific rubrics as the RL reward and reported relative gains up to 31% on HealthBench and 7% on GPQA-Diamond over a Likert LLM-judge baseline, and found that rubric-based rewards let smaller judge models align with human preferences at lower variance 7. Reinforcement Learning with Rubric Anchors scaled the idea to a reward system of more than 10,000 rubrics 8. OpenRubrics is the scalable-authoring answer: its Contrastive Rubric Generation derives criteria by contrasting preferred against rejected responses, and the resulting reward model beat size-matched baselines by 8.4% 9.
The practical consequence for this page: whatever you write as an eval rubric is now also a candidate reward spec, so the falsifiability discipline above is no longer just about clean measurement. A vague criterion that a human rater quietly averages over becomes, in an RL loop, a gradient the policy will climb in whatever way is cheapest.
Static rubrics get hacked
A fixed rubric loses discriminative power as the policy improves against it: the criteria that separated good from bad at the start become things every sample now satisfies, the reward saturates, and the optimizer starts collecting points in ways the author never meant. EvoRubrics names this as the reason to co-evolve rubrics alongside the policy rather than freeze them 10. The failure has a taxonomy: an analysis of rubric-based RL identifies two distinct divergence sources, a weak verifier that scores generously and rubric-design limitations, and documents concrete exploits like partial satisfaction of a compound criterion (the model does one of the three things the criterion bundled) and treating implicit content as explicit 11. The exploitation grows over training, so it is invisible in a one-shot eval and obvious in a reward curve.
Three rules fall out, and they are good rubric hygiene even if you never train on the rubric. Decompose compound criteria into separate binaries, which is the same advice as the multi-dimensional section above with higher stakes. Use a strong verifier, because a generous grader is where most of the leaked reward comes from. And refresh the rubric as the model distribution shifts, rather than treating a rubric written against last quarter's model as permanent.
The production grader stack
When the rubric drives an automated grader, it stops being prose and becomes a typed object. OpenAI's grader API is a useful concrete map of the shapes a rubric criterion takes in production, and the same grader object serves both offline evals and reinforcement fine-tuning 12:
| Grader type | What it checks |
|---|
string_check | Exact or case-insensitive string match (eq, neq, like, ilike) |
text_similarity | Fuzzy overlap: BLEU, ROUGE, METEOR, cosine, fuzzy_match |
score_model | An LLM scorer with a model, a numeric range, and a pass threshold |
label_model | An LLM classifier that assigns labels, with a set of passing labels |
python | Arbitrary code implementing a grade() that returns a float |
multi | Combines nested graders into one output |
The lesson from the binary section carries: reach for label_model with a small label set or a python check with a deterministic rule before score_model, because a model returning a number is exactly the mushy scalar this page argues against, one layer down.
Validate the rubric like a measurement instrument
The most rigorous public example of rubric validation is HealthBench, whose criteria were written by 262 physicians and total 48,562 unique items across 5,000 conversations, each carrying a weight from -10 to +10 so that a violated safety criterion subtracts rather than merely failing to add 13. What makes it a model to copy is not the scale but the meta-evaluation: HealthBench measured its own grader against physician labels across tens of thousands of meta-examples and reported a grader macro F1 of 0.709, benchmarked against physician-versus-physician agreement rather than an absolute bar 13. That is the move to steal. Meta-evaluate your grader against human raters per criterion, and compare it to how well your humans agree with each other, not to 100%. Rubric versioning in production, by contrast, has no published primary literature yet; treat version-pinning your rubric as engineering discipline (the criteria-drift section covers why), not as a cited practice.
For the field-level map of how rubrics now span evaluation, RL, and safety alignment, the 2026 survey is a useful index 14.
When to use a scalar after all
Three legitimate cases. The downstream consumer is a preference-trained reward model (preference data with scalars trains stable reward heads; rubric-scored rewards are the 2026 layer's territory and stay binary per criterion). The eval is a ranking, not a release gate (Chatbot Arena fits scalars from binary pairwise preferences, not direct collection). The metric is genuinely continuous (token-overlap, embedding-similarity, latency, cost; covered on reference-based metrics and embedding similarity).
For a production judge program whose output is a release decision, none apply. Stick to binary, write the critique, pin the schema, version the rubric. The judge prompt structure chapter walks the template.