A judge that agrees with humans 95% of the time on a rubric humans only agree on 60% of the time is not calibrated; it is matching one rater's idiosyncrasy. Inter-rater agreement is the prerequisite measurement: before you trust a labelled set as ground truth, find out whether two qualified humans agree on the labels. If they do not, the rubric is the bug, and no amount of judge calibration fixes a rubric humans cannot follow.
The operational discipline: two domain experts double-label fifty traces per rubric. Compute Cohen's kappa for two raters and Krippendorff's alpha for more than two. If kappa is below 0.6, do not move forward; revise the rubric and relabel. Shankar et al. document the mechanism: graders kept refining their own criteria as they graded outputs (they call it criteria drift), so a rubric that has never been checked against a second rater is still a draft 1.
This is the discipline for a rubric that more than one person applies. A team with a single domain expert making the quality call can skip it; Hamel and Shankar's FAQ recommends one expert as the decision maker for most teams, and reaches for Cohen's kappa once multiple annotators label the same traces 2. A rubric that becomes a gold set or a release gate usually crosses that line.
Why raw agreement is misleading
A raw agreement of 90% sounds good. It can also be 90% on a class-imbalanced rubric where both raters say "pass" on 90% of the items because most items are easy and the rubric is uninformative on the rest. The raw rate is consistent with two raters who are coin-flipping on the hard cases and parking on "pass" everywhere else.
Cohen's kappa corrects for chance agreement. It is the observed agreement above what would be expected if the raters were guessing with their own marginal class rates. Kappa = 1 is perfect agreement; kappa = 0 is chance; negative kappa is worse than chance. The bands most eval programs use:
- Kappa below 0.4: the rubric is broken or the categories are unstable. Revise.
- Kappa 0.4 to 0.6: workable but borderline. The rubric needs sharpening.
- Kappa 0.6 to 0.8: the rubric is usable; judges can be calibrated against this gold set.
- Kappa above 0.8: high agreement; the rubric is well specified.
These cutoffs track the Landis and Koch labels, which were set by judgment and are not universally accepted; kappa also shifts with class prevalence and with how differently the two raters spread their labels 3. Treat 0.6 and 0.8 as team policy, not a statistical law.
Hamel makes the same point about judge agreement: raw agreement misleads when classes are imbalanced, so treat the human labels as ground truth and report the judge's true positive rate and true negative rate separately 4.
Figure: Interpretation bands for Cohen's kappa on eval rubrics: below 0.4 the rubric is broken, 0.4 to 0.6 is borderline (the worked example lands at 0.46), 0.6 to 0.8 is usable for judge calibration, and 0.8 or higher marks a well specified rubric fit for a gold set. Krippendorff's alpha uses the same 0.8 bar for multi-rater labelling.
Cohen's kappa for two raters
"""
Cohen's kappa for two raters on the same items. No external deps.
"""
from collections import Counter
def cohens_kappa(rater_a: list[str], rater_b: list[str]) -> float:
assert len(rater_a) == len(rater_b)
n = len(rater_a)
if n == 0:
return float("nan")
agree = sum(1 for a, b in zip(rater_a, rater_b) if a == b)
p_o = agree / n
counts_a = Counter(rater_a)
counts_b = Counter(rater_b)
categories = set(counts_a) | set(counts_b)
p_e = sum((counts_a[c] / n) * (counts_b[c] / n) for c in categories)
if 1 - p_e == 0:
return float("nan")
return (p_o - p_e) / (1 - p_e)
# Example: two raters on a binary pass/fail rubric.
a = ["pass"] * 70 + ["fail"] * 30
b = ["pass"] * 61 + ["fail"] * 9 + ["pass"] * 13 + ["fail"] * 17
print(cohens_kappa(a, b)) # ~0.46 (borderline)
For binary rubrics, the formula collapses to a clean expression and gives a quick gut check during rubric design. The kappa often surfaces "I thought we agreed on 'helpful' but we never wrote down what that means" before the project commits to the rubric.
In production code, use a library instead of the sketch above: scikit-learn's cohen_kappa_score takes weights="linear" or weights="quadratic" for ordinal rubrics 5.
Krippendorff's alpha for more than two raters
When more than two raters annotate and the data has missing cells (rater C did not see item 47), Cohen's kappa is the wrong tool. Krippendorff's alpha generalises the idea, handles any number of raters, any level of measurement (nominal, ordinal, ratio), and missing data 6.
The construction (nominal case) is:
- For every pair of raters who saw the same item, count disagreements; aggregate across items.
- Compare against the expected disagreement under random pairing across the full dataset.
- Alpha = 1 - (observed disagreement / expected disagreement).
A serviceable Python implementation is short for the nominal case:
from itertools import combinations
from collections import Counter, defaultdict
def krippendorff_alpha_nominal(ratings: dict[str, dict[str, str]]) -> float:
"""
ratings = {rater_id: {item_id: label_or_None}}
Items with fewer than two raters are skipped.
"""
# Build per-item rating multisets
items: dict[str, list[str]] = defaultdict(list)
for rater, by_item in ratings.items():
for item, label in by_item.items():
if label is not None:
items[item].append(label)
# Observed disagreement
obs_num = 0.0
n = 0 # pairable values: every label on an item with two or more raters
label_counts: Counter = Counter()
for labels in items.values():
m_u = len(labels)
if m_u < 2:
continue
local_counts = Counter(labels)
label_counts.update(labels)
# Disagreeing ordered pairs, weighted by 1 / (m_u - 1) so an item
# that more raters labelled does not count extra
disagree_pairs = sum(
local_counts[a] * local_counts[b]
for a, b in combinations(local_counts, 2)
) * 2 # ordered pairs
obs_num += disagree_pairs / (m_u - 1)
n += m_u
# Expected disagreement
exp_num = sum(
label_counts[a] * label_counts[b]
for a, b in combinations(label_counts, 2)
) * 2
exp_den = n * (n - 1)
if n < 2 or exp_den == 0:
return float("nan")
d_o = obs_num / n
d_e = exp_num / exp_den
if d_e == 0:
return 1.0
return 1 - d_o / d_e
This matches the published worked example in the Wikipedia article (three coders, fifteen units, missing cells): nominal alpha 0.691. Weighting each item's disagreements by 1 / (m_u - 1) is what keeps items with different numbers of raters comparable; dropping it gives a slightly different number whenever items have different numbers of raters. If you would rather not maintain this, the krippendorff package on PyPI computes alpha from a raters-by-items matrix with np.nan for missing cells; its level_of_measurement defaults to "interval", so pass "nominal" for pass/fail labels, or "ordinal" for ordered scales 7.
A common bar adopted across literature: alpha greater than or equal to 0.8 for high-stakes labelling, alpha greater than or equal to 0.667 for an "acceptable" floor in earlier exploratory work 6. The 0.8 bar is the one we recommend for any labelled set used as a calibration target or release gate.
What to do when agreement is low
Three causes recur in practice; each maps to a different fix.
The first is rubric ambiguity. Two raters disagree because the rubric does not adjudicate the case. Fix the rubric: write examples for the borderline cases, add anchors, decompose the rubric into independent axes. Shankar's validators paper makes the case that this is the rubric authoring loop, not a side activity 1.
The second is rater training. The raters interpret the rubric differently because nobody walked them through the borderline cases. Fix the training: hold a calibration session, label twenty items together, talk through disagreements, then double-label the next fifty independently and re-measure.
The third is rater fatigue or pressure. The same rater labels a thousand items in a day and quality degrades through the run. Fix the throughput: smaller batches, rotation across raters, a quality sample (the same items relabelled later in the run).
TIP
Kappa below 0.6 is a rubric problem, not a rater problem. Do not push the rubric into automation while humans cannot follow it consistently 2.
How agreement interacts with judge calibration
The standard sequence is rubric authoring, then human IRR, then judge calibration. Judge calibration against a rubric humans cannot follow produces a confident judge with no measurable agreement against the ground truth. The judge can be re-calibrated; the rubric still cannot be followed; the headline metric is decoration.
After the rubric clears the IRR bar (kappa above 0.6, ideally 0.8 for high-stakes; alpha above 0.8 for multi-rater), the judge calibration loop is the second step. In Hamel's Honeycomb example it took three iterations to reach above 90% agreement between the judge and the domain expert 4. Treat that as one data point, not a target: the classes in that example were roughly balanced, and agreement on imbalanced classes needs the TPR and TNR split described above.
CITATION
Wikipedia, "Cohen's kappa" and "Krippendorff's alpha." The canonical references for the formulas and bands. en.wikipedia.org/wiki/Cohen%27s_kappa and en.wikipedia.org/wiki/Krippendorff%27s_alpha
What to do this week
- Pick the rubric your judge is most reliant on. Double-label fifty items with two domain experts.
- Compute Cohen's kappa. If it lands below 0.6, do not calibrate the judge; revise the rubric and relabel.
- For multi-rater settings, switch from "average agreement" reporting to Krippendorff's alpha. The agreement bar stays at 0.8 for any rubric used as a gold set.