Most teams are at stage one and think they are at stage three. The point of the ladder is to pick the next rung honestly, not the one you wish you were on. Each rung produces a small, named artifact. If you cannot produce that artifact in a week, the rung is wrong for now.
The five rungs
Hamel's three levels (L1 assertions, L2 human-plus-judge on a labeled set, L3 production A/B) form the spine of the model [1]. Stage 0 sits below L1 as the honest description of where most teams start. Stage 4 sits above L3 as the continuous quality program Shankar describes [2].
flowchart TD
S0["Stage 0\nVibe checking"]
S1["Stage 1\nAssertions + small seed set\n(Hamel L1)"]
S2["Stage 2\nLLM-judge on labeled dataset\n(Hamel L2)"]
S3["Stage 3\nProduction A/B with online judge\n(Hamel L3)"]
S4["Stage 4\nContinuous quality program\n(Shankar flywheel)"]
S0 --> S1 --> S2 --> S3 --> S4
S1 -. skip-ahead failure .-> S3
S2 -. skip-ahead failure .-> S4
The dashed arrows are the common skip-ahead failures. A team at stage 1 cannot make a useful A/B test because they do not know what to measure. A team at stage 2 cannot run a continuous program because they have not decided which segments to sample and why. Each rung exists because the rung below it pays back the cost of the rung above.
Stage 0: Vibe-checking
The product manager clicks through ten examples, declares the new prompt better than the old prompt, and you ship. There is no dataset. There is no metric. There is no recorded baseline. Two weeks later nobody can tell whether the regression report is a real regression or a different prompt-format swing [3].
Stage 0 is the right rung for the first week of a prototype. It is the wrong rung the moment two people on the team disagree on what counts as broken.
Stage 1: Unit-test-style assertions
The first useful artifact is a tiny seed set with binary assertions. Thirty hand-curated examples. Three or four code-based checks that catch the obvious classes of failure: did the model output valid JSON, did it include the policy disclaimer, did it stay under the token cap, did it call the right tool with the right arguments. Hamel calls these Level 1 and describes them as unit tests for LLMs: assertions, like you would write in pytest [1]. Seed the set from real failures, not invented cases: Anthropic's agent-evals guide puts the starting size at 20 to 50 tasks drawn from real failures, since early changes have large effects and large effects show up in small samples [4].
The mistake at stage 1 is reaching for a judge before the assertions exist. If json.loads would catch a third of your failures, write json.loads. The judge is for the cases code cannot reach.
import json
from typing import Callable
# A stage-1 eval is a list of cheap, deterministic checks plus a tiny seed set.
def is_valid_json(text: str) -> bool:
try:
json.loads(text)
return True
except json.JSONDecodeError:
return False
def has_policy_footer(text: str) -> bool:
return "Reply STOP to opt out" in text
def no_markdown_in_sms(text: str) -> bool:
# SMS does not render markdown. Reject responses that contain it.
return not any(c in text for c in ["**", "## ", "* ", "_ "])
ASSERTIONS: dict[str, Callable[[str], bool]] = {
"valid_json": is_valid_json,
"policy_footer": has_policy_footer,
"no_markdown_in_sms": no_markdown_in_sms,
}
def run_stage_1(seed: list[dict]) -> dict:
"""Run every assertion against every example. Return per-assertion pass-rate."""
results = {name: 0 for name in ASSERTIONS}
for example in seed:
output = example["output"]
for name, check in ASSERTIONS.items():
if check(output):
results[name] += 1
return {name: count / len(seed) for name, count in results.items()}
Stage 2: LLM-judge on a labeled set
When the failures stop being mechanical, the judgment has to come from a human or an LLM that has been calibrated to a human. Stage 2 is a judge prompt per failure mode and the labels to validate it. Hamel and Shankar's current guidance is 100 to 200 labeled traces for each failure mode, split into train, dev, and test, with the judge's true-positive and true-negative rates documented against the principal domain expert [5]. Report both rates instead of raw agreement: at a 5 percent failure rate, a judge that always says pass still agrees with the human 95 percent of the time [6].
The rubric is binary and scoped to one failure mode, the calibration loop runs on the dev split, and the final check is the held-out test split, where both rates have to clear a bar you set from the cost of a missed failure and the cost of a false alarm. This site defaults to 0.9 on each. Start the judge on the most capable model your budget allows and try cheaper ones only after it agrees with the labels; using the generator's own model as the judge is usually fine for a scoped binary check [5]. Hamel's Honeycomb case took three iterations to pass 90 percent agreement [6]. If you cannot get there in three iterations, the rubric is the bug, not the judge.
Stage 2 is also where dataset versioning starts to matter. If you relabel three borderline cases between the run that showed regression and the run that showed improvement, the improvement is in your labeling, not in the model.
Stage 3: Production A/B with an online judge
By stage 3 you have a calibrated offline judge, a labeled dataset that you maintain on a cadence, and the next step is to learn what offline cannot teach you: how the system behaves on inputs you did not seed. The online judge runs on a sampled fraction of production traces. The FAQ draws the line this way: CI evals protect against known regressions before deployment, and online monitoring finds failures in production traffic and estimates how often they occur [5].
Stage 3 needs three things stage 2 did not: a sampling policy (uniform vs oversample-on-failure vs oversample-on-segment), a way to attribute scores to model and prompt versions, and a release-gate decision rule. The decision rule is where most teams' eval programs fall apart. A 2-point quality bump that costs 30% more in tokens is not a win; cost and latency belong in the scorecard [7].
Stage 4: Continuous quality program
Stage 4 is the flywheel: trace logging, error analysis on a cadence, dataset refresh, judge recalibration, release gates, and postmortems all referencing the same metric. Shankar's framing is that the labeled set keeps evolving with the product: sample and label production responses on a schedule, and store them with timestamps so you can track recency [2]. Hamel and Shankar's FAQ puts numbers on the cadence: at least 100 fresh traces per error-analysis cycle, cycles of 2 to 4 weeks, and 10 to 20 traces a week in between [5]. Split the suite by purpose too. Capability evals start at a low pass rate and give you something to climb. Regression evals sit near 100 percent and flag backsliding. A capability eval that saturates graduates into the regression suite [4].
At stage 4 the eval program has a sustaining team, a budget line, and a written charter. The artifacts are: a dataset catalog, a judge registry, a calibration log, a release-gate policy, and a published quarterly readout. If you are pre-Series-B and pre-product-market-fit, you do not need stage 4 yet. If you are a public company with regulators in the loop, you needed it before launch.
Self-assessment checklist
Run this ten-item checklist honestly. Each yes counts one point. The score maps to the rung you can defend in a room.
| # | Question | Yes |
|---|
| 1 | We have a labeled dataset of at least 30 examples checked into git. | |
| 2 | Every prompt change is run against that dataset before merge. | |
| 3 | At least three deterministic assertions run against every output. | |
| 4 | We have an LLM judge with a documented binary rubric. | |
| 5 | We have measured TPR and TNR for the judge against a human expert. | |
| 6 | The dataset is content-hashed so we can pin results to a version. | |
| 7 | The judge runs on a sampled fraction of production traces. | |
| 8 | Cost per request and p95 latency are columns on the same scorecard as quality. | |
| 9 | A release-gate policy specifies what scorecard movement blocks rollout. | |
| 10 | Error analysis runs on a defined cadence and the rubric is on the table. | |
Score 0-1: stage 0. Score 2-3: stage 1. Score 4-6: stage 2. Score 7-8: stage 3. Score 9-10: stage 4.
The most common honest result is 2 or 3. Most teams that self-report stage 3 score 4 on this checklist. The most actionable score is the lowest no answer; that is the next rung.
The smallest useful artifact at each rung
If you cannot produce the artifact at the cost listed in a week, the rung is wrong for now. The cost estimate assumes a small team with the principal domain expert reachable on Slack [5].
0 -> 1 30 labeled examples, 3 assertions, a CI hook (1 person-week)
1 -> 2 100+ traces/mode, a judge prompt, TPR/TNR vs 1 human (2 person-weeks)
2 -> 3 sampling policy, online judge, weekly drift report (4 person-weeks)
3 -> 4 dataset versioning, release gates, charter, postmortem (1 quarter)
The geometric blow-up is real. Stage 0 to stage 1 is a week. Stage 3 to stage 4 is a quarter. The mistake is not the scope. The mistake is skipping rungs to chase the appearance of maturity.
CAUTION
A team that skips from stage 1 to stage 3 will spend the savings, and then some, debugging an unmeasurable system that nobody can roll back with confidence.
Figure: Cost to climb each rung of the AI eval maturity model: 1 person-week for assertions plus a 30-example seed set, 2 person-weeks for a calibrated LLM judge on 100 labeled traces, 4 person-weeks for an online judge with a sampling policy, and roughly 12 person-weeks (a full quarter) for a continuous quality program. The geometric blow-up is why skipping rungs fails.
Where to invest next
Pick the next rung by the smallest artifact that would change a decision you are making this quarter.
If you are at stage 1 and about to swap models, you need stage 2 first. Without a calibrated judge on a labeled set, the model swap will look like a gain or a loss that sits inside the noise: at 100 examples, a 95 percent Wilson interval on a pass rate near 50 percent is about 19 points wide [8]. Comparing both models on the same questions tightens the difference, but only if you have the labeled set to compare on [9].
If you are at stage 2 and seeing regressions reported by users that your offline eval missed, you need stage 3, not a smarter judge. The judge is missing the regressions because the dataset is missing the inputs. Online sampling is the way the dataset learns about the inputs you did not anticipate.
If you are at stage 3 and the eval team is bottlenecking every release, you need stage 4, not more judges. The bottleneck is governance, not throughput. A written release-gate policy unblocks half the conversations the team is having.
TIP
Pick the rung where the next decision is bottlenecked on a missing artifact, not on a missing dashboard.
The point of the model is to keep you honest about the cost. Eval programs that try to land at stage 4 in one quarter have a known failure mode: a half-built stage 2 that nobody trusts, a half-built stage 3 that drifts silently, and an audit binder that does not match the running system. Land at stage 1 properly. Then land at stage 2 properly. Each rung is its own win.
What gets cut
The corollary to picking the next rung is that everything above it gets postponed. The list of things to skip is short and specific.
Skip the dashboard before stage 2. A dashboard without a dataset and a judge is decoration. The graphs will go down and you will not know whether the model got worse or someone changed the input format [10].
Skip the multi-judge ensemble before stage 3. A single calibrated judge with documented TPR/TNR is more useful than three uncalibrated judges in a panel. Ensembles compound bias when the bases are biased the same way.
Skip the governance framework before the loop is closed. A risk register that does not reference a measured metric is a Word document, not a control. The governance section of this site covers what changes when you are at stage 4 and a regulator is the audience. Until then, the audience is the team, and the team needs a working judge before it needs a written policy.
The maturity model is not aspirational. It is a sequencing constraint.