A coding agent can write an eval harness in minutes. Left alone, it also tends to invent the labels, score answers out of five, and report raw agreement, which is how a team ends up trusting a number that measures nothing. This recipe splits the work so that doesn't happen. You read traces, write the labels and make the calls. The agent builds the viewer, clusters your notes, writes the code, runs calibration and wires up CI.
Plan on about a working day, spread over a few sessions. Reading the first 100 traces takes a few hours, labeling for the judge takes a few more, and the agent's parts are quick. The reading is where the value is.
The agent follows this guide's method through the ai-evals skill from the agent pack. A skill is a folder with a SKILL.md file that the agent loads when a task matches its description, so the method arrives only when you ask for eval work 1. The same folder works in Claude Code and in Codex 2, because both read the Agent Skills format 3.
What you build
By the end, your repo holds:
evals/
data/traces.jsonl the traces you read
data/labeled.jsonl your human labels
data/gate_inputs.jsonl the release-gate inputs, with must_pass and check fields
scripts/generate_gate_outputs.py runs your feature on the gate inputs
codebook.md failure modes with definitions and examples
rubrics/<mode>.md one rubric per judged failure mode
calibration/summary.json the judge's calibration record
baselines/results.jsonl the run your release gate compares against
evals/ the starter's harness: judge, calibration, release gate
.github/workflows/evals.yml
Step 1: install the skill and the starter
In Claude Code, run these one at a time:
/plugin marketplace add ombharatiya/ai-evals-skills
/plugin install ai-evals@ai-evals-skills
In Codex:
$skill-installer install https://github.com/ombharatiya/ai-evals-skills/tree/main/skills/ai-evals
Then copy the Python starter into a new evals/ folder in your repo and check that it runs. The tests make no API calls.
git clone https://github.com/ombharatiya/ai-evals-skills.git /tmp/ai-evals-skills
mkdir evals
cp -r /tmp/ai-evals-skills/starters/python/. evals/
cd evals
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest
Every command below runs from inside evals/.
Step 2: give the agent your traces
Export at least 100 inputs and outputs from the feature as JSON Lines, one object per line, with an id that never changes:
{"id": "t-001", "query": "Can I get a refund after 60 days?", "response": "Yes, we refund any order within 90 days."}
Save them as data/traces.jsonl. Sample across the kinds of requests the feature gets, not only the recent ones. If the feature isn't live yet, ask the agent to run it on 100 realistic inputs and save the outputs in that shape. To practice first, use data/labeled.example.jsonl, a support bot for a fictional store checked for one failure mode: stating the refund policy wrong.
Step 3: run error analysis, and do the reading yourself
Ask for the method, not a metric list:
We need evals for the refund answers in our support bot. The traces are in data/traces.jsonl. Start with error analysis: build me a simple viewer where I can write a one-line note per trace, and don't propose metrics yet.
A good answer builds a small viewer (a static HTML page or a short script) that shows one trace at a time and saves your notes to a file. It doesn't hand you a list of generic metrics.
Read the first 100 traces yourself, at 15 to 45 seconds each. For each one, write a note on the first thing that went wrong, in your own words, or nothing if it's fine. This step can't be delegated: the failure modes only become clear while a person is reading 4. Once you've coded at least 30, the agent can start proposing clusters, but you keep reading to 100. Then ask:
Cluster my notes into failure modes. For each, write a 2-to-4-word label, a definition with its pass/fail test, two PASS and two FAIL trace ids, and a count. Save it as codebook.md.
Expect four to seven failure modes. Push back if the agent proposes generic labels such as "hallucination" before reading your notes, or merges two failures that need different fixes. Rank the modes by frequency times severity, with you setting the severity.
Step 4: fix the cheap failures first
For each failure mode in codebook.md, say whether a prompt change or a code check can handle it, and make those changes.
Prompt fixes come first. Code checks come next: the starter reads must_include, must_not_include and max_words from each row, and you can add your own checks to evals/checks.py. A row that fails a code check fails without a judge call. Keep an LLM judge only for the subjective failures that remain.
Step 5: write one judge and label for it
Take the remaining failure mode that ranked highest:
Write the rubric for a binary judge for "wrong refund policy" in rubrics/refund_policy.md: the role, exact definitions, and the PASS and FAIL criteria, ending with "when in doubt, FAIL". Don't add examples or an output format; the starter adds few-shots from the train split and the JSON schema itself.
That keeps examples out of the rubric file, where they could come from rows you later score. rubrics/refund_policy.md in the starter is a complete example.
Now label. Hamel and Shankar ask for 100 to 200 labels per failure mode, with 30 to 50 PASS and 30 to 50 FAIL in each of the dev and test splits 4. Dev is about 40% of your labels, so 30 FAIL rows in dev takes about 75 FAIL labels on average. The starter's split is a hash of each id, not stratified, so plan on about 200 labels with failures oversampled when they're rare; with 150, dev misses 30 of a class more often than not. The run prints the per-class counts, so you'll know. Each labeled row adds a label and, where it helps, a one-sentence critique:
{"id": "t-001", "query": "Can I get a refund after 60 days?", "response": "Yes, we refund any order within 90 days.", "label": "FAIL", "critique": "States a 90-day window; the policy allows 30 days from delivery."}
The agent may draft labels to speed this up. The skill has it mark them "provisional": true, and you confirm or change each one before it goes into data/labeled.jsonl.
Step 6: calibrate the judge on dev
export ANTHROPIC_API_KEY=...
export EVAL_BUDGET_USD=15
python -m evals.calibrate --data data/labeled.jsonl --rubric rubrics/refund_policy.md --repeats 2
EVAL_BUDGET_USD is a ceiling for each run, not an estimate, and every run prints its estimated cost. If a run reaches the ceiling, it says so: raise the ceiling and run the same command again.
The run splits your labels into train, dev and test, about 15 / 40 / 45. Each row's split comes from a hash of its id, so labeling more rows later never moves a row you already read in test into train or dev. The run puts four train examples in the prompt, judges the dev split twice, and prints this shape:
split=dev n=... train/dev/test=.../.../... judge=claude-opus-5-5@medium
PASS recall (TPR) 0.xxx 95% CI [0.xx, 0.xx] n=.. of .. PASS rows
FAIL recall (TNR) 0.xxx 95% CI [0.xx, 0.xx] n=.. of .. FAIL rows
kappa 0.xxx agreement 0.xxx (not the headline)
confusion TP=.. FN=.. FP=.. TN=..
Read FAIL recall first. It's the share of real failures the judge catches, and a weak rubric usually shows up there before anywhere else. Agreement can look fine while FAIL recall is poor 5. If labels flipped between the two repeats, the run prints how many: that tells you how much a one-point change could be noise.
Then hand the agent the disagreements:
Read runs/<latest>/disagreements.jsonl. Group the rows by what the judge missed and propose one change to the rubric. If a human label looks wrong, list it for me instead of changing the rubric.
Change one thing per run. The starter's bar is a verdict on every dev row, at least 30 rows of each class, PASS and FAIL recall of at least 0.90, and kappa of at least 0.6. Set your own recall bar from what a missed failure costs you; the run lists whatever is still short. Expect criteria to move while you do this: grading outputs changes what people think the criteria are, so keep the rubric under version control alongside the labels 6.
When dev clears the bar, read the test split once, again with two repeats so the record carries the judge's flip rate:
python -m evals.calibrate --data data/labeled.jsonl --rubric rubrics/refund_policy.md --split test --repeats 2
If the run ends with "Test clears the bar", keep that summary as the judge's calibration record; the release gate accepts only a passing test read:
mkdir -p calibration && cp runs/<this-test-run>/summary.json calibration/summary.json
If test misses the bar, don't copy that summary and don't tune on test: label a fresh batch, calibrate on dev again, and treat the next test read as a re-read.
Commit calibration/summary.json, because runs/ is git-ignored. Any later change to the rubric, the few-shots, the model or the effort changes the judge's fingerprint, and the release gate reports the run as invalid until you calibrate again, with fresh test labels for the new read.
Step 7: gate releases in CI
The gate compares the outputs of a pull request's code with a baseline, on the same inputs. Keep the inputs in data/gate_inputs.jsonl: an id and query per row, plus "must_pass": true and any code-check fields on rows that reproduce real bugs. Those rows are the regression set, and one failure among them blocks the release. Give them code checks where you can, so they don't depend on the judge.
Your feature produces the responses. Ask the agent for a small script that runs it on each input and writes data/gate.jsonl with the input fields plus a response. Then score the current version once to make the baseline:
python -m evals.run_eval --data data/gate.jsonl --rubric rubrics/refund_policy.md \
--labeled data/labeled.jsonl --calibration calibration/summary.json
mkdir -p baselines && cp runs/<this-run>/results.jsonl baselines/results.jsonl
Read the note the run prints about resolution. It says how wide the interval is for a candidate that matches the baseline on every row. A ship needs the whole interval inside the indifference window, so a window narrower than that can never ship an unchanged candidate. At a 76% pass rate, a 50-row gate set has a no-change interval of at least plus or minus 5.3 points, a 100-row set at least plus or minus 2.7, and the 1-point default window needs about 280 rows. Those are floors for a judge that never flips a label. If the judge flips 2% of its labels between identical runs, the 1-point window needs about 860 rows; the resolution note uses the flip rate measured by your --repeats 2 test read. Those are still floors. In CI the outputs are regenerated on every pull request, and generation noise widens the interval further, so size the window from an A/A run: generate and score the unchanged system twice, compare the second run with the first using --baseline, and set --indifference a little above the interval that prints. Smaller changes than that go undetected; growing the gate set is the only way to see them.
Then ask the agent:
Add a GitHub Actions job for pull requests that change the prompts, the feature code or the evals. It regenerates evals/data/gate.jsonl with the PR's code, scores it against evals/baselines/results.jsonl with ANTHROPIC_API_KEY from secrets and EVAL_BUDGET_USD set, fails on investigate, block, a regression-set failure or an invalid run, reports a hold as a warning for a person to read, and uploads the run folder. Skip it for pull requests from forks.
A good job looks like this. Replace the two feature steps with your own install and generate commands:
name: evals
on:
pull_request:
paths: ["prompts/**", "src/**", "evals/**"]
permissions:
contents: read
jobs:
gate:
# Fork PRs get no secrets, so skip them instead of failing.
if: github.event.pull_request.head.repo.full_name == github.repository
runs-on: ubuntu-latest
defaults:
run:
working-directory: evals
steps:
- uses: actions/checkout@v7
- uses: actions/setup-python@v7
with:
python-version: "3.12"
- run: pip install -e .
- name: Install the feature
run: pip install -e ..
- name: Run the feature on the gate inputs
# Your feature's own model calls happen here. EVAL_BUDGET_USD doesn't cap them.
run: python scripts/generate_gate_outputs.py --inputs data/gate_inputs.jsonl --out data/gate.jsonl
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
- name: Score and gate
run: >
python -m evals.run_eval --data data/gate.jsonl --rubric rubrics/refund_policy.md
--labeled data/labeled.jsonl --calibration calibration/summary.json
--baseline baselines/results.jsonl --indifference 0.03 --fail-on investigate
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
EVAL_BUDGET_USD: "10"
- name: Keep the results
if: always()
uses: actions/upload-artifact@v7
with:
name: eval-run
path: evals/runs/
Use the --indifference value your A/A run gives; 0.03 is only an example. Set EVAL_BUDGET_USD above the estimated cost your baseline run printed, since a gate run the ceiling stops is invalid and fails. With --fail-on investigate, the job fails on a confirmed regression (investigate or block) or a failed regression row, and a hold shows as a warning on the pull request with the delta and its interval. A hold means the interval can't tell the change from no change, so a person reads the rows whose verdict changed, in the uploaded results.jsonl, before merging; see release gates for what each band means. The job also fails, whatever --fail-on says, when the run is invalid: fewer than 30 rows pair with the baseline, rows are missing on either side, a row got no verdict or a fallback model's verdict, or the judge's fingerprint doesn't match a passing calibration record. A job skipped by the fork guard reports success, so for pull requests from forks a maintainer pushes the branch to the main repository to run the gate. When a release ships, refresh baselines/results.jsonl from its run.
What to push back on
| If the agent | Say |
|---|
| Proposes metrics before anyone has read a trace | "Error analysis first. Build the viewer." |
| Writes labels and treats them as ground truth | "Mark those provisional. I'll confirm each one." |
| Asks the judge for a score from 1 to 5 | "One binary judge per failure mode, PASS or FAIL with a critique." |
| Reports agreement as the result | "Give me PASS recall and FAIL recall with their N." |
| Puts examples in the rubric file, or dev and test rows in the few-shots | "Few-shots come from train only, and the starter adds them." |
| Calls version B better from one run | "Same inputs for both, the paired difference, and its interval." |
What this doesn't cover
Pairwise comparisons of two systems are covered in pairwise versus direct. Scoring live traffic is covered in online versus offline. Multi-turn conversations are covered in the multi-turn recipe. Checked against the starter's offline test suite on 2026-11-10, and the install commands against the Claude Code and Codex skill documentation on 2026-11-11; the judge calls were not run against the live API.