Coding agents write eval code fast, and they get the method wrong in the same few ways. They make up labels, report raw agreement, reach for 1-to-5 scales, and call one prompt better than another from a single run with no interval. The agent pack puts this guide's method into your agent, so the eval code it writes measures the right thing.
Everything is free and open source at github.com/ombharatiya/ai-evals-skills.
What's in the pack
| Part | What it does | Use it when |
|---|
The ai-evals skill | Loads when you ask your agent for evals. It carries 12 ground rules, five workflows (add evals, write a judge, audit an eval, compare versions, set up CI) and six reference files with the guide's numbers and sources. | You work in Claude Code or Codex. |
| Cursor rule and AGENTS.md section | The same ground rules, condensed. Cursor attaches the rule to eval work; an AGENTS.md section is read on every task. | You work in Cursor, or in an agent that reads AGENTS.md. |
| Starter harnesses | Python and TypeScript code for a binary judge on Claude, calibration against your labels, code checks, a spend cap, and a release gate for CI. Tests run without an API key. | You want a working harness in your repo today. |
The skill tells the agent how to work. The starters give it working code to start from, so it doesn't write a harness from scratch.
Install
Claude Code
Run these two commands in Claude Code, one at a time. Pasted together, they arrive as one command and the second doesn't run.
/plugin marketplace add ombharatiya/ai-evals-skills
/plugin install ai-evals@ai-evals-skills
Claude loads the skill on its own when a task matches. You can also call it directly with /ai-evals:ai-evals. The skill's name and description add about 166 tokens to every session, and the full skill adds about 3,000 tokens when it loads. To get a newer version, run claude plugin update ai-evals@ai-evals-skills and restart Claude Code. The Claude Code skills documentation covers where skills live and how they load.
Codex
The same skill folder works in Codex. Ask Codex's built-in installer:
$skill-installer install https://github.com/ombharatiya/ai-evals-skills/tree/main/skills/ai-evals
Or copy the folder into ~/.agents/skills/ (for you) or .agents/skills/ at the root of your repo (for your team), then restart Codex. Codex uses the skill when a task matches its description, and you can call it by name from /skills. See the Codex skills documentation for the folders Codex reads.
Cursor
Copy rules/cursor/ai-evals.mdc into your project's .cursor/rules/ folder. Cursor attaches it when you work on evals.
Any agent that reads AGENTS.md
Paste rules/agents/AGENTS.md into the AGENTS.md at your repo root.
Your first session
The cookbook recipe walks through one feature end to end. The short version:
- Give the agent real traces. Export 100 or more inputs and outputs from your system as JSON Lines, one object per line, with a stable
id. If the feature isn't live, ask the agent to run it on 100 realistic inputs and save what comes back.
- Ask for the method, not a metric list. For example: "We need evals for the refund answers in our support bot. The traces are in
evals/data/traces.jsonl. Start with error analysis." A good answer proposes reading traces with you before writing any judge.
- Do the first read yourself. Read the first 100 traces and write a one-line note on the first thing that went wrong in each. Once you've coded at least 30, the agent can cluster your notes into four to seven failure modes and count them. See the open coding workflow.
- Fix the cheap failures first. Prompt fixes and code checks come before any judge.
- Build one judge for one subjective failure mode, label enough examples that dev holds at least 30 of each class (plan on about 200 labels, with failures oversampled when they're rare), and calibrate it with the starter. The agent reports PASS recall and FAIL recall with their N, not agreement.
- Read the test split once, then put the regression set and the judge in CI.
Prompts that work
| You want | Ask | A good answer |
|---|
| Evals for a feature that has none | "Add evals to the support bot. Start from the traces in evals/data, not from a list of metrics." | Asks you to read traces, plans error analysis, and holds off on judges |
| A judge for one failure mode | "Write a binary judge for refund-policy errors and calibrate it against our labels." | Writes the four-part judge prompt, splits train, dev and test, reports PASS and FAIL recall with N |
| A review of an existing eval | "Use the ai-evals skill to audit evals/." | Lists problems with file and line: agreement as the headline, few-shots leaking into test, an unpinned judge |
| A comparison of two prompts | "Is prompt v2 better than v1 on the release-gate set?" | Runs both on the same inputs and reports the paired difference, its interval and a ship, hold, investigate or block band |
| Evals in CI | "Run the regression set on every PR and post a result table." | A smoke tier with no judge, a full tier on prompt changes, and a verdict on the PR |
You don't need to name the skill. Asking for evals, a judge, labels or an eval step in CI is enough for the agent to load it.
What the skill won't do
- Make up ground truth. The agent can draft labels, marked provisional, for a person to confirm. It won't present its own labels as human judgments, and it won't let a model write both an input and its gold answer. See synthetic versus human data.
- Headline agreement. A judge that always says PASS agrees 95% of the time when 5% of outputs fail, so it reports the two recalls separately. See calibrating a judge.
- Use 1-to-5 scales. It writes one binary judge per failure mode and pushes back when asked for a score out of five. See the judge introduction.
- Read your test split twice. Edits after a test read turn the test split into a second dev set.
- Call a winner without an interval. It compares versions on the same inputs and says how large a change the run could have detected. See confidence intervals.
If your team's policy differs, change the rules. They're plain markdown under CC BY 4.0, so fork the repo and edit them.
Get the most out of it
- Keep eval data in the repo. Traces, labels, the codebook and rubrics belong in git next to the prompt they test. The agent can only reason about what it can read.
- Review its work like a new teammate's pull request. Check that each metric traces back to a failure someone saw in a trace, and that each judge has a calibration record.
- Keep the calibration summary you rely on. The starters write
runs/<timestamp>-calibrate-<split>/summary.json with the judge's fingerprint: model, effort, a hash of the full prompt, and the few-shot ids. Commit the one your release gate depends on; when the judge no longer matches it, the gate reports the run as invalid and fails.
- Count flips before trusting small changes. Run calibration with
--repeats 2. If labels flip between identical runs, a one-point move is noise.
- Cap spend. Set
EVAL_BUDGET_USD, and the starters stop making judge calls when the estimated spend reaches it.
- Give the regression set teeth. Mark reproduced bugs
"must_pass": true. One failure blocks the release.
The starters in five minutes
git clone https://github.com/ombharatiya/ai-evals-skills.git
cd ai-evals-skills/starters/python
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest
export ANTHROPIC_API_KEY=...
python -m evals.calibrate --data data/labeled.example.jsonl --rubric rubrics/refund_policy.md
The example data is a support bot for a fictional store, checked for one failure mode: getting the refund policy wrong. It's small, so it shows the loop without clearing the calibration bar. The TypeScript starter (Node 22.12+) runs the same loop. From the Python starter's folder:
cd ../typescript
npm install
npm test
npm run calibrate -- --data data/labeled.example.jsonl --rubric rubrics/refund_policy.md
Both READMEs cover reading the test split once and gating a release.
Troubleshooting
| What you see | Why | Fix |
|---|
| Claude Code: the plugin never installs after you paste both commands | Pasted together, the two commands arrive as one, and the second doesn't run. | Run them one at a time: /plugin marketplace add ombharatiya/ai-evals-skills, wait for it to finish, then /plugin install ai-evals@ai-evals-skills. |
| Claude Code: the agent writes evals without following the skill | The skill loads only when a task matches its description. | Ask for evals, a judge, labels or an eval step in CI, or call it directly with /ai-evals:ai-evals. |
| Claude Code: you can't or don't want to use the plugin system | The plugin is one of two ways to install. | Copy the folder instead: clone the repo, then copy skills/ai-evals into ~/.claude/skills/ (for you) or .claude/skills/ in your project (for your team). |
| Claude Code: an update doesn't show | Plugin updates apply after a restart. | Run claude plugin update ai-evals@ai-evals-skills and restart Claude Code. |
| Codex doesn't pick the skill up | A copied skill needs a restart, and Codex reads only its skill folders. | Check the folder is ~/.agents/skills/ai-evals or .agents/skills/ai-evals at your repo root, restart Codex, and call it with $ai-evals to confirm it loads. |
| Cursor ignores the rule | The rule file is in the wrong place. | Put ai-evals.mdc in your project's .cursor/rules/ folder, not the repo root. |
Python starter: pip install fails | The starter needs Python 3.10 or later. | Create the virtual environment with a newer Python (python3.12 -m venv .venv), activate it, and install again. |
TypeScript starter: npm install warns about the engine, or the scripts fail | The starter needs Node 22.12 or later. | Upgrade Node, delete node_modules, and run npm install again. |
| Calibration fails with an authentication error | The starter can't find your Anthropic credentials. | export ANTHROPIC_API_KEY=... in the same shell, or sign in with ant auth login. The tests run without a key; only calibration and scoring call the API. |
| The example calibration doesn't clear the bar | Expected. The example set has 40 rows, and its dev split holds only 8 FAIL rows. | Use your own labels: plan on 100 to 200 per failure mode, with 30 to 50 PASS and 30 to 50 FAIL in each of dev and test. |
| The release gate says INVALID | The run can't be trusted. The starter READMEs list six causes: a row with no verdict (a refusal, a budget stop, malformed output); ids missing between the run and its baseline; fewer than 30 paired rows; a verdict from a fallback model; a judge that differs from the calibration record; or a calibration record that isn't a passing test read. The example data always hits the last two, because its sets are too small. | Use your own data with at least 30 rows and a judge that passed calibration on its test split. Raise EVAL_BUDGET_USD if the run stopped on budget. If you changed the judge's model, effort or prompt, calibrate again and commit the new record. |
Still stuck? Open an issue on the repo with the command you ran and what it printed.
Questions
Does the skill send my code or data anywhere? No. The skill is text that loads into your agent's context. The starters call the Anthropic API only when you run calibration or scoring, and they send only the traces you pass them.
Which model does the starter judge use? claude-opus-5-5 at medium effort, set in JUDGE_MODEL and JUDGE_EFFORT. Calibrate again after changing either, because the calibration numbers belong to one exact judge.
My app runs on another model provider. Does this still work? Yes. The judge grades outputs, whatever produced them. To run the judge itself on another provider, replace the client call in judge.py or judge.ts and calibrate again.
What does it cost? The pack is free. The skill and rules are licensed CC BY 4.0 and the code MIT. Judge calls bill to your own Anthropic account.
Where do I report a problem? Open an issue on the repo. If a rule disagrees with this guide, link the page; the guide wins.