A CI that runs every eval on every PR will be too slow, too expensive, and will get bypassed within a month. A CI that runs no evals will let regressions through and surface them in production. The working pattern lives between: tier the suite, run the smoke set on every push, the full set on merge candidates, and the nightly set on schedule. Cap the cost per PR, publish the result table as a PR comment, and gate the merge on the change in metric, not the absolute value.
Hamel's argument that evals are a level of testing applies straight: Level 1 is pytest-style assertions, Level 2 is human-plus-judge on a labelled set, Level 3 is production A/B 1. CI's job is to make Level 1 cheap to fail and Level 2 fast to inspect. Production A/B lives downstream.
The three tiers
Smoke. Ten to fifty assertions covering the highest-value workflows. Runs on every push, in under two minutes, at single-digit-dollar cost. The point is to catch a malformed prompt change, a regression on a canonical happy path, or a broken tool wiring before the PR sits in review. Smoke does not need a judge; deterministic checks (schema validation, regex on outputs, exact-match on a tiny golden set) are enough for the layer.
Full. One to five hundred labelled examples covering the full feature surface, run with the calibrated judge. Triggered on PRs labelled eval and on every merge candidate. Budget five to fifteen minutes and tens of dollars. The full tier is the merge gate; the result is a quality, cost, and latency triple with intervals against the baseline.
Gate on the regression half of that suite. Anthropic's guide to agent evals expects regression evals to sit near a 100% pass rate and capability evals to start low, with capability evals that reach high pass rates graduating into the regression suite; it also puts automated evals in CI on each agent change and model upgrade 2. A capability eval that starts at a low pass rate is a progress metric, not a merge gate. Hamel's FAQ makes the cost argument for keeping CI lean: a small, curated CI set built around core features, past bugs, and edge cases, deterministic checks favored over judges, and representative examples added when production error analysis turns up a new failure pattern 3. That is why the judge runs in full and not in smoke.
Nightly. The expanded regression suite plus contamination probes, robustness sweeps (format, order, temperature), and drift comparisons against the previous nightly. Runs on a schedule, posts the result to a dashboard, and pages on regression. The nightly tier catches the slow leaks the merge tier cannot afford to.
Figure: Tiered CI for LLM evals: a smoke suite of 10-50 deterministic assertions runs on every push in under 2 minutes with a $5 budget cap, the full suite of 100-500 judged examples runs 4-way sharded on eval-labelled PRs under a hard EVAL_BUDGET_USD cap, and a nightly 06:00 UTC cron adds robustness, contamination, and drift checks. Only the full tier gates the merge, blocking when the quality delta interval excludes zero.
A working GitHub Actions setup
# .github/workflows/evals.yml
name: evals
on:
pull_request:
types: [opened, synchronize, reopened, labeled]
schedule:
- cron: "0 6 * * *" # 06:00 UTC nightly
jobs:
smoke:
if: github.event_name == 'pull_request'
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@v7
- uses: actions/setup-python@v7
with:
python-version: "3.12"
- run: pip install -e .[dev]
- run: pytest evals/smoke -q --maxfail=3
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
EVAL_BUDGET_USD: "5"
full:
if: contains(github.event.pull_request.labels.*.name, 'eval')
runs-on: ubuntu-latest
timeout-minutes: 20
strategy:
matrix:
shard: [0, 1, 2, 3]
steps:
- uses: actions/checkout@v7
- uses: actions/setup-python@v7
with:
python-version: "3.12"
- run: pip install -e .[dev]
- name: run shard
run: |
python -m evals.runner --suite full --shard ${{ matrix.shard }}/4 \
--judge-version "$(cat configs/judge.version)" \
--dataset-hash "$(cat datasets/full.hash)" \
--out shard-${{ matrix.shard }}.json
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
EVAL_BUDGET_USD: "15"
- uses: actions/upload-artifact@v7
with:
name: shard-${{ matrix.shard }}
path: shard-${{ matrix.shard }}.json
report:
needs: full
if: contains(github.event.pull_request.labels.*.name, 'eval')
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: actions/setup-python@v7
with:
python-version: "3.12"
- run: pip install -e .[dev]
- uses: actions/download-artifact@v8
with:
pattern: shard-*
merge-multiple: true
- run: python -m evals.report --merge shard-*.json --baseline main \
--pr ${{ github.event.pull_request.number }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
The example uses the current major of each action (checkout v7, setup-python v7, upload-artifact v7, download-artifact v8), all of which run on Node 24. GitHub made Node 24 the default runtime on runners on June 16, 2026 and scheduled Node 20 for removal on September 23, 2026 (fetched as of 2026-10-01) 4. Older pins such as checkout v4, setup-python v5, upload-artifact v4, and download-artifact v4 still declare Node 20 in their action.yml, so bump them in existing workflows. Without merge-multiple: true, download-artifact puts each shard in its own directory and the shard-*.json glob in the report step matches nothing.
The workflow runs on pull_request, so a PR from a fork gets no ANTHROPIC_API_KEY: GitHub passes only GITHUB_TOKEN to fork-triggered runs 5. The usual workaround is pull_request_target with the PR head checked out, which runs untrusted code next to your key. actions/checkout v7 (June 18, 2026) refuses that checkout by default, GitHub backported the same default to the older supported majors on July 20, 2026, and the opt-out input is allow-unsafe-pr-checkout 6. Do not set it on an eval job. Fork PRs do not run the paid tiers; a maintainer pushes the branch to the main repository when they should.
A pytest-style harness for the smoke tier keeps the failure surface familiar. Assertions can be deterministic (assert response.startswith("Refund of")) or judge-backed (assert score_faithfulness(...).label == "pass"); the runner caps cost via the EVAL_BUDGET_USD env and writes the verdict to a JSON report that the comment step reads.
# evals/smoke/test_refund_flow.py
import pytest
from evals.harness import run_workflow, judge
@pytest.mark.parametrize("case", load_cases("evals/cases/refund_smoke.jsonl"))
def test_refund_flow(case, judge_version):
out = run_workflow("refund", case["input"])
assert out["schema_valid"], out["raw"]
v = judge.faithfulness(out["response"], context=case["context"], version=judge_version)
assert v.label == "pass", v.rationale
Sharding and cost control
A four-shard full suite cuts wall time by roughly four with a small overhead and keeps any one shard's failure surface small enough to debug. The sharding key should be the dataset row, not the test file; rows shuffle deterministically by hash so reruns are reproducible. Each shard inherits the same judge version and dataset hash from configs/. Pin both per release.
Cost control is a hard cap at the runner level, not a soft warning. A EVAL_BUDGET_USD env that the runner enforces by stopping before the next call when the projected spend exceeds the cap is enough to keep a runaway loop from costing a five-figure surprise. The TypeScript version of the same harness is a Vitest runner with a custom reporter. Xata's write-up shows one that writes the response, a trace, and the judge verdict per case 7, and Vercel's intro covers the dataset, runner, and scorer split 8. Neither covers spend caps, so the cap goes in your runner.
Publishing the result to the PR
The comment is the user interface. A useful default: a four-column table per slice, baseline vs candidate, with the delta and a 95% interval, plus a one-line "merge / hold / block" verdict at the top. Block when any slice's quality delta excludes zero on the negative side; hold when cost or latency regressions exceed the configured thresholds; merge when the deltas are within the bands. Do not bury the result in a job log: the reviewer adjudicates the change in the PR thread, so the verdict has to be there.
The merge gate
Two opinions. First, gate on delta with intervals, not on absolute pass rate. A PR that takes a 92% baseline to 91% with overlapping intervals is not a regression; one that takes it to 88% with a delta interval excluding zero is. Second, no automatic overrides. A PR that fails the gate can ship after a human override with a comment explaining why; silent green is worse than loud red.
TIP
A CI eval that never fails is theatre. If your suite has not blocked a PR in the last month, the suite is too lenient, the data is stale, or the team is gaming it. Hamel's FAQ says the same about pass rates: a suite passing 100% is probably not challenging the system 3. The exception is a regression set held near 100% on purpose, which should stay quiet until something breaks. Walk traces; refresh the dataset; tighten the thresholds 9.
What goes in the nightly
The nightly tier is where the expensive checks live. A useful default:
- The full regression suite at the latest dataset version.
- A robustness sweep: each prompt run across five plausible formats and three random seeds.
- A contamination probe: the canary strings, a rephrased subset, and a private hold-out.
- A drift comparison against the previous nightly: input distribution, output distribution, per-eval score distribution.
Nightly pages on regression; the runbook is the same as the production drift one 10.
What to do this week
- Split your current suite into smoke (a few minutes) and full (about fifteen minutes). Move the deterministic checks to smoke; keep the judge-backed checks in full.
- Add an
EVAL_BUDGET_USD env to the runner and enforce it. Pin the judge version and dataset hash in the workflow.
- Wire the result to the PR comment. Block on delta intervals, not on absolute rates. Override requires a human comment.