SDET and QA engineers ship the CI integration, the regression sets, the flakiness budgets, and the release gate. This track is test-engineering pragmatic: it ships pytest and Vitest patterns, a GitHub Actions workflow, a per-run cost cap, and a power table. It assumes test-engineering fluency and treats LLM evals as one more flaky-by-default test category to discipline.
The first three steps cover the architecture. Steps four through six are dataset and statistical discipline. The last four are operational (regression sets, platform choice, suite sizing, release gates).
The track
- Anatomy of an AI CI pipeline (12 min). The smoke / full / nightly tier pattern: what runs on every PR, what runs on merge, what runs at 2 AM, with a cost envelope per tier 1. Sharding, cost caps, and the GitHub Actions wiring that publishes the metric table to the PR. The Hamel and Shreya FAQ gives the rule for what earns a CI slot: keep the CI set small and purpose-built, favor deterministic assertions over judges there, and schedule every other eval by its cost, how saturated it is, and what a miss costs the business.
- Online vs offline: what CI can catch (12 min). Offline suites on frozen datasets gate merges; online evals sample live traffic after the fact. The handoffs between the two, and why CI alone never sees the traffic distribution.
- Making LLM judges deterministic enough (14 min). Position swap on pairwise, pinned rubric schemas, format-sensitivity checks, and the prompt-hash discipline. The pre-ship checklist that turns a judge from flaky to test-eligible.
- Golden set management (14 min). Content hashing, dataset versioning, the contract between PR author and eval. How to pin a result to a dataset version.
- Variance and flakiness budgets (16 min). Why naive standard errors are too small on clustered data. Wilson intervals on rates. When two points of drop is inside the noise band 2.
- Cost control: distilled judges (14 min). The cost math for swapping a frontier judge for a small fine-tuned one, what the swap silently costs, and the inline / calibration / gold architecture that keeps the cheap judge honest.
- Regression sets and tiering (8 min). The set whose job is to fail loudly when a fixed bug returns. What goes in, the capture workflow from production failures, and how the set tiers into smoke on every PR, full nightly, and adversarial probes weekly.
- Choosing the eval platform (10 min). Langfuse, Phoenix, Braintrust, LangSmith, or JSONL in git. The three decision axes, the source-of-truth principle, and what stays out of the vendor regardless of choice.
- How big does the eval set need to be (12 min). Power analysis for test engineers: pick the delta you care about, compute the N that detects it, and stop arguing about score changes the suite was never sized to see. Includes a power table to steal.
- Release gates from eval scores (15 min). The decision rule that gates promotion: bands, not one pass line. Ship, hold, investigate, or block, set by where the delta interval sits against a 2-point quality window (15% for p95 latency, 10% for cost), plus the override discipline.
What comes after the track
Once the ten pages are read, the natural next moves: write the eval CI workflow YAML; pin the smoke and full datasets in your repo; write the PR-comment formatter; pick a runner. Do not start on the hosted OpenAI Evals platform: it goes read-only on 2026-10-31 and shuts down on 2026-11-30, and OpenAI now points new users to Datasets 3. The open-source openai/evals repo is still public, but its README still points at that dashboard, so treat it as a registry of reference evals, not a CI dependency 4. A pytest harness over your JSONL is enough to start. Promptfoo is the declarative option, with YAML suites and a GitHub Action that runs a before-and-after comparison on every PR that touches a prompt 5; OpenAI agreed to acquire it in March 2026, and the maintainers say it stays open source 6. Inspect AI, built by the UK AI Security Institute and Meridian Labs, is the right secondary choice for research-grade trajectory evals that the CI hits weekly 7. The Applied LLMs essay is the right read on the discipline beyond the tooling 8.
TIP
The flakiness budget conversation (step 5) is the one where you build credibility with the rest of engineering. A team that can answer "is this score change real" with statistics has a different shipping cadence than a team that says "looks worse, retry."