AI Evals › Papers

Foundational eval theory

What an eval is, what a benchmark is, and why rankings flip when nothing changed. The papers that built the vocabulary.

Nine papers that explain why eval design is harder than it looks. MMLU and BIG-Bench are the canonical knowledge benchmarks. HELM is the canonical attempt to evaluate more than one axis at once. The Benchmark Lottery, Stop Uploading Test Data, MMLU-Redux, and Sclar et al. on format sensitivity together show why benchmark numbers move when the model does not. Miller's Adding Error Bars makes the statistical argument that an eval run is a sample, not a measurement.

AI Evals

The most useful, opinionated reference for AI evals.

AI Evals is a curated technical reference for evaluating large language model systems in production. The site covers error analysis, LLM-as-judge calibration, RAG evaluation, agentic eval, statistical rigor, and twenty task-specific eval playbooks. Every claim is cited.

Why evals matter
Your first eval in 30 minutes
LLM-as-Judge introduction
RAG eval in two layers
Trajectory eval for agents
Glossary
Papers