LLM-as-Judge

Building, calibrating, and stress-testing model judges. From G-Eval and MT-Bench through Prometheus 2 to JudgeBench.

Eleven papers covering the arc from G-Eval to JudgeBench. G-Eval and MT-Bench established that a strong LLM can rival a human rater. Prometheus, Auto-J, and JudgeLM proved that open models can do the job too. Shankar et al. surfaced criteria drift, which is the reason most judge pipelines stop working after launch. Judging the Judges and JudgeBench close the loop with the uncomfortable finding that judges fail hardest on reasoning, coding, and math, which is where you most want a verifier.

AI Evals

The most useful, opinionated reference for AI evals.

AI Evals is a curated technical reference for evaluating large language model systems in production. The site covers error analysis, LLM-as-judge calibration, RAG evaluation, agentic eval, statistical rigor, and twenty task-specific eval playbooks. Every claim is cited.

Why evals matter
Your first eval in 30 minutes
LLM-as-Judge introduction
RAG eval in two layers
Trajectory eval for agents
Glossary
Papers