# AI Evals

> The most useful, opinionated reference for AI evals.

## About this resource

AI Evals is a curated technical reference on evaluating large language model systems in production. The site covers the methodology (error analysis, LLM-as-judge, statistical rigor), the tools (Langfuse, Phoenix, Promptfoo, Inspect, RAGAS), the canonical papers, and twenty-six task-specific eval playbooks. Every claim is cited. Content is licensed CC BY 4.0; please cite https://www.aievals.co when you reference material from here.

Last generated: 2026-09-07T23:17:26.730Z.
Author: Om Bharatiya (https://www.linkedin.com/in/ombharatiya/).

## Top entry points

- [Start Here for each persona](https://www.aievals.co/start): linear reading path per role.
- [Cookbook](https://www.aievals.co/cookbook): runnable worked examples in code.
- [The Glossary](https://www.aievals.co/glossary): definitions for every term used.
- [All papers](https://www.aievals.co/papers): ten themed reading lists.
- [All tools](https://www.aievals.co/tools): vendor and OSS positioning.
- [Free PDF downloads](https://www.aievals.co/downloads): every track and section as a PDF field guide; free account required.

## Most cited pages on this site

> Pages with the highest citation density. Useful for answer-engine grounding.

- [LLM-as-Judge](https://www.aievals.co/techniques/llm-as-judge) (14 citations, Techniques): The comprehensive reference: what an LLM judge is, when it works, the four-part prompt, calibration discipline, the three biases, distillation, where judges fail.
- [Rubric design](https://www.aievals.co/techniques/rubric-design) (14 citations, Techniques): Binary versus Likert, multi-dimensional decomposition, criteria drift, the case against 1-5 scales, the case for written critiques.
- [Evals without a gold dataset](https://www.aievals.co/learn/foundations/evals-without-a-gold-dataset) (13 citations, Foundations): Where to start with zero labeled data: which tasks deserve gold first, which start reference-free, and how judge calibration labels become your gold set.
- [Distilled judges](https://www.aievals.co/learn/llm-as-judge/distilled-judges) (12 citations, LLM-as-Judge): What a small fine-tuned judge buys you, what it silently costs, the seven models to know, and the arithmetic that justifies the swap.
- [Autonomous coding](https://www.aievals.co/task-types/autonomous-coding) (11 citations, Task-type playbooks): Given an issue and a repo, autonomously edit files until hidden tests pass. Execution-grounded scoring on SWE-bench Verified is the bar; agent scaffolding matters as much as the model.
- [Evaluating agents under adversarial content](https://www.aievals.co/learn/safety-and-red-team/evaluating-agents-under-attack) (11 citations, Safety & Red-Team): Indirect prompt injection and MCP tool poisoning are the failure modes agentic products miss. How to measure them: the utility-security frontier, AgentDojo, and the numbers vendors actually report.
- [Reference-based metrics](https://www.aievals.co/techniques/reference-based-metrics) (11 citations, Techniques): BLEU, ROUGE, METEOR, chrF, BERTScore, COMET, and exact match. What each correlates with on modern systems and when it lies.
- [Reward hacking detection](https://www.aievals.co/learn/agentic-evals/reward-hacking) (11 citations, Agentic Evals): Once an objective is optimized, it gets gamed. Reward hacking is empirically common; eval for it explicitly.
- [Computer and browser-use agents](https://www.aievals.co/task-types/computer-browser-use) (10 citations, Task-type playbooks): Agent drives a real OS or browser via screenshots and keyboard or mouse. OSWorld 2.0 and WebArena are the public benchmarks; step budgets and partial credit are the 2026 methodology.
- [Pass^k: the metric that catches inconsistent agents](https://www.aievals.co/learn/agentic-evals/pass-k-and-consistency) (10 citations, Agentic Evals): Pass@1 hides catastrophic inconsistency. Report pass^k or you do not know how often the agent will repeatedly succeed.
- [SWE-bench Verified walkthrough](https://www.aievals.co/cookbook/swe-bench-verified-walkthrough) (10 citations, Cookbook): Run a repo-stratified 25-instance SWE-bench Verified subset with mini-SWE-agent, verify patches with the official Docker harness, and read the report without over-claiming.
- [Synthetic query generation for golden sets](https://www.aievals.co/cookbook/synthetic-query-generation) (10 citations, Cookbook): Build a RAG eval set from your own corpus: seeded chunk sampling, multi-hop and unanswerable query synthesis, MinHash dedup, and a pinned manifest you can diff.
- [Why AI evals matter](https://www.aievals.co/learn/foundations/why-evals-matter) (10 citations, Foundations): Evals are the moat for AI products: teams that measure quality catch the failures users would find. The one-page case, with evidence and a starting plan.
- [Deep research agents](https://www.aievals.co/task-types/deep-research-agents) (9 citations, Task-type playbooks): How to evaluate deep research agents: RACE rubric scoring, FACT citation precision, BrowseComp and GAIA, judge pitfalls, and a day-1 starter.
- [Judge biases you must defuse](https://www.aievals.co/learn/llm-as-judge/biases-and-mitigations) (9 citations, LLM-as-Judge): Position, verbosity, self-preference, format, sycophancy. Each with the empirical evidence and the mitigation you bake into the harness.

## Learn

### Foundations

> Why evals matter, the maturity model, and what makes LLM evaluation hard.

- [Why AI evals matter](https://www.aievals.co/learn/foundations/why-evals-matter): Evals are the moat for AI products: teams that measure quality catch the failures users would find. The one-page case, with evidence and a starting plan.
- [The eval maturity model](https://www.aievals.co/learn/foundations/eval-maturity-model): Five stages from vibe-checking to a continuous quality program. Where are you?
- [AI development as scientific method](https://www.aievals.co/learn/foundations/scientific-method-for-ai): The observe, hypothesize, experiment, measure, iterate loop mapped onto an AI product team.
- [Why LLM evals are hard](https://www.aievals.co/learn/foundations/non-determinism-and-subjectivity): Non-determinism, subjectivity, prompt sensitivity, and the reasons classical ML eval frameworks do not transfer.
- [Eval vs monitoring vs guardrails](https://www.aievals.co/learn/foundations/eval-vs-monitoring-vs-guardrails): Three concepts that share traces but do different jobs, with the architectural placement of each in a request lifecycle.
- [Evals without a gold dataset](https://www.aievals.co/learn/foundations/evals-without-a-gold-dataset): Where to start with zero labeled data: which tasks deserve gold first, which start reference-free, and how judge calibration labels become your gold set.

### Error Analysis

> Open coding, dimensional sampling, and the 60-80% rule.

- [The 60-80% rule](https://www.aievals.co/learn/error-analysis/the-60-80-rule): Three failure modes usually account for most of your bugs. Find them, fix them, repeat.
- [Open coding for AI traces](https://www.aievals.co/learn/error-analysis/open-coding-workflow): Read traces, write notes, cluster notes into failure modes, prioritize fixes. The three-stage process and the LLM-assisted parts.
- [Dimensional sampling](https://www.aievals.co/learn/error-analysis/dimensional-sampling): Synthetic input construction across features, scenarios, and personas to cover the failure surface, not the demo path.
- [Failure-mode taxonomies](https://www.aievals.co/learn/error-analysis/failure-mode-taxonomies): Bottom-up coding beats top-down templates, and why generic labels like hallucination mislead.
- [Build your trace viewer in an afternoon](https://www.aievals.co/learn/error-analysis/the-data-viewer): The single highest-return tool for an eval program. Streamlit and Next.js variants, with the minimum feature set.
- [Case study: NurtureBoss](https://www.aievals.co/learn/error-analysis/case-study-nurtureboss): One team's path from 66 percent date errors to 5 percent in a single error-analysis cycle.

### LLM-as-Judge

> Judge prompt structure, calibration against humans, biases and mitigations.

- [LLM-as-Judge: the practical introduction](https://www.aievals.co/learn/llm-as-judge/introduction): What an LLM judge is, when to use one, when not to, and the three pitfalls that show up on every team's first attempt.
- [The four-part judge prompt](https://www.aievals.co/learn/llm-as-judge/judge-prompt-structure): A copyable template: role and definitions, criteria, few-shot from your train set, output format. Plus the runnable invocation.
- [Calibrating your judge against humans](https://www.aievals.co/learn/llm-as-judge/calibration-to-humans): TPR and TNR over agreement percent. Train, dev, test splits. The 100-example minimum. Iterate the prompt until both rates clear 0.90.
- [Judge biases you must defuse](https://www.aievals.co/learn/llm-as-judge/biases-and-mitigations): Position, verbosity, self-preference, format, sycophancy. Each with the empirical evidence and the mitigation you bake into the harness.
- [Pairwise vs direct scoring](https://www.aievals.co/learn/llm-as-judge/pairwise-vs-direct): When a side-by-side ranking beats a binary verdict, the swap discipline that makes pairwise honest, and the cost table that decides at ten candidates.
- [Distilled judges](https://www.aievals.co/learn/llm-as-judge/distilled-judges): What a small fine-tuned judge buys you, what it silently costs, the seven models to know, and the arithmetic that justifies the swap.
- [Where LLM judges fail](https://www.aievals.co/learn/llm-as-judge/judgebench-honest-take): JudgeBench: GPT-4o is roughly random on hard correctness judgments. The implications for your pipeline are concrete, not theoretical.

### Datasets & Golden Sets

> Construction, versioning, lineage, and dimensional coverage.

- [Constructing your eval dataset](https://www.aievals.co/learn/datasets/construction): How to build the first hundred labeled examples, then grow them past five hundred without losing signal.
- [Synthetic vs human-authored examples](https://www.aievals.co/learn/datasets/synthetic-vs-human): Where each shape of data buys signal, where each one leaks, and how to mix them without poisoning the set.
- [Versioning your eval set](https://www.aievals.co/learn/datasets/versioning-lineage): A content-hashed eval set pinned to every result is a contract with your past self.
- [Building regression sets](https://www.aievals.co/learn/datasets/regression-sets): The set whose job is to fail loudly the next time you reintroduce a bug you already fixed.
- [Dimensional coverage: features, scenarios, personas](https://www.aievals.co/learn/datasets/dimensional-coverage): A three-axis grid that tells you which cells of the failure surface your set actually probes.
- [Where to keep your eval datasets](https://www.aievals.co/learn/datasets/dataset-platforms): Langfuse, Phoenix, Braintrust, LangSmith, or a JSONL in git. The choice that actually matters.

### RAG Evals

> Two-layer evaluation: retrieval metrics plus generation faithfulness.

- [Eval RAG in two layers](https://www.aievals.co/learn/rag-evals/retrieval-vs-generation): Separate retrieval and generation or you cannot fix what breaks.
- [RAG retrieval metrics: which to use and when](https://www.aievals.co/learn/rag-evals/retrieval-metrics): Recall@K, Precision@K, MRR, NDCG and Hit-Rate compared: what each answers, the labels it needs, and when to reach for it.
- [Recall@K: measuring retrieval coverage in RAG](https://www.aievals.co/learn/rag-evals/recall-at-k): Recall@K is the fraction of a query's relevant chunks that land in the top K retrieved. It scores coverage and ignores rank.
- [Precision@K: signal purity in RAG retrieval](https://www.aievals.co/learn/rag-evals/precision-at-k): Precision@K is the fraction of the top K retrieved chunks that are relevant. It measures noise, and it is not Ragas Context Precision.
- [Mean Reciprocal Rank (MRR): time-to-first-relevant](https://www.aievals.co/learn/rag-evals/mrr): MRR is the average of 1 over the rank of the first relevant chunk. It rewards a hit near the top and ignores everything after it.
- [NDCG: graded-relevance ranking for RAG retrieval](https://www.aievals.co/learn/rag-evals/ndcg): NDCG rewards putting the most relevant chunks highest, normalized to [0,1] so scores compare across queries.
- [Retriever debugging: the elimination path for RAG recall](https://www.aievals.co/learn/rag-evals/retriever-debugging): When retrieval metrics drop, the cause is one findable failure: chunking, embedding fit, query mismatch, filters, or K. Here is the elimination path.
- [Context precision and recall](https://www.aievals.co/learn/rag-evals/context-precision-recall): The Ragas pair that tells you whether the retriever found the right chunks and whether they were the only chunks worth showing the model.
- [RAG generation metrics: faithfulness to completeness](https://www.aievals.co/learn/rag-evals/generation-metrics): Generation quality is a vector, not a scalar. Faithfulness, groundedness, correctness, answer relevancy, and completeness each catch a different failure.
- [Faithfulness and atomic facts](https://www.aievals.co/learn/rag-evals/generation-faithfulness): Decompose the answer into atomic claims, judge each claim binary, and stop measuring hallucination as a single yes/no.
- [Groundedness vs faithfulness: span-level attribution](https://www.aievals.co/learn/rag-evals/groundedness): Groundedness asks whether every claim in an answer traces to a specific span of retrieved evidence. Stricter than faithfulness.
- [Answer correctness, relevancy, and completeness](https://www.aievals.co/learn/rag-evals/answer-correctness): The three generation dimensions faithfulness misses: matching reality, addressing the question, and covering every part of it.
- [Synthetic query generation](https://www.aievals.co/learn/rag-evals/synthetic-query-generation): Bootstrap a labeled RAG eval set from your own documents, without inventing ground truth.
- [Long-context RAG and RULER](https://www.aievals.co/learn/rag-evals/long-context-rag): Why advertised context lengths exceed effective context lengths, and the benchmarks that expose the gap.
- [Non-English RAG eval](https://www.aievals.co/learn/rag-evals/non-english-rag): What to measure when your corpus and your users do not share a language with most of the literature.

### Agentic Evals

> Trajectory vs outcome, tool-call evaluation, reward hacking, pass^k.

- [Trajectory-level evaluation](https://www.aievals.co/learn/agentic-evals/trajectory-vs-outcome): Outcome-only scoring ranks agents in the wrong order. Score the trajectory and the result.
- [Tool-call evaluation: schema, arguments, sequence, state](https://www.aievals.co/learn/agentic-evals/tool-call-evaluation): The four-layer tool-call eval, the AST-vs-execution tradeoff, BFCL's category structure, and an error taxonomy worth stealing.
- [Sub-goal decomposition: plan graphs as eval objects](https://www.aievals.co/learn/agentic-evals/sub-goal-decomposition): Real agent tasks are graphs. Evaluating only the leaf hides 5-of-6 sub-goal progress. Sub-goal completion is the planning metric that surfaces it.
- [Pass^k: the metric that catches inconsistent agents](https://www.aievals.co/learn/agentic-evals/pass-k-and-consistency): Pass@1 hides catastrophic inconsistency. Report pass^k or you do not know how often the agent will repeatedly succeed.
- [Agent-as-a-judge: the trajectory-scoring pattern](https://www.aievals.co/learn/agentic-evals/agent-as-judge): Single-LLM judges lose the thread on long trajectories. An agent judge with its own tools can walk the steps and grade them independently.
- [Reward hacking detection](https://www.aievals.co/learn/agentic-evals/reward-hacking): Once an objective is optimized, it gets gamed. Reward hacking is empirically common; eval for it explicitly.
- [Environmental drift: robustness to tool and API change](https://www.aievals.co/learn/agentic-evals/environmental-drift): Most agent benchmarks freeze the environment. Production does not. Inject latency, schema shifts, and transient errors. Score the recovery, not the happy path.
- [Regression suites for agent loops](https://www.aievals.co/learn/agentic-evals/regression-suites): Trajectory regression is harder than answer regression. Replay 50 prod trajectories on every change and diff outcome, step count, and tool-call sequence.

### Production & Observability

> Online vs offline, trace schema, drift detection, CI integration.

- [Online vs offline: where each fires](https://www.aievals.co/learn/production/online-vs-offline): Offline evals run on a frozen dataset for every change. Online evals sample real traffic continuously. They answer different questions and both belong in production.
- [Trace schema: what to log](https://www.aievals.co/learn/production/trace-schema): The trace is the unit of analysis for every eval program. This is the schema, the fields that matter, and the storage shape that makes downstream queries cheap.
- [Sampling strategies for production traces](https://www.aievals.co/learn/production/sampling-strategies): Uniform sampling is the default and the wrong default for tail slices. Stratify on the dimensions you care about, oversample failures, and reweight.
- [Drift detection for production AI](https://www.aievals.co/learn/production/drift-detection): Track input and output distributions over time. Page on changes that matter; ignore the ones that do not. The hardest part is deciding which is which.
- [Cost and latency belong in the scorecard](https://www.aievals.co/learn/production/cost-latency-eval): A model that wins quality and triples p95 latency is not a win. Put quality, cost, and latency on one scorecard and pick on the Pareto frontier.
- [Wiring evals into CI](https://www.aievals.co/learn/production/ci-integration): Tier the suite into smoke, full, and nightly. Cap cost. Shard. Publish to the PR. Make the merge gate boring and the merge decision data-driven.
- [Release gates from eval results](https://www.aievals.co/learn/production/release-gates): A release gate is a contract between the team and the user. Define the bands, automate the comparison, and hold the line when a regression candidate wants out.
- [Incident response for AI products](https://www.aievals.co/learn/production/incident-response): An AI postmortem looks different from a service postmortem. The root cause is rarely a line of code. The actions are usually eval-set changes.

### Statistics & Rigor

> Confidence intervals, bootstrap, power, inter-rater agreement.

- [Confidence intervals: Wilson and Clopper-Pearson](https://www.aievals.co/learn/statistics/confidence-intervals): Wald is wrong. Wilson is the practical default. Clopper-Pearson is the conservative exact. With code and the case for why N=10 evals tell you almost nothing.
- [Bootstrap and paired tests for noisy eval metrics](https://www.aievals.co/learn/statistics/bootstrap-and-paired-tests): When the metric is not a clean Bernoulli, bootstrap. When the systems share inputs, pair. Both close the gap between point estimates and actionable decisions.
- [Statistical power: how big does the eval set need to be](https://www.aievals.co/learn/statistics/statistical-power): Power is the probability of detecting the delta you care about. Pick the delta first, derive the N, then commit. Reversing the order produces evals nobody can act on.
- [Clustered standard errors: when CIs lie](https://www.aievals.co/learn/statistics/clustered-standard-errors): When eval questions cluster (multiple questions per topic, multiple turns per session), naive standard errors are three times too small. The cluster-aware variance formula and code, with the...
- [Inter-rater agreement: Cohen's kappa and Krippendorff's alpha](https://www.aievals.co/learn/statistics/inter-rater-agreement): Two qualified humans disagree on your rubric. That is information about the rubric, not about the raters. Measure agreement before you trust labels.
- [Multiple comparisons: Bonferroni vs BH-FDR](https://www.aievals.co/learn/statistics/multiple-comparisons): Run twenty tests at alpha 0.05 and you expect one false positive. Correct for it. Bonferroni for strict family-wise control; Benjamini-Hochberg for false-discovery control.
- [Effect size: Cohen's d and absolute deltas](https://www.aievals.co/learn/statistics/effect-size): A statistically significant 0.3-point win on a 0-100 rubric is not worth shipping. Report the magnitude, not just the p-value.

### Safety & Red-Team

> OWASP LLM Top 10, HarmBench, AILuminate, jailbreaks and defenses.

- [Walking the OWASP Top 10 for LLM apps](https://www.aievals.co/learn/safety-and-red-team/owasp-llm-top-10): The ten failure categories every LLM application owner should be probing, with the smallest useful test for each.
- [Designing a red-team program](https://www.aievals.co/learn/safety-and-red-team/red-team-program-design): The operating model for an LLM red team: frequency, staffing, intake, scoring, and where it sits relative to automated regression.
- [HarmBench in practice](https://www.aievals.co/learn/safety-and-red-team/harmbench-walkthrough): A code walk through the HarmBench corpus, the StrongREJECT scoring classifier, and how to read the leaderboard without overfitting to it.
- [AILuminate in practice](https://www.aievals.co/learn/safety-and-red-team/ailuminate-walkthrough): MLCommons' standardized safety benchmark, the twelve hazard categories, and how to read its grade scale alongside HarmBench.
- [Over-refusal and the balance question](https://www.aievals.co/learn/safety-and-red-team/over-refusal-and-balance): Why refusal rate alone is a misleading metric, the XSTest pattern for paired benign cases, and how to grade refusal against necessity.
- [Jailbreaks and defenses](https://www.aievals.co/learn/safety-and-red-team/jailbreaks-and-defenses): GCG-style adversarial suffixes, the empirical floor on defense success, and what your guardrail stack should actually do.
- [Evaluating agents under adversarial content](https://www.aievals.co/learn/safety-and-red-team/evaluating-agents-under-attack): Indirect prompt injection and MCP tool poisoning are the failure modes agentic products miss. How to measure them: the utility-security frontier, AgentDojo, and the numbers vendors actually report.

### Responsible AI

> Fairness metrics, Fairlearn and AIF360, SHAP and Integrated Gradients, bias probes, data integrity.

- [Fairness metrics you can defend](https://www.aievals.co/learn/responsible-ai/fairness-metrics): Demographic parity, equalized odds, equal opportunity, predictive parity, and calibration: what each one measures, the impossibility results, and how to choose by harm type.
- [Fairlearn in practice](https://www.aievals.co/learn/responsible-ai/fairlearn-in-practice): MetricFrame disaggregation, demographic parity and equalized odds gap metrics, and mitigation with ThresholdOptimizer and reductions, in scikit-learn idiom.
- [AIF360 in practice](https://www.aievals.co/learn/responsible-ai/aif360-in-practice): IBM's AI Fairness 360: the metric zoo, the dataset classes that trip people up, pre-, in-, and post-processing mitigation, and when to choose it over Fairlearn.
- [SHAP and LIME](https://www.aievals.co/learn/responsible-ai/shap-and-lime): Local surrogates versus Shapley values: which explainer fits which model, the pitfalls that invalidate the pictures, and using attributions to catch leakage and spurious features.
- [Integrated Gradients and Captum](https://www.aievals.co/learn/responsible-ai/gradient-attributions): Gradient attributions for deep models: the Integrated Gradients axioms, baseline choice, convergence checks, Captum's API for text models, and the sanity checks saliency methods fail.
- [Bias and toxicity probes for LLMs](https://www.aievals.co/learn/responsible-ai/llm-bias-probes): BBQ, BOLD, and RealToxicityPrompts as public anchors, counterfactual probes built from your own traces, toxicity scoring, and how to report bias numbers for generative models.
- [Data integrity checks](https://www.aievals.co/learn/responsible-ai/data-integrity-checks): Provenance and datasheets, label leakage, split hygiene, contamination, poisoning, and PII scanning: the checklist that keeps eval and training data trustworthy.

### Governance, Risk, Compliance

> EU AI Act, NIST AI RMF, RSPs, risk registers, trust artifacts.

- [EU AI Act cheatsheet](https://www.aievals.co/learn/governance/eu-ai-act-cheatsheet): Deadlines, scope splits, and the eval obligations the EU AI Act actually puts on you. The CTO-grade summary.
- [NIST AI RMF mapped to eval activities](https://www.aievals.co/learn/governance/nist-ai-rmf-mapped): Govern, Map, Measure, Manage as a cross-walk to the eval artifacts your team already produces or needs to build.
- [Anthropic Responsible Scaling Policy](https://www.aievals.co/learn/governance/anthropic-rsp): The capability-threshold logic, the AI Safety Level scheme, and what an internal team can copy from the public document.
- [OpenAI Preparedness Framework](https://www.aievals.co/learn/governance/openai-preparedness): The tracked risk categories, the production-readiness gates, and where the Preparedness Framework differs from Anthropic's RSP.
- [Building an AI risk register](https://www.aievals.co/learn/governance/ai-risk-register): A practical template: columns, scoring scale, review cadence, and the mapping from register row to eval task.
- [Customer trust artifacts](https://www.aievals.co/learn/governance/customer-trust-artifacts): Model cards, system cards, public eval pages, and the trust portal pattern that closes procurement conversations.
- [Board readout templates](https://www.aievals.co/learn/governance/board-readout-templates): A 12-minute quarterly board update on AI risk, with the three slides that hold up and the slide that does not.

## Techniques

> Methodology reference: deeper than the Learn chapters, organized by technique.

- [Agent trajectory evaluation](https://www.aievals.co/techniques/agent-trajectory-eval): Step-level scoring beats outcome-only. Tool-call F1, plan analysis, reward-hacking probes, environmental drift. The eval pattern that catches partial failures.
- [Contamination detection](https://www.aievals.co/techniques/contamination-detection): N-gram matches, canary strings, perplexity tests, MMLU-CF rephrase patterns, held-out sets. How to tell whether a benchmark score reflects training-set memorization.
- [Embedding similarity for evals](https://www.aievals.co/techniques/embedding-similarity): Cosine over sentence embeddings as an eval signal. When it works (retrieval, dedup, semantic regression). When it fails (open-ended quality, ranking systems).
- [Error analysis (open coding)](https://www.aievals.co/techniques/error-analysis): Open, axial, structured coding for AI traces. The 60-80 percent rule, sample-size saturation, LLM-assisted clustering, the bottom-up failure taxonomy.
- [Golden dataset methodology](https://www.aievals.co/techniques/golden-datasets): Features times scenarios times personas, synthetic plus production, stop rules, versioning, the case for content-hashed eval sets.
- [Guardrails versus evals architecture](https://www.aievals.co/techniques/guardrails-vs-evals): Two concepts that share data and differ in latency, intent, and consequence. Where each fires in the request path, how they share traces, the mistake of building one and skipping the other.
- [LLM-as-Judge](https://www.aievals.co/techniques/llm-as-judge): The comprehensive reference: what an LLM judge is, when it works, the four-part prompt, calibration discipline, the three biases, distillation, where judges fail.
- [Online evaluation patterns](https://www.aievals.co/techniques/online-evaluation): Trace sampling, sampled judging at production, drift detection, A/B with intervals. The four patterns that turn an offline eval program into a production quality program.
- [Pairwise comparison and Elo](https://www.aievals.co/techniques/pairwise-and-elo): Bradley-Terry, Elo, and Chatbot Arena methodology. When pairwise beats absolute scoring, how to control position bias, how to read a leaderboard with intervals.
- [Reference-based metrics](https://www.aievals.co/techniques/reference-based-metrics): BLEU, ROUGE, METEOR, chrF, BERTScore, COMET, and exact match. What each correlates with on modern systems and when it lies.
- [Robustness testing](https://www.aievals.co/techniques/robustness-testing): Format sweeps, order sweeps, temperature sweeps, CheckList invariance and directional tests. How to surface the variance hiding behind a single accuracy number.
- [Rubric design](https://www.aievals.co/techniques/rubric-design): Binary versus Likert, multi-dimensional decomposition, criteria drift, the case against 1-5 scales, the case for written critiques.
- [Statistical methods](https://www.aievals.co/techniques/statistical-methods): Wilson, Clopper-Pearson, bootstrap, paired tests, clustered standard errors, inter-rater agreement. The minimum statistics an eval program needs.

## Tools

> Eval platforms and libraries with positioning, license, and trade-offs.

- [Arize Phoenix](https://www.aievals.co/tools/arize-phoenix): OpenTelemetry-native open-source tracing and evaluation, plus the Arize AX enterprise platform on top.
- [Braintrust](https://www.aievals.co/tools/braintrust): Framework-agnostic hosted AI observability and eval platform built around the Brainstore trace search engine.
- [Comet Opik](https://www.aievals.co/tools/comet-opik): Apache 2.0 LLM evaluation and observability with 30+ judge metrics, a prompt optimizer, and an in-IDE coding assistant.
- [Confident AI / DeepEval](https://www.aievals.co/tools/confident-ai-deepeval): Pytest-style LLM unit testing in DeepEval plus the Confident AI hosted layer for dataset management and dashboards.
- [Galileo](https://www.aievals.co/tools/galileo): Eval-to-guardrail platform: the Luna distilled-judge models grade pre-production and become production guardrails.
- [Giskard](https://www.aievals.co/tools/giskard): AI red-team and agent-vulnerability scanner with 50+ adversarial attack types and SOC 2 / GDPR / HIPAA reporting.
- [HELM](https://www.aievals.co/tools/helm): Stanford CRFM's Holistic Evaluation of Language Models: a maintained scenario-and-metric matrix with public leaderboards.
- [HoneyHive](https://www.aievals.co/tools/honeyhive): OpenTelemetry-native LLM observability and evals with a 'virtual data planes' deployment model for enterprise data residency.
- [Inspect AI](https://www.aievals.co/tools/inspect-ai): UK AISI's open-source eval framework, reviewed: 200+ prebuilt evals, real sandboxing, multi-agent support, and where it beats the hosted platforms.
- [Langfuse](https://www.aievals.co/tools/langfuse): Open-source LLM engineering platform: tracing, prompts, datasets, scores, and a genuinely self-hostable cloud.
- [LangSmith](https://www.aievals.co/tools/langsmith): LangChain's hosted observability and evaluation platform. The default for teams already on LangChain or LangGraph.
- [LlamaIndex evals](https://www.aievals.co/tools/llamaindex-evals): First-party RAG evaluation modules inside LlamaIndex, with integrations into UpTrain, DeepEval, Ragas, and Tonic.
- [lm-evaluation-harness](https://www.aievals.co/tools/lm-evaluation-harness): EleutherAI's de facto academic harness for raw model benchmarks; backend for the HuggingFace Open LLM Leaderboard.
- [OpenAI Evals](https://www.aievals.co/tools/openai-evals): The canonical OpenAI evaluation framework: model regression for OpenAI models, now also wired into the OpenAI Dashboard.
- [Promptfoo](https://www.aievals.co/tools/promptfoo): MIT-licensed CLI and library for eval and red-team; acquired by OpenAI with the OSS license preserved.
- [Ragas](https://www.aievals.co/tools/ragas): The RAG-eval metrics standard: faithfulness, context precision, context recall, answer relevancy, and a synthetic test-set generator.
- [TruLens](https://www.aievals.co/tools/trulens): OpenTelemetry-emitting feedback functions for RAG and agent evals, stewarded by Snowflake.
- [Weights and Biases Weave](https://www.aievals.co/tools/w-and-b-weave): LLM evals and monitoring inside the W&B Models stack: trace trees, leaderboards, online eval, and multimodal tracking.

## Papers

> Curated reading list across ten themes, each paper annotated.

- [Agentic eval](https://www.aievals.co/papers/agentic): Trajectory, outcome, and the harder benchmarks that came after AgentBench. WebArena, SWE-bench, GAIA, OSWorld, tau-bench, MLE-bench, AgentDojo, SWE-Lancer, BrowseComp, Cybench.
- [Foundational eval theory](https://www.aievals.co/papers/foundational): What an eval is, what a benchmark is, and why rankings flip when nothing changed. The papers that built the vocabulary.
- [LLM-as-Judge](https://www.aievals.co/papers/llm-as-judge): Building, calibrating, and stress-testing model judges. From G-Eval and MT-Bench through Prometheus 2 to JudgeBench.
- [Long-context](https://www.aievals.co/papers/long-context): Why '1M tokens' rarely means a million tokens. Lost in the Middle, LongBench, RULER, BABILong, NoCha, Loong, NoLiMa.
- [Methodology and production](https://www.aievals.co/papers/methodology-production): SPADE, RewardBench, MixEval, WildBench, IFEval, InfoBench, FollowBench, plus the 2025 agent-eval and observability work.
- [Multimodal](https://www.aievals.co/papers/multimodal): MMBench, MM-Vet, MMMU, BLINK, Vibe-Eval, Video-MME, MMMU-Pro, ChartQA.
- [RAG, retrieval, factuality](https://www.aievals.co/papers/rag-factuality): BEIR, FActScore, RAGAS, Self-RAG, ARES, SAFE, SimpleQA, HalluLens.
- [Reasoning and math](https://www.aievals.co/papers/reasoning-and-math): From MATH and Chain-of-Thought to FrontierMath, ARC-AGI-2, Humanity's Last Exam, and the faithfulness problem.
- [Safety, red-team, robustness](https://www.aievals.co/papers/safety-red-team): TruthfulQA, BBQ, Constitutional AI, GCG, sycophancy, HarmBench, sandbagging, alignment faking.
- [Tool use and function calling](https://www.aievals.co/papers/tool-use): API-Bank, Gorilla, ToolBench, BFCL, TRAJECT-Bench, plus the tool-injection security work in AgentDojo.

## Cookbook

> Runnable worked examples. Estimated minutes per recipe.

- [Your first eval in 30 minutes](https://www.aievals.co/cookbook/your-first-eval-30-minutes): Pick one feature, label fifty traces, write a binary judge, run it, read the confusion matrix. A complete walkthrough using the Anthropic Python SDK.
- [LLM-judge calibration loop](https://www.aievals.co/cookbook/llm-judge-calibration): Push TPR and TNR past 0.90 by iterating the rubric, not the model. A complete loop adapted from the HuggingFace LLM-as-judge cookbook.
- [RAG eval with Ragas](https://www.aievals.co/cookbook/rag-eval-with-ragas): A two-layer RAG eval (retrieval plus generation faithfulness) using Ragas on a small synthetic corpus. Runs locally with the OpenAI Python SDK.
- [Agent eval with τ-bench: pass@1 vs pass^4](https://www.aievals.co/cookbook/agent-eval-with-tau-bench): Run τ-bench (tau-bench) retail tasks against your agent and report pass@1 versus pass^4: the consistency gap between a demo and a product.
- [Multi-turn conversation eval](https://www.aievals.co/cookbook/multi-turn-conversation-eval): Per-turn binary rubrics plus a per-session state check, logged to Langfuse, for an assistant that has to remember context and follow rules across turns.
- [SWE-bench Verified walkthrough](https://www.aievals.co/cookbook/swe-bench-verified-walkthrough): Run a repo-stratified 25-instance SWE-bench Verified subset with mini-SWE-agent, verify patches with the official Docker harness, and read the report without over-claiming.
- [Measure effective context length with RULER](https://www.aievals.co/cookbook/long-context-eval-with-ruler): A 90-minute harness that finds the context length your model actually sustains: RULER's tasks as a six-task smoke suite, run from 4k to 128k tokens.
- [Text-to-SQL eval with execution](https://www.aievals.co/cookbook/text-to-sql-eval-with-execution): Score SQL by executing it against a seeded SQLite warehouse and comparing result sets, with a failure taxonomy for syntax, schema, shape, and value errors.
- [Online eval with Langfuse](https://www.aievals.co/cookbook/online-eval-with-langfuse): Instrument an app with Langfuse, judge a 5% sample of production traces plus every thumbs-down, and promote confirmed failures into the offline dataset.
- [Online eval with Phoenix](https://www.aievals.co/cookbook/online-eval-with-phoenix): Trace a live app into self-hosted Phoenix over OpenTelemetry, judge sampled spans with the evals classifiers, and bank the failures as a regression dataset.
- [TypeScript eval with Vercel AI SDK](https://www.aievals.co/cookbook/typescript-eval-with-vercel-ai-sdk): Vitest as the eval runner, AI SDK 6 generateText under test, a zod-typed binary judge, a JSON case file, and CI wiring with per-run cost accounting.
- [Adding error bars to your evals](https://www.aievals.co/cookbook/adding-error-bars): Wilson intervals on pass rates, cluster-aware standard errors, and a paired McNemar test for two-system comparisons, with runnable statsmodels code.
- [Contamination detection for code-gen and reasoning evals](https://www.aievals.co/cookbook/contamination-detection): Date-window your test set against each model's training cutoff, then catch verbatim and paraphrased leaks with MinHash overlap against public mirrors.
- [HarmBench red-team run](https://www.aievals.co/cookbook/harmbench-red-team): Run 100 HarmBench behaviors and all 450 XSTest prompts against your model, then report attack success rate and over-refusal rate as one table.
- [Synthetic query generation for golden sets](https://www.aievals.co/cookbook/synthetic-query-generation): Build a RAG eval set from your own corpus: seeded chunk sampling, multi-hop and unanswerable query synthesis, MinHash dedup, and a pinned manifest you can diff.
- [Fairness audit with Fairlearn](https://www.aievals.co/cookbook/fairness-audit-with-fairlearn): A runnable audit on the adult income dataset: disaggregate with MetricFrame, read the gaps with group sizes, mitigate with ThresholdOptimizer, and report before and after honestly.
- [Model explanations with SHAP](https://www.aievals.co/cookbook/model-explanations-with-shap): Train a classifier with a planted leaky feature, catch it with TreeExplainer global importance, read local explanations for individual errors, and port the workflow to Captum.
- [Agent injection eval with AgentDojo](https://www.aievals.co/cookbook/agent-injection-eval-with-agentdojo): Measure whether a tool-using agent resists indirect prompt injection: run AgentDojo's banking suite clean and under attack, compute utility and attack success rate together, then add a defense and...

## Task types

> Twenty-six task-specific eval playbooks, one per task shape.

- [Autonomous coding](https://www.aievals.co/task-types/autonomous-coding): Given an issue and a repo, autonomously edit files until hidden tests pass. Execution-grounded scoring on SWE-bench Verified is the bar; agent scaffolding matters as much as the model.
- [Long-horizon planning and agentic loops](https://www.aievals.co/task-types/long-horizon-planning): Multi-step decomposition, replanning, recovery. The agent works for minutes. Pass^k on real workflows is the metric that surfaces consistency, which pass@1 alone hides.
- [Reasoning depth and thinking-mode analysis](https://www.aievals.co/task-types/reasoning-depth): Evaluate the thinking traces of reasoning models. Correctness on hard reasoning plus chain-of-thought faithfulness, not just answer accuracy.
- [Long-context RAG](https://www.aievals.co/task-types/long-context-rag): Answer over very long contexts: codebases, contract bundles, transcripts. RULER plus answer faithfulness, not vanilla needle-in-a-haystack.
- [Native multimodal](https://www.aievals.co/task-types/native-multimodal): Inputs cross modalities: image plus text Q and A, voice to text to image, screen plus instruction. Task accuracy plus cross-modal grounding, not single-modality fallback.
- [Tool use and function calling](https://www.aievals.co/task-types/tool-use-function-calling): Choose the right function, fill correct arguments, handle errors, chain calls. AST-match plus execution-grounded success on BFCL v4 is the bar.
- [Computer and browser-use agents](https://www.aievals.co/task-types/computer-browser-use): Agent drives a real OS or browser via screenshots and keyboard or mouse. OSWorld 2.0 and WebArena are the public benchmarks; step budgets and partial credit are the 2026 methodology.
- [Structured output and JSON mode](https://www.aievals.co/task-types/structured-output-json): Emit valid JSON conforming to a schema with correct types and values. Schema validity times semantic correctness, not validity alone.
- [Text-to-SQL](https://www.aievals.co/task-types/text-to-sql): Natural-language question to SQL query to correct rows. Execution accuracy on BIRD-bench plus efficiency, not exact-match scoring.
- [Conversational quality](https://www.aievals.co/task-types/conversational-quality): Multi-turn assistant. Stay on persona, remember context, recover from misunderstandings. Pairwise preference plus per-turn rubric, not 1-to-5 scales.
- [Summarization](https://www.aievals.co/task-types/summarization): Compress source documents into a faithful summary. Faithfulness plus coverage, not ROUGE.
- [Translation](https://www.aievals.co/task-types/translation): Source language to target language. COMET and chrF for automatic scoring, MQM or ESA for human; not BLEU alone in 2026.
- [Code review](https://www.aievals.co/task-types/code-review): Model reads a diff and flags real issues with actionable comments. Precision plus recall of seeded bugs, not recall alone (which produces nit machines).
- [Document understanding and extraction](https://www.aievals.co/task-types/document-understanding): Extract structured info or answer questions from documents: PDFs, scans, forms. Field-level F1 and ANLS, not holistic JSON-looks-right scoring.
- [Voice. STT, TTS, conversational](https://www.aievals.co/task-types/voice): STT (audio to text), TTS (text to audio), and full-duplex conversational voice. WER, MOS, TTFB; not WER on clean audio alone.
- [Image generation](https://www.aievals.co/task-types/image-generation): Text prompt to image. Human preference plus CLIP score plus FID; not FID alone (correlates poorly with text-image alignment).
- [Embeddings and retrieval quality](https://www.aievals.co/task-types/embeddings-retrieval): Embed queries and docs, retrieve top-K relevant. NDCG@10 plus Recall@K on a domain set; MTEB rankings don't predict domain ranking.
- [Recommendation and personalization](https://www.aievals.co/task-types/recommendation): Rank items for a user. Offline NDCG plus hit-rate to filter; online A/B for ground truth; temporal splits, never random.
- [Classification](https://www.aievals.co/task-types/classification): Assign one or more labels per input. Macro-F1 plus calibration on a stratified set, not accuracy on imbalanced data.
- [Safety, refusal, and jailbreak resistance](https://www.aievals.co/task-types/safety-jailbreak): Refuse harmful requests; resist adversarial prompts; don't over-refuse. Attack success rate plus over-refusal rate, never one alone.
- [Deep research agents](https://www.aievals.co/task-types/deep-research-agents): How to evaluate deep research agents: RACE rubric scoring, FACT citation precision, BrowseComp and GAIA, judge pitfalls, and a day-1 starter.
- [Spec generation agents](https://www.aievals.co/task-types/spec-generation-agents): Evaluating spec and design-doc generation: SpecBench, ambiguity detection, EARS acceptance criteria, and spec2code downstream success.
- [GraphRAG](https://www.aievals.co/task-types/graphrag): Evaluating GraphRAG: local vs global query classes, counterbalanced pairwise judging, entity-coverage ceilings, and token-cost accounting.
- [Multi-agent systems](https://www.aievals.co/task-types/multi-agent-systems): Evaluating multi-agent LLM systems: milestone KPIs, the MAST 14-failure-mode taxonomy, orchestrator vs worker splits, and cost explosion.
- [MCP and tool-registry evals](https://www.aievals.co/task-types/mcp-tool-evals): Evaluating agents on MCP servers: MCP-Universe, tool-routing failures at registry scale, schema-drift CI checks, and AgentDojo injection probes.
- [Agent memory](https://www.aievals.co/task-types/agent-memory): Evaluating agent memory: the five LongMemEval abilities, contradiction and deletion probes, pass^k reliability, and the missing privacy benchmark.

## Start Here

> Linear numbered reading paths, one track per persona.

- [Start Here for Backend Engineers](https://www.aievals.co/start/backend-engineer): Ten steps from why evals matter through release gates. Trace logging from services, structured output, tool calls, execution-checked SQL, online eval, CI wiring, latency and cost budgets.
- [Start Here for CTOs and Directors of Technology](https://www.aievals.co/start/cto): Ten steps from the executive case for evals through a board readout. Risk posture, regulatory mapping, frontier-lab benchmarks.
- [Start Here for Senior Data Engineers](https://www.aievals.co/start/data-engineer): Nine steps from trace schema through drift detection. JSONL and Parquet schemas, sampling policies, dataset versioning and lineage.
- [Start Here for Senior Engineering Managers](https://www.aievals.co/start/eng-manager): Eight steps from the maturity model through the skill bar for eval-literate engineers. Strategy first, tables and decision flowcharts, light on code.
- [Start Here for Frontend Engineers](https://www.aievals.co/start/frontend-engineer): Nine steps from why evals matter through online monitoring. TypeScript-first tooling, judge basics, a trace viewer you build, multi-turn and voice quality, client-side instrumentation.
- [AI evals for engineers: the 7-step starting path](https://www.aievals.co/start/principal-ai-engineer): Go from zero to a running eval suite in seven readings: your first eval in 30 minutes, judge calibration, RAG metrics, and a tool-using agent eval.
- [Start Here for Senior Product Managers](https://www.aievals.co/start/product-manager): Nine steps from the AI quality bar through a foundational reading list. Product-thinking, customer-friendly vocabulary, light on math.
- [Start Here for AI Research Engineers and Research Scientists](https://www.aievals.co/start/research-engineer): Nine steps from the agent-eval survey through publishing eval datasets responsibly. Citation-heavy, ablation tables, reproducibility checklists.
- [Start Here for SDET and QA Engineers](https://www.aievals.co/start/sdet-qa): Ten steps from anatomy of an AI CI pipeline to release gates. Test-engineering pragmatic, config snippets, pytest and Vitest patterns.

## Canonical citations

> Primary sources cited across the site. Grouped by kind so answer engines can match a citation to a content shape.

### Practitioner blog posts

> Long-form posts by working AI practitioners (Hamel Husain, Eugene Yan, Chip Huyen, Shreya Shankar, et al). The opinions backing most of our recommendations.

- [Your AI Product Needs Evals by Hamel Husain](https://hamel.dev/blog/posts/evals/)
- [Creating an LLM-as-a-Judge That Drives Business Results by Hamel Husain](https://hamel.dev/blog/posts/llm-judge/)
- [A Field Guide to Rapidly Improving AI Products by Hamel Husain](https://hamel.dev/blog/posts/field-guide/)
- [LLM Evals FAQ by Hamel Husain & Shreya Shankar](https://hamel.dev/blog/posts/evals-faq/)
- [The Revenge of the Data Scientist by Hamel Husain](https://hamel.dev/blog/posts/revenge/)
- [Fuck You, Show Me The Prompt by Hamel Husain](https://hamel.dev/blog/posts/prompt/)
- [Patterns for Building LLM-based Systems & Products by Eugene Yan](https://eugeneyan.com/writing/llm-patterns/)
- [LLM Evaluators by Eugene Yan](https://eugeneyan.com/writing/llm-evaluators/)
- [Evaluation & Hallucination Detection for Abstractive Summaries by Eugene Yan](https://eugeneyan.com/writing/evals/)
- [Abstractive Summary eval by Eugene Yan](https://eugeneyan.com/writing/abstractive/)
- [OOD Finetuning to Bootstrap Hallucination Detection by Eugene Yan](https://eugeneyan.com/writing/finetuning/)
- [The AI Engineering Flywheel by Shreya Shankar](https://www.sh-reya.com/blog/ai-engineering-flywheel/)
- [Building a Generative AI Platform by Chip Huyen](https://huyenchip.com/2024/07/25/genai-platform.html)
- [900 Most Popular Open Source AI Tools by Chip Huyen](https://huyenchip.com/2024/03/14/ai-oss.html)
- [Challenges in Evaluating AI Systems by Anthropic](https://www.anthropic.com/news/evaluating-ai-systems)
- [Introducing Computer Use by Anthropic](https://www.anthropic.com/news/3-5-models-and-computer-use)
- [Chatbot Arena Benchmarking Blog by LMSYS](https://lmsys.org/blog/2023-05-03-arena/)
- [Chatbot Arena BT analysis by LMSYS](https://lmsys.org/blog/2023-12-07-leaderboard/)
- [Aligning LLM-as-Judge w/ Human Preferences by LangChain](https://www.langchain.com/blog/aligning-llm-as-a-judge-with-human-preferences)

### Papers

> Academic and lab-published papers on eval methodology, benchmarks, and judge calibration.

- [MMLU (2020) by Hendrycks et al](https://arxiv.org/abs/2009.03300)
- [Beyond Accuracy: CheckList (2020) by Ribeiro et al](https://arxiv.org/abs/2005.04118)
- [BIG-Bench (2022) by Srivastava et al](https://arxiv.org/abs/2206.04615)
- [BBH (2022) by Suzgun et al](https://arxiv.org/abs/2210.09261)
- [HELM (2022) by Liang et al](https://arxiv.org/abs/2211.09110)
- [The Benchmark Lottery (2021) by Dehghani, Tay et al](https://arxiv.org/abs/2107.07002)
- [Stop Uploading Test Data in Plain Text (2023) by Jacovi et al](https://arxiv.org/abs/2305.10160)
- [Are We Done with MMLU? (2024) by Gema et al](https://arxiv.org/abs/2406.04127)
- [Adding Error Bars to Evals (2024) by Evan Miller](https://arxiv.org/abs/2411.00640)
- [G-Eval (2023) by Liu et al](https://arxiv.org/abs/2303.16634)
- [Judging LLM-as-a-Judge with MT-Bench (2023) by Zheng et al](https://arxiv.org/abs/2306.05685)
- [Prometheus (2023) by Kim et al](https://arxiv.org/abs/2310.08491)
- [Auto-J (2023) by Li et al](https://arxiv.org/abs/2310.05470)
- [JudgeLM (2023) by Zhu et al](https://arxiv.org/abs/2310.17631)
- [Who Validates the Validators? (2024) by Shankar et al](https://arxiv.org/abs/2404.12272)
- [Prometheus 2 (2024) by Kim et al](https://arxiv.org/abs/2405.01535)
- [Judging the Judges (2024) by Thakur et al](https://arxiv.org/abs/2406.12624)
- [JudgeBench (2024) by Tan et al](https://arxiv.org/abs/2410.12784)
- [Chatbot Arena (2024) by Chiang et al](https://arxiv.org/abs/2403.04132)
- [Arena Hard / BenchBuilder (2024) by Li et al](https://arxiv.org/abs/2406.11939)
- [LLM Comparator (2024) by Kahng et al](https://arxiv.org/abs/2402.10524)
- [Length-Controlled AlpacaEval (2024) by Dubois et al](https://arxiv.org/abs/2404.04475)
- [PandaLM (2023) by Wang et al](https://arxiv.org/abs/2306.05087)
- [Luna (Galileo) (2024) by Belyi et al](https://arxiv.org/abs/2406.00975)
- [Lynx (Patronus AI) (2024) by Ravi et al](https://arxiv.org/abs/2407.08488)
- [AgentBench (2023) by Liu et al](https://arxiv.org/abs/2308.03688)
- [WebArena (2023) by Zhou et al](https://arxiv.org/abs/2307.13854)
- [VisualWebArena (2024) by Koh et al](https://arxiv.org/abs/2401.13649)
- [WorkArena (2024) by Drouin et al](https://arxiv.org/abs/2403.07718)
- [SWE-bench (2023) by Jimenez et al](https://arxiv.org/abs/2310.06770)
- [SWE-bench Multimodal (2024) by Yang et al](https://arxiv.org/abs/2410.03859)
- [GAIA (2023) by Mialon et al](https://arxiv.org/abs/2311.12983)
- [AgentBoard (2024) by Ma et al](https://arxiv.org/abs/2401.13178)
- [OSWorld (2024) by Xie et al](https://arxiv.org/abs/2404.07972)
- [τ-bench (2024) by Yao et al](https://arxiv.org/abs/2406.12045)
- [MLE-bench (2024) by Chan et al](https://arxiv.org/abs/2410.07095)
- [ScienceAgentBench (2024) by Chen et al](https://arxiv.org/abs/2410.05080)
- [AgentDojo (2024) by Debenedetti et al](https://arxiv.org/abs/2406.13352)
- [SWE-Lancer (2025) by Miserendino et al](https://arxiv.org/abs/2502.12115)
- [BrowseComp (2025) by Wei et al](https://arxiv.org/abs/2504.12516)
- [Cybench (2024) by Zhang et al](https://arxiv.org/abs/2408.08926)
- [LLM Agent Eval Survey (2025) by Yehudai et al](https://arxiv.org/abs/2503.16416)
- [Agent Eval Survey (2025) by Mohammadi et al](https://arxiv.org/abs/2507.21504)
- [Agent-as-a-Judge (2024) by Zhuge et al](https://arxiv.org/abs/2410.10934)
- [TRAJECT-Bench (2025) by He et al](https://arxiv.org/abs/2510.04550)
- [Beyond Black-Box Benchmarking (2025) by Moshkovich et al](https://arxiv.org/abs/2503.06745)
- [In-Context Reward Hacking (2024) by Pan et al](https://arxiv.org/abs/2402.06627)
- [Concrete Problems in AI Safety (2016) by Amodei et al](https://arxiv.org/abs/1606.06565)
- [AI Sandbagging (2024) by van der Weij et al](https://arxiv.org/abs/2406.07358)
- [Alignment Faking (2024) by Greenblatt et al](https://arxiv.org/abs/2412.14093)
- [DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents (2025) by Du et al](https://arxiv.org/abs/2506.11763)
- [Why Do Multi-Agent LLM Systems Fail? (2025) by Cemri et al](https://arxiv.org/abs/2503.13657)
- [MultiAgentBench: Evaluating the Collaboration and Competition of LLM Agents (2025) by Zhu et al](https://arxiv.org/abs/2503.01935)
- [Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks (2024) by Fourney et al](https://arxiv.org/abs/2411.04468)
- [MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Servers (2025) by Luo et al](https://arxiv.org/abs/2508.14704)
- [LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools? (2025) by Mo et al](https://arxiv.org/abs/2508.01780)
- [LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (2024) by Wu et al](https://arxiv.org/abs/2410.10813)
- [Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions (2025) by Hu et al](https://arxiv.org/abs/2507.05257)
- [Measuring AI Ability to Complete Long Software Tasks (METR) (2025) by Kwa et al](https://arxiv.org/abs/2503.14499)
- [Lost in the Middle (2023) by Liu et al](https://arxiv.org/abs/2307.03172)
- [LongBench (2023) by Bai et al](https://arxiv.org/abs/2308.14508)
- [∞Bench (2024) by Zhang et al](https://arxiv.org/abs/2402.13718)
- [RULER (2024) by Hsieh et al](https://arxiv.org/abs/2404.06654)
- [BABILong (2024) by Kuratov et al](https://arxiv.org/abs/2406.10149)
- [One Thousand and One Pairs (2024) by Karpinska et al](https://arxiv.org/abs/2406.16264)
- [Loong (2024) by Wang et al](https://arxiv.org/abs/2406.17419)
- [LongBench v2 (2024) by Bai et al](https://arxiv.org/abs/2412.15204)
- [NoLiMa (2025) by Modarressi et al](https://arxiv.org/abs/2502.05167)
- [On the Measure of Intelligence (2019) by Chollet](https://arxiv.org/abs/1911.01547)
- [MATH (2021) by Hendrycks et al](https://arxiv.org/abs/2103.03874)
- [Chain-of-Thought (2022) by Wei et al](https://arxiv.org/abs/2201.11903)
- [Self-Consistency (2022) by Wang et al](https://arxiv.org/abs/2203.11171)
- [Let's Verify Step by Step (2023) by Lightman et al](https://arxiv.org/abs/2305.20050)
- [GPQA (2023) by Rein et al](https://arxiv.org/abs/2311.12022)
- [Skill-Mix (2023) by Yu et al](https://arxiv.org/abs/2310.17567)
- [FrontierMath (2024) by Glazer et al](https://arxiv.org/abs/2411.04872)
- [ARC-AGI-2 (2025) by Chollet et al](https://arxiv.org/abs/2505.11831)
- [Humanity's Last Exam (2025) by Phan et al](https://arxiv.org/abs/2501.14249)
- [TruthfulQA (2021) by Lin et al](https://arxiv.org/abs/2109.07958)
- [BBQ (2021) by Parrish et al](https://arxiv.org/abs/2110.08193)
- [Red Teaming Language Models (2022) by Ganguli et al](https://arxiv.org/abs/2209.07858)
- [Constitutional AI (2022) by Bai et al](https://arxiv.org/abs/2212.08073)
- [Universal Adversarial Attacks (2023) by Zou et al](https://arxiv.org/abs/2307.15043)
- [Towards Understanding Sycophancy (2023) by Sharma et al](https://arxiv.org/abs/2310.13548)
- [Quantifying LM Sensitivity to Spurious Features (2023) by Sclar et al](https://arxiv.org/abs/2310.11324)
- [HarmBench (2024) by Mazeika et al](https://arxiv.org/abs/2402.04249)
- [BOLD (2021) by Dhamala et al](https://arxiv.org/abs/2101.11718)
- [HaluEval (2023) by Li et al](https://arxiv.org/abs/2305.11747)
- [XSTest: Exaggerated Safety Behaviours (2023) by Rottger et al](https://arxiv.org/abs/2308.01263)
- [MCPTox: A Benchmark for Tool Poisoning on Real-World MCP Servers (2025) by Wang et al](https://arxiv.org/abs/2508.14925)
- [Equality of Opportunity in Supervised Learning (2016) by Hardt et al](https://arxiv.org/abs/1610.02413)
- [Inherent Trade-Offs in the Fair Determination of Risk Scores (2016) by Kleinberg et al](https://arxiv.org/abs/1609.05807)
- [AI Fairness 360 (2018) by Bellamy et al](https://arxiv.org/abs/1810.01943)
- [Why Should I Trust You: Explaining the Predictions of Any Classifier (2016) by Ribeiro et al](https://arxiv.org/abs/1602.04938)
- [A Unified Approach to Interpreting Model Predictions (2017) by Lundberg & Lee](https://arxiv.org/abs/1705.07874)
- [Axiomatic Attribution for Deep Networks (2017) by Sundararajan et al](https://arxiv.org/abs/1703.01365)
- [Sanity Checks for Saliency Maps (2018) by Adebayo et al](https://arxiv.org/abs/1810.03292)
- [RealToxicityPrompts (2020) by Gehman et al](https://arxiv.org/abs/2009.11462)
- [Datasheets for Datasets (2018) by Gebru et al](https://arxiv.org/abs/1803.09010)
- [Model Cards for Model Reporting (2018) by Mitchell et al](https://arxiv.org/abs/1810.03993)
- [Poisoning Web-Scale Training Datasets is Practical (2023) by Carlini et al](https://arxiv.org/abs/2302.10149)
- [OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks (2026) by Yuan et al](https://arxiv.org/abs/2606.29537)
- [Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge (2025) by Gou et al](https://arxiv.org/abs/2506.21506)
- [TheAgentCompany (2024) by Xu et al](https://arxiv.org/abs/2412.14161)
- [DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Report (2026) by Li et al](https://arxiv.org/abs/2601.08536)
- [DRACO (2026) by Perplexity AI](https://arxiv.org/abs/2602.11685)
- [Search-Time Contamination in Deep Research Agents (2026)](https://arxiv.org/abs/2606.05241)
- [AI Agents That Matter (2024) by Kapoor et al](https://arxiv.org/abs/2407.01502)
- [Rubrics as Rewards (2025) by Gunjal et al](https://arxiv.org/abs/2507.17746)
- [Reinforcement Learning with Rubric Anchors (2025) by Huang et al](https://arxiv.org/abs/2508.12790)
- [OpenRubrics (ACL 2026) (2025) by Liu et al](https://arxiv.org/abs/2510.07743)
- [EvoRubrics (2026) by Ding et al](https://arxiv.org/abs/2606.23038)
- [Reward Hacking in Rubric-Based Reinforcement Learning (2026) by Mahmoud et al](https://arxiv.org/abs/2605.12474)
- [HealthBench (OpenAI) (2025) by Arora et al](https://arxiv.org/abs/2505.08775)
- [Rubrics Across the Evolving LLM Landscape (survey) (2026) by Chen et al](https://arxiv.org/abs/2606.08625)
- [BEIR (2021) by Thakur et al](https://arxiv.org/abs/2104.08663)
- [FActScore (2023) by Min et al](https://arxiv.org/abs/2305.14251)
- [RAGAS (2023) by Es et al](https://arxiv.org/abs/2309.15217)
- [Self-RAG (2023) by Asai et al](https://arxiv.org/abs/2310.11511)
- [ARES (2023) by Saad-Falcon et al](https://arxiv.org/abs/2311.09476)
- [SAFE (2024) by Wei et al](https://arxiv.org/abs/2403.18802)
- [SimpleQA (2024) by Wei et al](https://arxiv.org/abs/2411.04368)
- [HalluLens (2025) by Bang et al](https://arxiv.org/abs/2504.17550)
- [SummEval: Re-evaluating Summarization Evaluation (2020) by Fabbri et al](https://arxiv.org/abs/2007.12626)
- [Know What You Don't Know: Unanswerable Questions for SQuAD (2018) by Rajpurkar et al](https://arxiv.org/abs/1806.03822)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering (2018) by Yang et al](https://arxiv.org/abs/1809.09600)
- [RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models (2024) by Niu et al](https://arxiv.org/abs/2401.00396)
- [From Local to Global: A Graph RAG Approach to Query-Focused Summarization (2024) by Edge et al](https://arxiv.org/abs/2404.16130)
- [RAG vs. GraphRAG: A Systematic Evaluation and Key Insights (2025) by Han et al](https://arxiv.org/abs/2502.11371)
- [MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries (2024) by Tang et al](https://arxiv.org/abs/2401.15391)
- [MMBench (2023) by Liu et al](https://arxiv.org/abs/2307.06281)
- [MM-Vet (2023) by Yu et al](https://arxiv.org/abs/2308.02490)
- [MMMU (2023) by Yue et al](https://arxiv.org/abs/2311.16502)
- [BLINK (2024) by Fu et al](https://arxiv.org/abs/2404.12390)
- [Vibe-Eval (2024) by Padlewski et al](https://arxiv.org/abs/2405.02287)
- [Video-MME (2024) by Fu et al](https://arxiv.org/abs/2405.21075)
- [MMMU-Pro (2024) by Yue et al](https://arxiv.org/abs/2409.02813)
- [ChartQA (2022) by Masry et al](https://arxiv.org/abs/2203.10244)
- [DocVQA (2020) by Mathew et al](https://arxiv.org/abs/2007.00398)
- [API-Bank (2023) by Li et al](https://arxiv.org/abs/2304.08244)
- [Gorilla (2023) by Patil et al](https://arxiv.org/abs/2305.15334)
- [ToolLLM/ToolBench (2023) by Qin et al](https://arxiv.org/abs/2307.16789)
- [SPADE (2024) by Shankar et al](https://arxiv.org/abs/2401.03038)
- [RewardBench (2024) by Lambert et al](https://arxiv.org/abs/2403.13787)
- [MixEval (2024) by Ni et al](https://arxiv.org/abs/2406.06565)
- [WildBench (2024) by Lin et al](https://arxiv.org/abs/2406.04770)
- [InfoBench (2024) by Qin et al](https://arxiv.org/abs/2401.03601)
- [IFEval (2023) by Zhou et al](https://arxiv.org/abs/2311.07911)
- [FollowBench (2023) by Jiang et al](https://arxiv.org/abs/2310.20410)
- [Data-Centric AI for LLMs (2024) by Xu et al](https://arxiv.org/abs/2406.14473)
- [COMET: A Neural Framework for MT Evaluation (2020) by Rei et al](https://arxiv.org/abs/2009.09025)
- [CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task (2022) by Rei et al](https://arxiv.org/abs/2209.06243)
- [Are LLMs Breaking MT Metrics? Results of the WMT24 Metrics Shared Task by Freitag et al](https://aclanthology.org/2024.wmt-1.2/)
- [HumanEval (2021) by Chen et al](https://arxiv.org/abs/2107.03374)
- [MBPP (2021) by Austin et al](https://arxiv.org/abs/2108.07732)
- [APPS (2021) by Hendrycks et al](https://arxiv.org/abs/2105.09938)
- [HumanEval+/MBPP+ (2023) by Liu et al](https://arxiv.org/abs/2305.01210)
- [LiveCodeBench (2024) by Jain et al](https://arxiv.org/abs/2403.07974)
- [BigCodeBench (2024) by Zhuo et al](https://arxiv.org/abs/2406.15877)
- [RepoBench (2023) by Liu, Xu, McAuley](https://arxiv.org/abs/2306.03091)
- [CRUXEval (2024) by Gu et al](https://arxiv.org/abs/2401.03065)
- [ClassEval (2023) by Du et al](https://arxiv.org/abs/2308.01861)
- [Multi-SWE-bench (2025) by Zan et al](https://arxiv.org/abs/2504.02605)
- [BIRD: Big Bench for Large-Scale Database Grounded Text-to-SQLs (2023) by Li et al](https://arxiv.org/abs/2305.03111)
- [Semantic Evaluation for Text-to-SQL with Distilled Test Suites (2020) by Zhong, Yu, Klein](https://arxiv.org/abs/2010.02840)
- [SpecBench: Evaluating Specification-Level Reasoning for Software Engineering LLM Agents (2026) by Hamblin et al](https://arxiv.org/abs/2605.30314)
- [SpecEval: Evaluating Code Comprehension in Large Language Models via Program Specifications (2024) by Ma et al](https://arxiv.org/abs/2409.12866)
- [Using LLMs in Software Requirements Specifications: An Empirical Evaluation (2024) by Krishna et al](https://arxiv.org/abs/2404.17842)
- [Commit0: Library Generation from Scratch (2024) by Zhao et al](https://arxiv.org/abs/2412.01769)
- [CLIP (2021) by Radford et al](https://arxiv.org/abs/2103.00020)
- [FID (Two Time-Scale Update Rule) (2017) by Heusel et al](https://arxiv.org/abs/1706.08500)
- [SimCSE (2021) by Gao, Yao, Chen](https://arxiv.org/abs/2104.08821)
- [Robust Speech Recognition via Large-Scale Weak Supervision (2022) by Radford et al](https://arxiv.org/abs/2212.04356)
- [WMT24 Findings](https://aclanthology.org/2024.wmt-1.1/)

### Tool documentation

> Vendor and open-source eval platform documentation.

- [Langfuse Docs](https://langfuse.com/docs)
- [LangSmith Evaluation Concepts](https://docs.langchain.com/langsmith/evaluation-concepts)
- [Langfuse Scores](https://langfuse.com/docs/scores/overview)
- [Braintrust Docs](https://www.braintrust.dev/docs)
- [Phoenix Tutorials/Evals](https://github.com/Arize-ai/phoenix/tree/main/tutorials/evals)
- [Phoenix Cookbook Evaluation](https://arize.com/docs/phoenix/cookbook/evaluation)
- [Inspect AI Docs](https://inspect.aisi.org.uk/)
- [RAGAS Docs](https://docs.ragas.io/)
- [RAG eval by RAGAS Getting Started](https://docs.ragas.io/en/stable/getstarted/rag_eval/)
- [RAGAS Faithfulness metric](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/)
- [TruLens Docs](https://www.trulens.org/)
- [Promptfoo Docs](https://www.promptfoo.dev/)
- [Promptfoo Red-team](https://www.promptfoo.dev/docs/red-team/)
- [Evaluate an LLM Application by LangSmith](https://docs.langchain.com/langsmith/evaluate-llm-application)
- [LLM-as-a-Judge evaluators by Langfuse Docs](https://langfuse.com/docs/evaluation/evaluation-methods/llm-as-a-judge)
- [Open-sourcing the Langfuse product (MIT) by Langfuse](https://langfuse.com/blog/2025-06-04-open-sourcing-langfuse-product)
- [Testset Generation for RAG by RAGAS Docs](https://docs.ragas.io/en/stable/concepts/test_data_generation/rag/)
- [Running Evals on Traces by Phoenix Docs](https://arize.com/docs/phoenix/tracing/how-to-tracing/feedback-and-annotations/evaluating-phoenix-traces)
- [Self-hosting from the terminal by Phoenix Docs](https://arize.com/docs/phoenix/self-hosting/deployment-options/terminal)
- [Phoenix evals package (arize-phoenix-evals)](https://github.com/Arize-ai/phoenix/tree/main/packages/phoenix-evals)
- [Langfuse Pricing](https://langfuse.com/pricing)
- [Langfuse Integrations](https://langfuse.com/integrations)
- [Home by LangSmith Docs](https://docs.langchain.com/langsmith/home)
- [Trace with OpenTelemetry by LangSmith Docs](https://docs.langchain.com/langsmith/trace-with-opentelemetry)
- [Manage prompts programmatically by LangSmith Docs](https://docs.langchain.com/langsmith/manage-prompts-programmatically)
- [Braintrust Pricing](https://www.braintrust.dev/pricing)
- [Start by Braintrust Docs](https://www.braintrust.dev/docs/start)
- [Trace integrations by Braintrust Docs](https://www.braintrust.dev/docs/guides/traces/integrations)
- [Phoenix overview by Arize Docs](https://arize.com/docs/phoenix)
- [Phoenix LICENSE (Elastic License 2.0)](https://github.com/Arize-ai/phoenix/blob/main/LICENSE)
- [Customize models by Ragas Docs](https://docs.ragas.io/en/stable/howtos/customizations/customize_models/)
- [Available metrics by Ragas Docs](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/)
- [Model-graded metrics by Promptfoo Docs](https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/)
- [Scorers by Inspect AI Docs](https://inspect.aisi.org.uk/scorers.html)
- [Models by Inspect AI Docs](https://inspect.aisi.org.uk/models.html)
- [Log viewer by Inspect AI Docs](https://inspect.aisi.org.uk/log-viewer.html)
- [Getting started by TruLens Docs](https://www.trulens.org/getting_started/)

### Standards and regulations

> NIST, EU AI Act, OWASP, ITU, MLCommons.

- [The TREC-8 Question Answering Track Report by Voorhees](https://trec.nist.gov/pubs/trec8/papers/qa_report.pdf)
- [ITU-T P.808. Subjective MOS via crowdsourcing](https://www.itu.int/rec/T-REC-P.808)
- [MLCommons AILuminate](https://mlcommons.org/benchmarks/ailuminate/)
- [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework)
- [official portal by EU AI Act](https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai)
- [AI Omnibus enters into force (Regulation (EU) 2026/1744) (2026) by European Commission](https://digital-strategy.ec.europa.eu/en/news/ai-omnibus-enters-force)
- [The General-Purpose AI Code of Practice by European Commission](https://digital-strategy.ec.europa.eu/en/policies/contents-code-gpai)
- [OWASP Top 10 for LLM Applications](https://owasp.org/www-project-top-10-for-large-language-model-applications/)
- [OWASP Top 10 for Agentic Applications (2026) (2026)](https://genai.owasp.org/2025/12/09/owasp-top-10-for-agentic-applications-the-benchmark-for-agentic-security-in-the-age-of-autonomous-ai/)

### Other sources

- [Yan, Bischof, Frye, Husain, Liu, Shankar by Applied LLMs](https://applied-llms.org/)
- [LLM Powered Autonomous Agents by Lilian Weng](https://lilianweng.github.io/posts/2023-06-23-agent/)
- [Extrinsic Hallucinations in LLMs by Lilian Weng](https://lilianweng.github.io/posts/2024-07-07-hallucination/)
- [Adversarial Attacks on LLMs by Lilian Weng](https://lilianweng.github.io/posts/2023-10-25-adv-attack-llm/)
- [Specification Gaming Examples by Victoria Krakovna](https://vkrakovna.wordpress.com/2018/04/02/specification-gaming-examples-in-ai/)
- [LLM Evaluation Guidebook by Clémentine Fourrier](https://huggingface.co/blog/clefourrier/llm-evaluation)
- [Evaluating MMLU by HuggingFace](https://huggingface.co/blog/evaluating-mmlu-leaderboard)
- [Hard Stuff Nobody Talks About by Honeycomb](https://www.honeycomb.io/blog/hard-stuff-nobody-talks-about-llm)
- [Building Effective Agents by Anthropic](https://www.anthropic.com/engineering/building-effective-agents)
- [SWE-bench Sonnet by Anthropic](https://www.anthropic.com/research/swe-bench-sonnet)
- [Measuring Faithfulness in CoT by Anthropic](https://www.anthropic.com/research/measuring-faithfulness-in-chain-of-thought-reasoning)
- [Reasoning Models Don't Say What They Think by Anthropic](https://www.anthropic.com/research/reasoning-models-dont-say-think)
- [Measuring Persuasiveness of LMs by Anthropic](https://www.anthropic.com/research/measuring-model-persuasiveness)
- [Anthropic Responsible Scaling Policy](https://www.anthropic.com/responsible-scaling-policy)
- [Agentic Misalignment by Anthropic](https://www.anthropic.com/research/agentic-misalignment)
- [Adding Error Bars to Evals (blog) by Anthropic](https://www.anthropic.com/research/statistical-approach-to-model-evals)
- [Anthropic Tool Use Overview (strict mode)](https://platform.claude.com/docs/en/docs/agents-and-tools/tool-use/overview)
- [Anthropic Evaluation Tool (Console)](https://platform.claude.com/docs/en/test-and-evaluate/eval-tool)
- [Define success criteria and build evaluations by Anthropic](https://platform.claude.com/docs/en/docs/test-and-evaluate/develop-tests)
- [Claude API pricing by Anthropic](https://platform.claude.com/docs/en/about-claude/pricing)
- [OpenAI Evals (GitHub)](https://github.com/openai/evals)
- [Introducing SWE-bench Verified by OpenAI](https://openai.com/index/introducing-swe-bench-verified/)
- [OpenAI Preparedness Framework](https://openai.com/safety/preparedness/)
- [A Practical Guide to Building Agents by OpenAI](https://openai.com/business/guides-and-resources/a-practical-guide-to-building-ai-agents/)
- [Getting Started with OpenAI Evals by OpenAI Cookbook](https://developers.openai.com/cookbook/examples/evaluation/getting_started_with_openai_evals)
- [Eval LLMs for SQL Generation by OpenAI Cookbook](https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_evaluate_LLMs_for_SQL_generation.ipynb)
- [Evaluate RAG with LlamaIndex by OpenAI Cookbook](https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb)
- [OpenAI Preparedness GitHub (MLE-bench, SWE-Lancer, PaperBench)](https://github.com/openai/preparedness)
- [OpenAI SWE-Lancer Benchmark](https://github.com/openai/swelancer-benchmark)
- [Graders guide (Evals API and RFT) by OpenAI](https://developers.openai.com/api/docs/guides/graders)
- [Time Horizon 1.1 by METR](https://metr.org/blog/2026-1-29-time-horizon-1-1/)
- [Needle in a Haystack by Greg Kamradt](https://github.com/gkamradt/LLMTest_NeedleInAHaystack)
- [Fairness and Machine Learning by Barocas, Hardt & Narayanan](https://fairmlbook.org/)
- [xlang.ai. OSWorld-Verified](https://xlang.ai/blog/osworld-verified)
- [More compute, more capability (test-time compute in agent evals) by UK AISI](https://www.aisi.gov.uk/blog/more-compute-more-capability-why-ai-agent-evals-need-to-account-for-test-time-compute)
- [Introduction to Information Retrieval by Manning, Raghavan & Schütze](https://nlp.stanford.edu/IR-book/)
- [Cumulated Gain-Based Evaluation of IR Techniques by Järvelin & Kekäläinen](https://dl.acm.org/doi/10.1145/582415.582418)
- [BenchmarkQED: Automated Benchmarking of RAG Systems by Microsoft Research](https://www.microsoft.com/en-us/research/blog/benchmarkqed-automated-benchmarking-of-rag-systems/)
- [BFCL (Patil, Mao et al., ICML 2025) (2025)](https://gorilla.cs.berkeley.edu/leaderboard.html)
- [AST vs executable evaluation by BFCL blog](https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html)
- [Multi-turn and state-based evaluation by BFCL blog](https://gorilla.cs.berkeley.edu/blogs/13_bfcl_v3_multi_turn.html)
- [BIRD-bench](https://bird-bench.github.io/)
- [Spider by Yu et al](https://yale-lily.github.io/spider)
- [NLP4RE ReqEval Task. Ambiguity Detection and Disambiguation in Requirements](https://nlp4re.github.io/2020/reqeval.html)
- [Scale AI SWE-bench Pro Public Leaderboard](https://labs.scale.com/leaderboard/swe_bench_pro_public)
- [Laude Institute Terminal-Bench](https://www.tbench.ai/)
- [PartiPrompts by Google Research](https://github.com/google-research/parti)
- [MTEB Benchmark](https://github.com/embeddings-benchmark/mteb)
- [Whisper repo (Common Voice 15, Fleurs WER) by OpenAI](https://github.com/openai/whisper)
- [Comet Opik](https://github.com/comet-ml/opik)
- [Inspect Evals Registry](https://github.com/UKGovernmentBEIS/inspect_evals)
- [Inspect AI GitHub](https://github.com/UKGovernmentBEIS/inspect_ai)
- [RAGAS GitHub](https://github.com/explodinggradients/ragas)
- [Fairlearn Docs](https://fairlearn.org/)
- [AIF360 GitHub](https://github.com/Trusted-AI/AIF360)
- [SHAP Docs](https://shap.readthedocs.io/en/latest/)
- [Captum Docs](https://captum.ai/)
- [Perspective API](https://perspectiveapi.com/)
- [OpenTelemetry GenAI Semantic Conventions repository](https://github.com/open-telemetry/semantic-conventions-genai)
- [DeepEval GitHub](https://github.com/confident-ai/deepeval)
- [Giskard GitHub](https://github.com/Giskard-AI/giskard)
- [LlamaIndex Evaluation](https://developers.llamaindex.ai/python/framework/module_guides/evaluating/)
- [Outlines GitHub](https://github.com/outlines-dev/outlines)
- [Instructor GitHub](https://github.com/jxnl/instructor)
- [HuggingFace LLM-as-judge Cookbook](https://huggingface.co/learn/cookbook/en/llm_judge)
- [HarmBench GitHub](https://github.com/centerforaisafety/HarmBench)
- [NVIDIA garak](https://github.com/NVIDIA/garak)
- [WebArena GitHub](https://github.com/web-arena-x/webarena)
- [VisualWebArena GitHub](https://github.com/web-arena-x/visualwebarena)
- [OSWorld site](https://os-world.github.io/)
- [SWE-bench site](https://www.swebench.com/)
- [SWE-bench Verified](https://www.swebench.com/verified.html)
- [τ-bench GitHub](https://github.com/sierra-research/tau-bench)
- [AgentBench GitHub](https://github.com/THUDM/AgentBench)
- [NVIDIA RULER](https://github.com/NVIDIA/RULER)
- [BABILong HF dataset](https://huggingface.co/datasets/RMT-team/babilong)
- [OpenAI HumanEval](https://github.com/openai/human-eval)
- [BigCodeBench GitHub](https://github.com/bigcode-project/bigcodebench)
- [LiveCodeBench GitHub](https://github.com/LiveCodeBench/LiveCodeBench)
- [DocVQA](https://www.docvqa.org/)
- [Intro to evals by Vercel](https://vercel.com/kb/guide/an-introduction-to-evals)
- [LLM evals with Vercel AI SDK + Vitest by Xata](https://xata.io/blog/llm-evals-with-vercel-ai-and-vitest)
- [FastChat MT-Bench README](https://github.com/lm-sys/FastChat/blob/main/fastchat/llm_judge/README.md)
- [Prompt_evaluations by Anthropic Courses](https://github.com/anthropics/courses/tree/master/prompt_evaluations)
- [Building_evals.ipynb by Anthropic Cookbook](https://github.com/anthropics/claude-cookbooks/blob/main/misc/building_evals.ipynb)
- [SWE-bench harness GitHub](https://github.com/SWE-bench/SWE-bench)
- [SWE-bench Verified HF dataset](https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified)
- [mini-SWE-agent GitHub](https://github.com/SWE-agent/mini-swe-agent)
- [SWE-bench sb-cli docs](https://www.swebench.com/sb-cli/)
- [Speed, Structure, Smarts by Notion AI](https://www.notion.com/blog/speed-structure-and-smarts-the-notion-ai-way)
- [Decision-Time Guidance by Replit](https://replit.com/blog/decision-time-guidance)
- [RAG eval with Ragas by Langfuse cookbook](https://github.com/langfuse/langfuse-docs/blob/main/cookbook/evaluation_of_rag_with_ragas.ipynb)
- [Error analysis by Langfuse cookbook](https://github.com/langfuse/langfuse-docs/blob/main/cookbook/error-analysis-llm-applications.ipynb)
- [Multi-turn eval by Langfuse cookbook](https://github.com/langfuse/langfuse-docs/blob/main/cookbook/example_evaluating_multi_turn_conversations.ipynb)
- [Pinecone RAG eval](https://www.pinecone.io/learn/series/vector-databases-in-production-for-busy-engineers/rag-evaluation/)
- [MLE-bench repo](https://github.com/openai/mle-bench)
- [PaperBench repo](https://github.com/openai/preparedness/tree/main/project/paperbench)
- [Luna-2 evaluation SLMs by Galileo](https://galileo.ai/luna-2)
- [HHEM-2.1-Open model card by Vectara](https://huggingface.co/vectara/hallucination_evaluation_model)
- [External evaluation pipelines by Langfuse cookbook](https://github.com/langfuse/langfuse-docs/blob/main/cookbook/example_external_evaluation_pipelines.ipynb)
- [datasketch. MinHash and MinHash LSH docs](https://ekzhu.com/datasketch/)
- [Vercel AI SDK docs](https://ai-sdk.dev/docs/introduction)
- [Vercel AI SDK 6 migration guide](https://ai-sdk.dev/docs/migration-guides/migration-guide-6-0)
- [Vitest docs](https://vitest.dev/)
- [DeepResearch Bench GitHub (RACE + FACT scoring)](https://github.com/Ayanami0730/deep_research_bench)
- [MAST GitHub (taxonomy, LLM annotator, trace dataset)](https://github.com/multi-agent-systems-failure-taxonomy/MAST)
- [MCP-Universe GitHub](https://github.com/SalesforceAIResearch/MCP-Universe)
- [ModelScope MCPBench GitHub](https://github.com/modelscope/MCPBench)
- [tau2-bench GitHub (successor line, ships tau3-bench) by Sierra Research](https://github.com/sierra-research/tau2-bench)
- [GitHub Spec Kit](https://github.com/github/spec-kit)
- [Kiro Specs Documentation](https://kiro.dev/docs/specs/)
- [LICENSE by Langfuse GitHub](https://github.com/langfuse/langfuse/blob/main/LICENSE)
- [LangChain Pricing](https://www.langchain.com/pricing)
- [Braintrust Python SDK (PyPI)](https://pypi.org/project/braintrust/)
- [LangWatch Pricing](https://langwatch.ai/pricing)
- [LangWatch Integrations](https://langwatch.ai/integrations)
- [Arize Pricing](https://arize.com/pricing/)
- [AX production monitoring by Arize Docs](https://arize.com/docs/ax/observe/production-monitoring)
- [Weights and Biases Pricing](https://wandb.ai/site/pricing)
- [Weave Docs](https://weave-docs.wandb.ai/)
- [Weave product page by Weights and Biases](https://wandb.ai/site/weave)
- [W&B Weave GitHub (license)](https://github.com/wandb/weave)
- [Helicone Pricing](https://www.helicone.ai/pricing)
- [Helicone Docs](https://docs.helicone.ai/)
- [Self-host overview by Helicone Docs](https://docs.helicone.ai/getting-started/self-host/overview)
- [Scores by Helicone Docs](https://docs.helicone.ai/features/advanced-usage/scores)
- [Experiments by Helicone Docs](https://docs.helicone.ai/features/experiments)
- [Helicone GitHub (license)](https://github.com/Helicone/helicone)
- [Vercel AI SDK integration by Helicone Docs](https://docs.helicone.ai/getting-started/integration-method/vercelai)
- [Acquired by Anthropic, platform sunset by Humanloop](https://humanloop.com/)
- [PromptLayer Pricing](https://www.promptlayer.com/pricing)
- [PromptLayer Docs](https://docs.promptlayer.com/)
- [PromptLayer homepage](https://www.promptlayer.com/)
- [Comet Pricing (Opik)](https://www.comet.com/site/pricing/)
- [Quickstart by Opik Docs](https://www.comet.com/docs/opik/quickstart)
- [Log traces by Opik Docs](https://www.comet.com/docs/opik/tracing/log_traces)
- [Online evaluation rules by Opik Docs](https://www.comet.com/docs/opik/production/rules)
- [Managing prompts in code by Opik Docs](https://www.comet.com/docs/opik/prompt_engineering/managing_prompts_in_code)
- [Promptfoo GitHub](https://github.com/promptfoo/promptfoo)
- [Getting started by DeepEval Docs](https://deepeval.com/docs/getting-started)
- [Build-eval doc by OpenAI Evals](https://github.com/openai/evals/blob/main/docs/build-eval.md)
- [Evals API guide (dashboard deprecation notice) by OpenAI](https://developers.openai.com/api/docs/guides/evals)
- [Evidently AI homepage](https://www.evidentlyai.com/)
- [Evidently Docs](https://docs.evidentlyai.com/)
- [Quickstart LLM by Evidently Docs](https://docs.evidentlyai.com/quickstart_llm)
- [Customize LLM judge by Evidently Docs](https://docs.evidentlyai.com/metrics/customize_llm_judge)
- [European Commission AI Act Service Desk. Implementation timeline](https://ai-act-service-desk.ec.europa.eu/en/ai-act/timeline/timeline-implementation-eu-ai-act)
- [Microsoft Responsible AI hub](https://www.microsoft.com/en-us/ai/responsible-ai)
- [MCP Security Notification: Tool Poisoning Attacks by Invariant Labs](https://invariantlabs.ai/blog/mcp-security-notification-tool-poisoning-attacks)
- [AI threats in the wild: prompt injections on the web by Google Security](https://blog.google/security/prompt-injections-web/)
- [Palo Alto Unit 42. Indirect Prompt Injection](https://unit42.paloaltonetworks.com/ai-agent-prompt-injection/)
- [Comet prompt injection (account takeover) by Brave](https://brave.com/blog/comet-prompt-injection/)
- [Claude for Chrome (injection mitigations) by Anthropic](https://claude.com/blog/claude-for-chrome)
- [Prompt injection defenses by Anthropic](https://www.anthropic.com/research/prompt-injection-defenses)
- [Binomial Proportion CI](https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval)
- [Cohen's kappa](https://en.wikipedia.org/wiki/Cohen%27s_kappa)
- [Krippendorff's alpha](https://en.wikipedia.org/wiki/Krippendorff%27s_alpha)
- [McNemar's test](https://en.wikipedia.org/wiki/McNemar%27s_test)
- [MinHash](https://en.wikipedia.org/wiki/MinHash)
- [AI Evals (Hamel & Shreya) by Maven course](https://maven.com/parlance-labs/evals)
- [Automated Testing for LLMOps by DLAI](https://www.deeplearning.ai/short-courses/automated-testing-llmops/)

## How to cite this site

If you reproduce material from AI Evals in an answer, please cite the source URL and the author. APA shape:

Bharatiya, O. (2026). AI Evals: The most useful, opinionated reference for AI evals. https://www.aievals.co

Content license: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/).

## Optional

- [llms-full.txt](https://www.aievals.co/llms-full.txt): full plain text of every page, separated by --- markers
- [Sitemap](https://www.aievals.co/sitemap.xml): every indexable URL
- [RSS](https://www.aievals.co/rss.xml): recently updated pages
- [Atom](https://www.aievals.co/atom.xml): Atom 1.0 mirror
- [Papers RSS](https://www.aievals.co/papers/rss.xml): only paper-theme updates