What "good" looks like
A deep research agent is evaluated on the report, not the run. Score a long cited document against per-task weighted rubric criteria and a reference report, plus a separate check that every citation actually supports the claim it is attached to. Free-form "which report is better" judging drifts; anchored rubric scoring does not.
The ship targets:
- RACE-style rubric quality, scored relative to a gold reference report rather than in isolation.
- Citation precision from the FACT pipeline: every citation must support the claim it is attached to.
- A retrieval floor from short-answer sets, treated as a check rather than the headline.
The RACE framework from DeepResearch Bench is the current best articulation. It scores comprehensiveness, insight and analytical depth, instruction-following, and readability using criteria generated dynamically per task, weighted, and scored relative to a gold reference report.
Why it holds up as a primary metric. The paper reports 71.33% pairwise agreement with human experts, above the 68.44% agreement rate between the human experts themselves. That comparison is what makes the method usable as a primary metric[1].
Paper-time anchors. The benchmark is 100 PhD-level research tasks across 22 fields. Gemini-2.5-Pro Deep Research led with a RACE score of 48.88 and citation accuracy of 81.44; OpenAI Deep Research scored 46.98 with 77.96 citation accuracy; Perplexity posted the highest citation accuracy at 90.24[1]. Treat these as paper-time anchors, not current standings. The leaderboard the repo points to (fetched as of 2026-10-01) lists 45 entries, with a top RACE overall of 58.03 (qianfan_deepresearch_0430), and shows the paper-time leaders at 49.71 RACE and 78.30 citation accuracy (Gemini-2.5-Pro Deep Research) and 46.45 and 75.01 (OpenAI Deep Research)[2]. Those are not the paper's 48.88 and 81.44 or 46.98 and 77.96, and the maintainers changed the judge in 2026 (below), so never mix paper figures with leaderboard figures.
The 2026 successors sharpen both axes. DeepResearch Bench II, from an overlapping author group, replaces per-task generated criteria with 9,430 expert-derived binary rubrics over 132 grounded tasks (a four-stage pipeline with over 400 human-hours of expert review), and at paper time even the strongest agents satisfied fewer than half of the rubrics[3]. That has aged: the live leaderboard (fetched as of 2026-10-01) has 19 entries, the top total score is 64.38 (AI21-DeepResearch), and the paper's strongest, OpenAI o3 Deep Research, sits at 45.40[4]. Presentation is saturated (the top twelve entries all score 89 or above); the spread is in information recall, 22.95 to 60.35 across the table[4]. Perplexity's DRACO takes the in-the-wild angle: 100 tasks sampled from de-identified real deep-research requests, graded on task-specific weighted rubric criteria across factual accuracy, breadth and depth, presentation, and citation quality[5]. Perplexity's own system leads its benchmark (70.5% normalized score with Opus 4.6, against 59.0% for Gemini Deep Research), and the primary judge was Gemini-3-Pro, with the ranking stable across two other judges but absolute scores shifting[5]. Take the method, and treat a vendor-authored ranking accordingly. Both successors are rubric-tree methods, which is the direction this whole category has settled on.
Your judge model can be retired under you. Google scheduled Gemini 2.5 Pro for retirement, and both DeepResearch Bench projects had to move off it. In May 2026 the original benchmark switched the official RACE judge to GPT-5.5 and the FACT judge to GPT-5.4-mini, after testing three candidate judges on its human-annotated subset (200 articles); GPT-5.5 reached 71.82% agreement against a 68.78% human inter-annotator baseline. Its stated plan was a dual-acceptance window to 2026-05-31 with a separate leaderboard per judge, then only the GPT-5.5 board from 2026-06-01. In September 2026 it also changed the article-cleaning model to GPT-5.6 Luna while keeping GPT-5.5 as the official judge[6]. DeepResearch Bench II made the same GPT-5.5 move in August 2026 and keeps the old evaluator on a gemini-25 branch[7]. Scores under different judges are not comparable, so record the judge model and version beside every number and re-baseline before you compare across a swap. This is the judge-version discipline this site teaches.
flowchart TD
Q["Research query"] --> AGENT["Deep research agent"]
AGENT --> REP["Long cited report"]
REP --> RACE["RACE: rubric quality vs gold reference report"]
REP --> FACT["FACT: extract claim-citation pairs, fetch source, verify support"]
REP --> FLOOR["Short-answer floor: BrowseComp / GAIA"]
RACE --> GATE{"Rubric quality high and citations supported?"}
FACT --> GATE
FLOOR --> GATE
GATE -->|yes| SHIP["Ship"]
GATE -->|no| FIX["Fix the weaker axis"]
Scope guard: this page owns report-shaped outputs. Pass^k and state-check harness design live in long-horizon planning; the browsing execution layer (clicks, page parsing, dead-end recovery) lives in computer and browser use.
Secondary metrics
| Metric | What it catches | How to score |
|---|
| Citation precision (FACT) | Claims whose cited source does not support them | Extract every claim-citation pair, fetch the cited source, verify support; report precision and effective citation count. Validated at 96% alignment with human judgments on supported claims and 92% on unsupported ones[1] |
| Short-answer search accuracy | Weak retrieval persistence, as a floor check not the headline | BrowseComp: 1,266 questions requiring hard-to-locate information. Paper-time: GPT-4o 0.6%, GPT-4o with browsing 1.9%, o1 9.9%, OpenAI Deep Research 51.5%, vs 29.2% for human trainers[8]. GAIA: 466 questions in three difficulty levels; humans 92%, GPT-4 with plugins roughly 15% at paper time[9]. On the public GAIA test leaderboard (fetched as of 2026-10-01) the best entry scores 93.69% and the top ten fall between 92.0% and 93.7%, past the paper's human figure, so GAIA no longer separates frontier systems[10]. Do not quote post-paper BrowseComp saturation claims without naming who ran the harness |
| Calibration error | An agent that cites sources and sounds certain but is wrong | Track confidence vs correctness separately. The BrowseComp authors observed that browsing raises the model's stated confidence while its errors become subtler[8] |
| Cost and latency per report | Reports too slow or expensive to be the same product | Measure per report; a slow, expensive report is a different product from a fast, cheap one |
Offline eval design
Build rubric trees with domain experts, not with a single prompt. The PaperBench precedent shows what rigor looks like at the limit: 20 ICML 2024 papers decomposed into 8,316 individually gradable leaf nodes, with the LLM judge itself audited against a separate JudgeEval set[11]. You do not need 8,316 nodes; you do need the two ideas, hierarchical decomposition into checkable leaves and an audited judge.
The three public benchmarks measure different things, and teams routinely confuse them:
| BrowseComp | GAIA | DeepResearch Bench |
|---|
| Task shape | Find one hard-to-locate fact | Multi-step assistant questions, 3 levels | Open-ended PhD-level research briefs |
| Output shape | Short answer | Short answer | Long cited report |
| Scoring | LLM grader for semantic equivalence to the reference | Quasi-exact-match | RACE rubric plus FACT citation checks |
| Does not measure | Synthesis, writing, citation | Synthesis, long-form quality | Retrieval persistence on adversarially hidden facts |
For a retrieval floor you can reproduce, pin the corpus. BrowseComp runs against the live web through whatever search API the agent has, so a score drifts as the backend changes. BrowseComp-Plus keeps 830 BrowseComp queries but pairs them with a fixed corpus of about 100K documents, with human-verified evidence and mined hard negatives per query, so you can swap retrievers and compare them; GPT-5 reached 55.9% on it, and 70.1% with the Qwen3-Embedding-8B retriever and fewer search calls[12]. OpenAI's simple-evals still hosts the BrowseComp reference implementation, but it stopped publishing results for new models in July 2025[13]; check who ran the harness before comparing two BrowseComp numbers.
Pick the primary and the floor. Run DeepResearch Bench (or a domain-specific clone of its method) as the primary, with BrowseComp-style short-answer sets as the retrieval floor. Use RACE-style reference-based relative scoring: absolute scalar scores on long reports are unstable across judge runs; scoring against a gold report is not[1].
Online eval design
Three loops:
- Sampled claim-level citation audits. Pull N reports per week, extract claim-citation pairs, and have a human or a FACT-style pipeline verify support; track precision over time.
- Counterbalanced human preference pairs. When comparing model versions, show reviewers both reports with order randomized per pair, and never let one side be systematically longer without a directness control.
- Cost and latency per report with alerts on drift. Deep research runs are expensive enough that a sizeable cost regression is a product incident, not a curiosity.
Top benchmarks
- DeepResearch Bench, https://deepresearch-bench.github.io/. 100 PhD-level tasks across 22 fields; RACE and FACT scoring; the method the category standardized on[1].
- DeepResearch Bench II, https://arxiv.org/abs/2601.08536. 132 grounded tasks scored by 9,430 expert-derived binary rubrics; the paper reported the strongest agents below half, and the live leaderboard's top total is now 64.38 (fetched as of 2026-10-01)[3] [4].
- DRACO, https://arxiv.org/abs/2602.11685. 100 tasks sampled from de-identified real deep-research requests, graded on weighted rubric criteria across four dimensions[5].
- BrowseComp, https://arxiv.org/abs/2504.12516. 1,266 hard-to-find-information questions; the retrieval-persistence floor[8].
- BrowseComp-Plus, https://arxiv.org/abs/2508.06600. 830 BrowseComp queries over a fixed 100K-document corpus; the reproducible version of the floor[12].
- GAIA, https://arxiv.org/abs/2311.12983. 466 general-assistant questions in 3 levels; broad but short-answer only, and top test scores are now above 93%[9] [10].
- DeepResearch Bench harness: the deep_research_bench repo is Apache-2.0 and ships the RACE and FACT scoring scripts, so you can point them at your own tasks rather than reimplementing the judging[6].
- Inspect AI: the harness for running agents against task sets with full trace capture.
- Langfuse or Braintrust: production report logging and the annotation queues that citation audits need.
- Agent-as-judge: where the judge itself must browse or run tools to verify claims, a judge with tool access grades intermediate artifacts, not just final text[14].
Common pitfalls
| Pitfall | Why it bites | Fix |
|---|
| Verbosity and self-preference bias in win-rate judging | Long reports win pairwise comparisons by default, and judges prefer their own model family's prose | Counterbalance ordering, add a directness dimension to the rubric, use a judge from a different family than the generator, and report a judge-human agreement number; if you cannot state that number, the win rate is unaudited |
| Treating citation accuracy as factual correctness | FACT verifies that a source supports a claim, not that the source is right; a report can be almost perfectly citation-accurate and still materially wrong | Verify source correctness separately from citation support |
| Letting short-answer benchmarks stand in for synthesis quality | BrowseComp and GAIA measure retrieval persistence; a model can ace them and still write shallow, badly structured reports | Keep RACE-style rubric scoring as the primary and treat short-answer sets as a floor |
| Ignoring contamination on GAIA | Its validation answers are public and old enough to be in training data | Treat strong GAIA scores from recent models with suspicion[9] |
| Search-time contamination | A search-enabled agent can retrieve the benchmark's own question context, metadata, or answers mid-run; the paper measured inflation of up to 4% across the six medical question-answering benchmarks it tested[15] | Sandbox the eval's network access, log and audit full search trajectories, and grep them for the benchmark's own hosts before trusting a score |
Canonical reading
The DeepResearch Bench paper is the canonical read because it solves the two hard problems of this category in one method: RACE makes long-report quality scoring reproducible via reference-based relative rubrics, and FACT makes citation quality measurable at claim level[1]. The 71.33% human agreement (against a 68.44% human-to-human baseline) and the 96%/92% claim-verification alignment are the evidence that judge-based scoring can be trusted here, provided you keep auditing it.
Figure: Why LLM-as-judge scoring is trusted for deep research agents: the RACE rubric judge agrees with human experts on 71.33% of pairwise report comparisons, above the 68.44% agreement rate between the human experts themselves, and FACT citation verification aligns with human judgments on 96% of supported and 92% of unsupported claims.
Day-1 starter
- Pick 20 real research queries from your production logs. Not synthetic prompts; the queries users actually ran.
- Write 5 to 10 weighted rubric criteria per query with domain experts, plus one reference report per query to anchor relative scoring.
- Score with a judge model from a different family than the generator, and hand-verify 3 citations per report. The delta between judge citation scores and your hand checks is your judge's error bar.
Cross-cutting links