What "good" looks like
A deep research agent is evaluated on the report, not the run. Score a long cited document against per-task weighted rubric criteria and a reference report, plus a separate check that every citation actually supports the claim it is attached to. Free-form "which report is better" judging drifts; anchored rubric scoring does not.
The ship targets:
- RACE-style rubric quality, scored relative to a gold reference report rather than in isolation.
- Citation precision from the FACT pipeline: every citation must support the claim it is attached to.
- A retrieval floor from short-answer sets, treated as a check rather than the headline.
The RACE framework from DeepResearch Bench is the current best articulation. It scores comprehensiveness, insight and analytical depth, instruction-following, and readability using criteria generated dynamically per task, weighted, and scored relative to a gold reference report.
Why it holds up as a primary metric. The paper reports 71.33% pairwise agreement with human experts, above the 68.44% agreement rate between the human experts themselves. That comparison is what makes the method usable as a primary metric[1].
Paper-time anchors. The benchmark is 100 PhD-level research tasks across 22 fields. Gemini-2.5-Pro Deep Research led with a RACE score of 48.88 and citation accuracy of 81.44; OpenAI Deep Research scored 46.98 with 77.96 citation accuracy; Perplexity posted the highest citation accuracy at 90.24[1]. Treat these as paper-time anchors, not current standings; check the live leaderboard before quoting them.
The 2026 successors sharpen both axes. DeepResearch Bench II, from an overlapping author group, replaces per-task generated criteria with 9,430 expert-derived binary rubrics over 132 grounded tasks (a four-stage pipeline with over 400 human-hours of expert review), and even the strongest agents satisfy fewer than half of the rubrics[2]. Perplexity's DRACO takes the in-the-wild angle: 100 tasks sampled from de-identified real deep-research requests, graded on task-specific weighted rubric criteria across factual accuracy, breadth and depth, presentation, and citation quality[3]. Both are rubric-tree methods, which is the direction this whole category has settled on. One operational detail from the original benchmark is worth copying into your own program: when its maintainers swapped the official RACE judge model, they ran a dual-acceptance window with the leaderboard maintained under both judges through the transition, which is exactly the judge-version discipline this site teaches.
flowchart TD
Q["Research query"] --> AGENT["Deep research agent"]
AGENT --> REP["Long cited report"]
REP --> RACE["RACE: rubric quality vs gold reference report"]
REP --> FACT["FACT: extract claim-citation pairs, fetch source, verify support"]
REP --> FLOOR["Short-answer floor: BrowseComp / GAIA"]
RACE --> GATE{"Rubric quality high and citations supported?"}
FACT --> GATE
FLOOR --> GATE
GATE -->|yes| SHIP["Ship"]
GATE -->|no| FIX["Fix the weaker axis"]
Scope guard: this page owns report-shaped outputs. Pass^k and state-check harness design live in long-horizon planning; the browsing execution layer (clicks, page parsing, dead-end recovery) lives in computer and browser use.
Secondary metrics
| Metric | What it catches | How to score |
|---|
| Citation precision (FACT) | Claims whose cited source does not support them | Extract every claim-citation pair, fetch the cited source, verify support; report precision and effective citation count. Validated at 96% alignment with human judgments on supported claims and 92% on unsupported ones[1] |
| Short-answer search accuracy | Weak retrieval persistence, as a floor check not the headline | BrowseComp: 1,266 questions requiring hard-to-locate information. Paper-time: GPT-4o 0.6%, GPT-4o with browsing 1.9%, o1 9.9%, OpenAI Deep Research 51.5%, vs 29.2% for human trainers[4]. GAIA: 466 questions in three difficulty levels; humans 92%, GPT-4 with plugins roughly 15% at paper time[5]. Do not quote post-paper saturation claims; the live leaderboards are the source of truth |
| Calibration error | An agent that cites sources and sounds certain but is wrong | Track confidence vs correctness separately. The BrowseComp authors observed that browsing raises the model's stated confidence while its errors become subtler[4] |
| Cost and latency per report | Reports too slow or expensive to be the same product | Measure per report; a slow, expensive report is a different product from a fast, cheap one |
Offline eval design
Build rubric trees with domain experts, not with a single prompt. The PaperBench precedent shows what rigor looks like at the limit: 20 ICML 2024 papers decomposed into 8,316 individually gradable leaf nodes, with the LLM judge itself audited against a separate JudgeEval set[6]. You do not need 8,316 nodes; you do need the two ideas, hierarchical decomposition into checkable leaves and an audited judge.
The three public benchmarks measure different things, and teams routinely confuse them:
| BrowseComp | GAIA | DeepResearch Bench |
|---|
| Task shape | Find one hard-to-locate fact | Multi-step assistant questions, 3 levels | Open-ended PhD-level research briefs |
| Output shape | Short answer | Short answer | Long cited report |
| Scoring | LLM grader for semantic equivalence to the reference | Exact-match | RACE rubric plus FACT citation checks |
| Does not measure | Synthesis, writing, citation | Synthesis, long-form quality | Retrieval persistence on adversarially hidden facts |
Pick the primary and the floor. Run DeepResearch Bench (or a domain-specific clone of its method) as the primary, with BrowseComp-style short-answer sets as the retrieval floor. Use RACE-style reference-based relative scoring: absolute scalar scores on long reports are unstable across judge runs; scoring against a gold report is not[1].
Online eval design
Three loops:
- Sampled claim-level citation audits. Pull N reports per week, extract claim-citation pairs, and have a human or a FACT-style pipeline verify support; track precision over time.
- Counterbalanced human preference pairs. When comparing model versions, show reviewers both reports with order randomized per pair, and never let one side be systematically longer without a directness control.
- Cost and latency per report with alerts on drift. Deep research runs are expensive enough that a sizeable cost regression is a product incident, not a curiosity.
Top benchmarks
- DeepResearch Bench harness: the deep_research_bench repo is Apache-2.0 and ships the RACE and FACT scoring scripts, so you can point them at your own tasks rather than reimplementing the judging[7].
- Inspect AI: the harness for running agents against task sets with full trace capture.
- Langfuse or Braintrust: production report logging and the annotation queues that citation audits need.
- Agent-as-judge: where the judge itself must browse or run tools to verify claims, a judge with tool access grades intermediate artifacts, not just final text[8].
Common pitfalls
| Pitfall | Why it bites | Fix |
|---|
| Verbosity and self-preference bias in win-rate judging | Long reports win pairwise comparisons by default, and judges prefer their own model family's prose | Counterbalance ordering, add a directness dimension to the rubric, use a judge from a different family than the generator, and report a judge-human agreement number; if you cannot state that number, the win rate is unaudited |
| Treating citation accuracy as factual correctness | FACT verifies that a source supports a claim, not that the source is right; a report can be almost perfectly citation-accurate and still materially wrong | Verify source correctness separately from citation support |
| Letting short-answer benchmarks stand in for synthesis quality | BrowseComp and GAIA measure retrieval persistence; a model can ace them and still write shallow, badly structured reports | Keep RACE-style rubric scoring as the primary and treat short-answer sets as a floor |
| Ignoring contamination on GAIA | Its validation answers are public and old enough to be in training data | Treat strong GAIA scores from recent models with suspicion[5] |
| Search-time contamination | A search-enabled agent can retrieve the benchmark's own answers, metadata, or leaderboard mid-run; measured inflation reaches 4% and the effect is widespread across public benchmarks[9] | Sandbox the eval's network access, log and audit full search trajectories, and grep them for the benchmark's own hosts before trusting a score |
Canonical reading
The DeepResearch Bench paper is the canonical read because it solves the two hard problems of this category in one method: RACE makes long-report quality scoring reproducible via reference-based relative rubrics, and FACT makes citation quality measurable at claim level[1]. The 71.33% human agreement (against a 68.44% human-to-human baseline) and the 96%/92% claim-verification alignment are the evidence that judge-based scoring can be trusted here, provided you keep auditing it.
Figure: Why LLM-as-judge scoring is trusted for deep research agents: the RACE rubric judge agrees with human experts on 71.33% of pairwise report comparisons, above the 68.44% agreement rate between the human experts themselves, and FACT citation verification aligns with human judgments on 96% of supported and 92% of unsupported claims.
Day-1 starter
- Pick 20 real research queries from your production logs. Not synthetic prompts; the queries users actually ran.
- Write 5 to 10 weighted rubric criteria per query with domain experts, plus one reference report per query to anchor relative scoring.
- Score with a judge model from a different family than the generator, and hand-verify 3 citations per report. The delta between judge citation scores and your hand checks is your judge's error bar.
Cross-cutting links