What "good" looks like
GraphRAG is scored by query class, not by one number. Global sensemaking questions get pairwise LLM-judged win rate on comprehensiveness, diversity, and empowerment; local factual questions get standard retrieval-plus-faithfulness metrics. A system that wins one class and loses the other is the normal case, so any single-number comparison against vector RAG hides the result.
The targets, split by class:
- Comprehensiveness win rate vs vector RAG: 72-83% on podcasts, 72-80% on news[1].
- Diversity win rate: 75-82% on podcasts, 62-71% on news[1].
- Answer order counterbalanced across trials[2], with directness as a control metric[1].
The reference protocol. The Edge et al. design is win/lose/tie pairwise judging on the three target criteria, with directness as a control and each comparison replicated and averaged. The paper does not describe counterbalancing answer order; BenchmarkQED's AutoE adds that, presenting each pair in counterbalanced order[2]. The win rates above come from two corpora: a roughly 1M-token podcast transcript set and a roughly 1.7M-token news set[1].
Scope guard. Retrieval metrics themselves (Recall@K, MRR, NDCG) are covered in retrieval metrics and embeddings and retrieval; faithfulness basics live in generation faithfulness. This page owns what changes when retrieval goes through a graph: query-class splits, reference-free pairwise judging, graph-construction ceilings, and index cost.
The load-bearing idea is the query-class split. BenchmarkQED's AutoQ formalizes a 2x2: DataLocal, ActivityLocal, DataGlobal, ActivityGlobal. In Microsoft's published comparison, LazyGraphRAG won all 96 same-model comparisons, all but one (relevance on DataLocal queries against 1M-token vector RAG) at statistical significance[2].
Winners flip by class. The Han et al. head-to-head shows why the split cannot be skipped. GraphRAG in local mode wins multi-hop questions, vector RAG wins single-hop detail questions, and GraphRAG in global mode struggles on detail questions and does poorly on null queries, which should be answered as insufficient information; the authors read that as added hallucination risk when the corpus holds no answer[3].
The headline win rates are a global-sensemaking result. On query-based summarization sets with human reference summaries (QMSum, ODSum, SQuALITY), Han et al. found vector RAG generally closer to the references than community-based GraphRAG, global search especially. In their pairwise judging, RAG won comprehensiveness against GraphRAG global while GraphRAG global won diversity[3]. The 72-83% figures hold for corpus-wide questions, not for questions about one role or event.
flowchart TD
Q["Incoming query"] --> CLS{"Local or global?"}
CLS -->|local factual| LOC["Retrieval plus faithfulness metrics"]
CLS -->|global sensemaking| GLO["Pairwise win rate: comprehensiveness, diversity, empowerment"]
LOC --> COV{"Answer entities in the KG?"}
GLO --> CB["Counterbalance order, directness as control"]
COV -->|no| CEIL["Recall capped by extraction"]
COV -->|yes| RETR["Retrieval can find it"]
CB --> REP["Report win rate per class, never pooled"]
RETR --> REP
CEIL --> REP
REP --> GATE{"Wins global only?"}
GATE -->|yes| ROUTE["Route by class, keep vector baseline"]
GATE -->|both| SHIP["Ship broader"]
Figure: GraphRAG vs vector RAG pairwise win rates from Edge et al. 2024: comprehensiveness 72-83% on the podcast corpus and 72-80% on news, diversity 75-82% on podcasts and 62-71% on news, all clearing the 50% coin-flip line.
Secondary metrics
| Metric | What it catches | How to score |
|---|
| Graph-construction entity coverage | Answer entities never extracted into the KG, a hard recall ceiling | Sample gold answers, check whether their entities exist in the graph. In one verified example only about 65.8% of answer entities existed in the KG built for HotpotQA, capping achievable recall before any retrieval[3][4] |
| Context tokens per query | Whether the graph pays its keep on cost | Root-level community summaries needed over 97% fewer context tokens than map-reduce over source text; low-level summaries 26-33% fewer[1] |
| Null-query hallucination rate | Fabricated answers on unanswerable queries; global mode is the offender | Share of unanswerable queries where the system answers instead of abstaining[3] |
| Indexing spend and staleness | The graph is a build artifact that decays as the corpus moves | Track indexing token spend and re-index lag alongside the per-query numbers |
A note on RAGAS transfer. Faithfulness and response relevancy transfer cleanly to local and extractive queries; run them there as usual[5]. Context precision and context recall do not transfer to global sensemaking questions, because there is no chunk-level gold context to score against when retrieval returns community summaries rather than source passages[6]. That gap is why the field shifted to reference-free pairwise judging for the global class. BenchmarkQED now also offers a judge-free route: AutoQ generates assertions (unit-test style factual statements that need manual review), and AutoE scores answers against them, with hierarchical assertions for global questions and a chunk-level mode that checks retrieved chunks against the assertions before synthesis[7].
Offline eval design
Build a comparison table: query class x metric x which-system-wins. Synthesize queries across the 2x2 (BenchmarkQED's AutoQ automates this against your own corpus), run each candidate system and your vector RAG baseline on the same set, and judge pairwise with counterbalanced answer order[2]. Report win rate per class per criterion; never pool classes.
Keep directness as a control. If the winning system also scores lower on directness, verbosity is doing some of the winning[1].
Add evidence-chain retrieval. MultiHop-RAG provides inference, comparison, temporal, and null query types with gold evidence chains[8]. The null type is the one most teams skip and most need.
Audit the graph itself, separately. Sample gold answers, check whether their entities and relations exist in the constructed KG, and report coverage as its own number. A retrieval fix cannot recover what extraction dropped.
Online eval design
Dashboard by query class, not in aggregate. Classify incoming queries into the 2x2 (a cheap classifier is fine) and track win-rate proxies, abstention rate, and latency per class.
Monitor null-query hallucination explicitly. Sample queries the system answered confidently and check whether the corpus supports any answer.
Track cost per correct answer with indexing amortized in. The graph must pay its build cost back in per-query context savings, and the root-level summary numbers show it can[1].
Alert on re-index staleness. The gap between corpus updates and the last graph build is silent recall decay.
Top benchmarks
- BenchmarkQED, https://www.microsoft.com/en-us/research/blog/benchmarkqed-automated-benchmarking-of-rag-systems/. AutoQ query synthesis across the 2x2, AutoE judging (pairwise, reference-based, or assertion-based), AutoD corpus prep[2][7].
- MultiHop-RAG, https://arxiv.org/abs/2401.15391. Evidence-chain retrieval with inference, comparison, temporal, and null query types[8].
- GraphRAG-Bench, https://arxiv.org/abs/2506.05690. Tasks of rising difficulty (fact retrieval, complex reasoning, contextual summarization, creative generation), scored across the pipeline from graph construction to generation. Its premise is that GraphRAG frequently underperforms vanilla RAG on real-world tasks, which is the reason to keep a vector baseline in every comparison[9].
- Edge et al. corpora, https://arxiv.org/abs/2404.16130. The podcast and news corpora plus the pairwise protocol that defined the category[1].
- Microsoft GraphRAG: the reference indexing and query pipeline, with local and global modes that map to the query classes. The README now says the project is largely in maintenance mode: bug fixes and dependency updates, no new features or PRs. The latest release is v3.2.0 (2026-09-24), and v3 deliberately slimmed the core library, so pin the version your eval ran on[10]. The docs also describe a Basic Search mode (plain top-k vector search), which gives you the baseline arm over the same corpus[11].
- BenchmarkQED: query synthesis and counterbalanced judging against your own corpus.
- RAGAS: the local-query half of the eval, where faithfulness and response relevancy still apply[5].
- Langfuse: production traces with per-class tagging, token accounting per query, and sampled review queues for null-query audits.
Common pitfalls
| Pitfall | Why it bites | Fix |
|---|
| Verbosity and position bias in win-rate judging | Judges reward longer answers and are sensitive to which answer comes first; Han et al. reversed the presentation order on summarization sets and got substantially different, sometimes opposite, verdicts[3] | Counterbalance answer order over an even number of trials, as AutoE does[7], and keep directness as a control metric, as Edge et al. do[1] |
| Evaluating only one query class | Global-mode strengths on sensemaking mask failures on detail questions and null queries; a global-only eval ships a system that hallucinates on unanswerable questions | Split by class; the Han et al. results show winners flip by class[3] |
| Ignoring index cost and staleness | The graph is expensive to build and goes stale as the corpus changes; scoring query quality alone prices half the system | Track indexing spend and re-index lag alongside query metrics |
| Treating extraction as free | Entity-coverage ceilings mean graph-construction quality bounds everything downstream | Measure coverage directly rather than inferring it from end-to-end scores[3] |
Canonical reading
Edge et al. (2024) is the canonical paper: it defines the local-vs-global distinction, the community-summary architecture, and the reference-free pairwise protocol with a directness control that the rest of the field adopted[1]. Read it for the eval design as much as the system design. Follow with BenchmarkQED for the automated, run-it-on-your-corpus version of the same protocol, with counterbalanced answer order built in[2]. Then read Han et al. for what the protocol does not show: results on detail questions, null queries, and reference-based summarization[3].
Day-1 starter
- Synthesize about 40 queries across the 2x2 classes (DataLocal, ActivityLocal, DataGlobal, ActivityGlobal) against your own corpus, plus a handful of null queries with no corpus-supported answer.
- Run counterbalanced pairwise judging of GraphRAG against your existing vector RAG baseline on comprehensiveness, diversity, and empowerment, with directness as the control.
- Report win rate per class, context tokens per query, and total indexing tokens. If GraphRAG only wins the global classes, that is the expected result; route by class rather than replacing the baseline.
Cross-cutting links