What "good" looks like
GraphRAG is scored by query class, not by one number. Global sensemaking questions get pairwise LLM-judged win rate on comprehensiveness, diversity, and empowerment; local factual questions get standard retrieval-plus-faithfulness metrics. A system that wins one class and loses the other is the normal case, so any single-number comparison against vector RAG hides the result.
The targets, split by class:
- Comprehensiveness win rate vs vector RAG: 72-83% on podcasts, 72-80% on news[1].
- Diversity win rate: 75-82% on podcasts, 62-71% on news[1].
- Answer order counterbalanced, with directness as a control metric[1].
The reference protocol. The Edge et al. design is win/lose/tie pairwise judging on the three target criteria, with directness as a control and answer order counterbalanced to neutralize position bias. Those win rates come from two corpora: a roughly 1M-token podcast transcript set and a roughly 1.7M-token news set[1].
Scope guard. Retrieval metrics themselves (Recall@K, MRR, NDCG) are covered in retrieval metrics and embeddings and retrieval; faithfulness basics live in generation faithfulness. This page owns what changes when retrieval goes through a graph: query-class splits, reference-free pairwise judging, graph-construction ceilings, and index cost.
The load-bearing idea is the query-class split. BenchmarkQED's AutoQ formalizes a 2x2: DataLocal, ActivityLocal, DataGlobal, ActivityGlobal. In Microsoft's published comparison, LazyGraphRAG won all 96 same-model comparisons, all but one (relevance on DataLocal against vector RAG) at statistical significance[2].
Winners flip by class. The Han et al. head-to-head shows why the split cannot be skipped. GraphRAG in local mode wins multi-hop questions, vector RAG wins single-hop detail questions, and GraphRAG in global mode fails both detail questions and null queries, hallucinating an answer when none exists in the corpus[3].
flowchart TD
Q["Incoming query"] --> CLS{"Local or global?"}
CLS -->|local factual| LOC["Retrieval plus faithfulness metrics"]
CLS -->|global sensemaking| GLO["Pairwise win rate: comprehensiveness, diversity, empowerment"]
LOC --> COV{"Answer entities in the KG?"}
GLO --> CB["Counterbalance order, directness as control"]
COV -->|no| CEIL["Recall capped by extraction"]
COV -->|yes| RETR["Retrieval can find it"]
CB --> REP["Report win rate per class, never pooled"]
RETR --> REP
CEIL --> REP
REP --> GATE{"Wins global only?"}
GATE -->|yes| ROUTE["Route by class, keep vector baseline"]
GATE -->|both| SHIP["Ship broader"]
Figure: GraphRAG vs vector RAG pairwise win rates from Edge et al. 2024: comprehensiveness 72-83% on the podcast corpus and 72-80% on news, diversity 75-82% on podcasts and 62-71% on news, all clearing the 50% coin-flip line with answer order counterbalanced.
Secondary metrics
| Metric | What it catches | How to score |
|---|
| Graph-construction entity coverage | Answer entities never extracted into the KG, a hard recall ceiling | Sample gold answers, check whether their entities exist in the graph. In one verified example only about 65.8% of answer entities existed in the KG built for HotpotQA, capping achievable recall before any retrieval[3][4] |
| Context tokens per query | Whether the graph pays its keep on cost | Root-level community summaries needed over 97% fewer context tokens than map-reduce over source text; low-level summaries 26-33% fewer[1] |
| Null-query hallucination rate | Fabricated answers on unanswerable queries; global mode is the offender | Share of unanswerable queries where the system answers instead of abstaining[3] |
| Indexing spend and staleness | The graph is a build artifact that decays as the corpus moves | Track indexing token spend and re-index lag alongside the per-query numbers |
A note on RAGAS transfer. Faithfulness and response relevancy transfer cleanly to local and extractive queries; run them there as usual[5]. Context precision and context recall do not transfer to global sensemaking questions, because there is no chunk-level gold context to score against when retrieval returns community summaries rather than source passages[6]. That gap is why the field shifted to reference-free pairwise judging for the global class.
Offline eval design
Build a comparison table: query class x metric x which-system-wins. Synthesize queries across the 2x2 (BenchmarkQED's AutoQ automates this against your own corpus), run each candidate system and your vector RAG baseline on the same set, and judge pairwise with counterbalanced answer order[2]. Report win rate per class per criterion; never pool classes.
Keep directness as a control. If the winning system also scores lower on directness, verbosity is doing some of the winning[1].
Add evidence-chain retrieval. MultiHop-RAG provides inference, comparison, temporal, and null query types with gold evidence chains[7]. The null type is the one most teams skip and most need.
Audit the graph itself, separately. Sample gold answers, check whether their entities and relations exist in the constructed KG, and report coverage as its own number. A retrieval fix cannot recover what extraction dropped.
Online eval design
Dashboard by query class, not in aggregate. Classify incoming queries into the 2x2 (a cheap classifier is fine) and track win-rate proxies, abstention rate, and latency per class.
Monitor null-query hallucination explicitly. Sample queries the system answered confidently and check whether the corpus supports any answer.
Track cost per correct answer with indexing amortized in. The graph must pay its build cost back in per-query context savings, and the root-level summary numbers show it can[1].
Alert on re-index staleness. The gap between corpus updates and the last graph build is silent recall decay.
Top benchmarks
- Microsoft GraphRAG: the reference indexing and query pipeline, with local and global modes that map to the query classes.
- BenchmarkQED: query synthesis and counterbalanced judging against your own corpus.
- RAGAS: the local-query half of the eval, where faithfulness and response relevancy still apply[5].
- Langfuse: production traces with per-class tagging, token accounting per query, and sampled review queues for null-query audits.
Common pitfalls
| Pitfall | Why it bites | Fix |
|---|
| Verbosity and position bias in win-rate judging | Judges reward longer answers and first-position answers | Counterbalance answer order and keep directness as a control metric, as the original protocol does[1] |
| Evaluating only one query class | Global-mode strengths on sensemaking mask failures on detail questions and null queries; a global-only eval ships a system that hallucinates on unanswerable questions | Split by class; the Han et al. results show winners flip by class[3] |
| Ignoring index cost and staleness | The graph is expensive to build and goes stale as the corpus changes; scoring query quality alone prices half the system | Track indexing spend and re-index lag alongside query metrics |
| Treating extraction as free | Entity-coverage ceilings mean graph-construction quality bounds everything downstream | Measure coverage directly rather than inferring it from end-to-end scores[3] |
Canonical reading
Edge et al. (2024) is the canonical paper: it defines the local-vs-global distinction, the community-summary architecture, and the counterbalanced pairwise protocol with a directness control that the rest of the field adopted[1]. Read it for the eval design as much as the system design. Follow with BenchmarkQED for the automated, run-it-on-your-corpus version of the same protocol[2].
Day-1 starter
- Synthesize about 40 queries across the 2x2 classes (DataLocal, ActivityLocal, DataGlobal, ActivityGlobal) against your own corpus, plus a handful of null queries with no corpus-supported answer.
- Run counterbalanced pairwise judging of GraphRAG against your existing vector RAG baseline on comprehensiveness, diversity, and empowerment, with directness as the control.
- Report win rate per class, context tokens per query, and total indexing tokens. If GraphRAG only wins the global classes, that is the expected result; route by class rather than replacing the baseline.
Cross-cutting links