What "good" looks like
A long-context RAG eval measures two things: whether the model actually uses the long context (RULER at your production length), and whether the answer is entailed by the retrieved passages (faithfulness). Vanilla needle-in-a-haystack no longer measures either.
The ship targets on a domain set:
- RULER scores that do not degrade past your production context length[1].
- Faithfulness above 0.9 on a domain-representative set[2].
Numbers below those are real product risk, not benchmark noise.
Why not vanilla NIAH. It is saturated. Every frontier model passes it, and the score tells you almost nothing about whether the model can actually use long context for real work[3].
What RULER adds. It extends NIAH with multi-needle retrieval, multi-hop tracing, value aggregation, and frequent-word extraction. The result is a credible probe of long-context behavior that NIAH does not catch[1].
RULER is still synthetic. At 128K across 35 instruction-tuned models, HELMET measured the RULER average's Spearman correlation with real-world task categories at no more than about 0.85. The multi-key variant, which scatters distractor needles around the target, tracked real tasks better than plain NIAH, and RAG-style QA such as HotpotQA tracked them better still[4]. Use RULER as the screen and your domain set as the verdict.
The gap is the eval. The Hsieh et al. paper reports that only half of 17 models maintained satisfactory performance at 32K despite all of them passing NIAH. Pair RULER with RAGAS faithfulness, context precision, and context recall on a domain set[2].
flowchart TD
Q["Query over long context"] --> RET["Retrieve passages"]
RET --> RULER["RULER at production context length"]
RET --> FAITH["RAGAS faithfulness, context precision, context recall"]
RULER --> DEPTH["Per-depth scores: first 10%, middle 80%, last 10%"]
DEPTH --> GATE{"RULER stable past your length and faithfulness above 0.9?"}
FAITH --> GATE
GATE -->|yes| SHIP["Ship"]
GATE -->|no| FIX["Fix the weaker axis"]
Figure: The NIAH-to-RULER gap in long-context evaluation: all 17 models tested in the RULER paper pass vanilla needle-in-a-haystack, but only about half maintain satisfactory performance at 32K context. Running RULER at your production context length exposes degradation that a saturated NIAH score hides.
Secondary metrics
| Metric | What it catches | How to score |
|---|
| Position bias | The "lost in the middle" curve[5] | RULER per-depth scores as the diagnostic |
| Faithfulness | Fabricated quotes or claims not in the context | Claims in the answer entailed by the retrieved context |
| Latency and cost per query | Long context is expensive and slow | Measured per query |
| Position-stratified recall | Where the answer sits changes retrieval quality | Split by first 10%, middle 80%, last 10% and report each separately |
Offline eval design
Evaluate at your production length, not the model's max. Run RULER at your production context length. If you ship 32K, evaluate at 32K.
Build a span-verifiable domain set. Build a 50-query domain test set with span-level gold answers (start and end character offsets in the source) so substring presence is verifiable.
Put distractors in and loosen the question-answer match. Chroma's context-rot study ran 18 models and found that every one got worse as input length grew, even on simple tasks. Degradation set in faster when the question and the passage holding the answer were less semantically similar, and even a single topically related distractor lowered scores, with four compounding the loss[6]. A domain set whose questions echo the source over a clean haystack will overstate what your users get. Write queries the way users phrase them, not the way the document does, and seed the corpus with near-miss passages.
Score the generation step with RAGAS. Use RAGAS faithfulness, context precision, context recall, and response relevancy[7]. The faithfulness metric specifically scores whether claims in the answer are entailed by the retrieved context; it is the single most useful number in this category.
Stratify by question type. Different failure modes show up in different cells:
- Extractive: answer is a direct span.
- Abstractive: answer requires synthesis.
- Multi-hop: answer requires two or more spans.
- Aggregation: answer requires counting or summing.
Online eval design
Log the retrieval evidence. Log retrieved chunks and cited spans for every query. Require span citation in the output and verify substring presence; a citation that does not appear in the retrieved context is a fabrication.
Alert on drift. Alert on citation-miss rate above 2% and on faithfulness drops on a continuously sampled subset.
Stratify online too. Stratify online metrics by query length and corpus size; both shift the cost-quality frontier.
Top benchmarks
- RULER, https://arxiv.org/abs/2404.06654. 13 tasks beyond NIAH; the current standard for long-context probing.
- RULERv2, https://neurips.cc/virtual/2025/122399. A NeurIPS 2025 workshop paper by Hsieh, Ladhak, Puvvada, and Ginsburg, also carried in the NVIDIA RULER repo. Difficulty steps up from basic retrieval to problem solving across multi-key NIAH, multi-value NIAH, and multi-doc QA. Across 31 models (seven closed-source, 24 open-weight) every model degraded with length, including those claiming million-token windows, and explicit retrieve-then-solve decomposition beat answering in one step[8].
- Lost in the Middle, https://arxiv.org/abs/2307.03172. The position-bias canon; not a benchmark you run so much as a diagnostic frame.
- HELMET, https://arxiv.org/abs/2410.02694. Seven application-centric categories at up to 128K tokens, run over 59 models; the evidence that synthetic scores need a downstream check[4].
- LongBench v2, used selectively when you need multi-document reasoning beyond what RULER probes; 503 multi-choice questions, 8K to 2M words[9].
- LongBench Pro, https://arxiv.org/abs/2601.02872. 1,500 naturally occurring English and Chinese samples across 11 primary tasks, 8K to 256K tokens, so real documents rather than generated haystacks. It reports that usable context lengths fall short of vendor claims[10].
- RULER (NVIDIA repo): the canonical implementation. Its README now marks the original Docker and
run.sh instructions as deprecated and points to the rulerv1-ns and rulerv2-ns branches, which run through NVIDIA NeMo-Skills[11].
- Ragas (over 15k stars; the repo now lives under the
vibrantlabsai GitHub organization): the generation-side metrics; faithfulness is the single most-used scorer in this category[7][12].
- Langfuse or LangSmith: production trace logging with chunk-level capture.
- Inspect AI: sandboxed, reproducible long-context evals across releases.
Common pitfalls
| Pitfall | Why it bites | Fix |
|---|
| Reporting only vanilla NIAH | Saturated, tells you almost nothing about real long-context behavior[3] | Run RULER, which exists to fix exactly this gap[1] |
| Conflating retrieval recall with answer faithfulness | A query can retrieve the right chunks and still produce a fabricated answer | Measure both |
| Skipping position-stratified analysis | Aggregate scores hide the real, product-meaningful lost-in-the-middle curve[5] | Report per-depth scores |
| Evaluating at the model's max context | The relevant number is the one at the length your users actually hit | Evaluate at your production context length |
Canonical reading
RULER is the canonical paper because it operationalizes the gap between NIAH and useful long-context behavior with 13 task categories[1]. The blunt line from the abstract is the one to remember: "almost all models exhibit large performance drops as context length increases" despite near-perfect NIAH scores.
Read it with Lost in the Middle[5] for the position-bias framing, and the RAGAS paper for the faithfulness scoring methodology[2].
Day-1 starter
- Build a 50-query test set over your real corpus with span-level gold answers (start and end offsets). This is the artifact that makes everything else credible.
- Run NIAH-multi (3 needles), multi-hop, and aggregation per RULER at your production context length, not the model's max. Report per-depth scores so the position bias is visible.
- Wire RAGAS faithfulness and context-precision to a CI job. Fail the build on a faithfulness drop greater than 5 points relative to the prior release.
Cross-cutting links