The number on the spec sheet (200K, 1M, 2M tokens) is the advertised context. The number your application can actually rely on is smaller, often a lot smaller, and depends on the task. The benchmark literature since 2023 has converged on this distinction, with RULER, BABILong, ∞Bench, NoCha, NoLiMa, MRCR, and HELMET each probing different failure modes. This page is the map of which benchmark answers which question for a RAG system.
The honest framing
Long-context evaluation is not one task. It is at least four: single-needle retrieval, multi-needle retrieval, multi-hop reasoning across distant chunks, and aggregation or counting across the entire context. Each has a different effective-context profile [1]. NIAH became the default early benchmark [2], and single-needle retrieval is saturated: RULER's authors saw near-perfect vanilla NIAH accuracy from models that still dropped sharply on their harder tasks as length grew, and HELMET found NIAH saturated for almost all the models it tested and found that synthetic tasks like NIAH do not reliably predict downstream performance [1][3]. A model that passes NIAH at 200K is not a model that can reason over 200K. The literature has moved on, and so should your eval.
RULER
RULER (NVIDIA, Hsieh et al., COLM 2024) is the benchmark to run first for finding the length where a model falls off. It generalizes NIAH into thirteen tasks across retrieval, multi-hop tracing, aggregation, and QA. The headline finding: of 17 models that all claimed 32K or more, only half held up at 32K. Effective context length is the longest input where the average score stays above a fixed bar, 85.6 (Llama2-7B's score at 4K), not a percentage of each model's own short-context score [1]. Run RULER on the chunking strategy and prompt you actually deploy.
Read the repo's results table by model version, not by vendor. As fetched 2026-10-01, it still lists most 128K-claimed models (GPT-4-1106-preview, Llama3.1-70B, Mistral-Large-2411) at an effective length of 64K or below, and its newest rows (Qwen3, EXAONE 4.0) are starred as numbers reported by the model authors rather than rerun by NVIDIA [4]. Clone the rulerv1-ns branch for the original thirteen-task pipeline: the run instructions on the main branch are struck through as deprecated, and a rulerv2-ns branch carries a second version built on NeMo-Skills [4].
BABILong
BABILong (Kuratov et al., NeurIPS 2024 Datasets and Benchmarks) embeds bAbI-style reasoning tasks inside long PG-19 passages, with splits up to 10 million tokens. The probes require identifying the few sentences that matter from a much longer passage and reasoning over them. The failure mode exposed: a model can find the relevant sentence and still reason incorrectly because surrounding context distracts the attention pattern [5]. The paper's headline: popular LLMs effectively use only 10 to 20 percent of the context, and a RAG baseline plateaus near 60 percent on single-fact questions whatever the length, so retrieval is not a free fix [5]. BABILong is the right benchmark when your application stuffs context and then reasons.
∞Bench
∞Bench (Zhang et al., 2024) packages twelve tasks at average length above 100K across English and Chinese, mixing retrieval, code understanding, math, and novel QA. The realism is the contribution: real documents instead of synthetic noise [6]. Use ∞Bench when you want one number that approximates "how well does the model handle realistic long inputs."
NoCha
NoCha (Karpinska et al., EMNLP 2024) tests true-or-false claims about full-length novels (often 100K-plus tokens), requiring whole-book understanding rather than isolated retrieval. It is scored per minimal pair: a model earns the point only when it gets both the true claim and its false twin right, so random guessing lands at 25 percent. GPT-4o scored 55.8 percent on the original release and no open-weight model beat chance [7]. As fetched 2026-10-01, the leaderboard (last updated July 2025) puts gpt-5 first at about 68.5 percent on the pairs it attempted, just ahead of o3 at 67.8, so the best entry still misses roughly a third of the pairs [8]. The right benchmark when the application asks holistic questions about long documents (legal contracts end-to-end, books, multi-chapter compliance).
NoLiMa
NoLiMa (Modarressi et al., ICML 2025) strips lexical overlap between query and needle. Where NIAH and earlier benchmarks let the model use direct token matches, NoLiMa forces a one-hop semantic association. Of 13 models that claim at least 128K of context, 11 dropped below half of their short-context baseline at 32K, and GPT-4o fell from 99.3 to 69.7 [9]. NoLiMa's own effective-length column uses a relative bar (the longest context that keeps 85 percent of the model's base score), unlike RULER's fixed 85.6. In the repo's table, last updated July 2025, GPT-4.1 advertises 1M tokens, holds that bar only to 16K, and scores 64.7 at 128K [10]. The benchmark that exposes whether your application leans on lexical match or on actual semantic understanding. The fix is usually better embeddings or hybrid lexical-plus-dense retrieval, not a model swap.
MRCR
OpenAI's MRCR (multi-round co-reference resolution) buries 2, 4, or 8 identical requests in a synthetic multi-turn conversation and asks for one specific instance, in context bins from 4,096 to 1,048,576 tokens. The score is the difflib SequenceMatcher ratio, and an answer without the required hash prefix scores 0 [11]. It tests what single-needle NIAH cannot: telling apart eight look-alike items when the user meant one of them. Check the dataset version before you compare scores. The v2 release of December 5, 2025 fixed generation errors that gave roughly 10 percent of datapoints excess target needles and roughly 5 percent an incorrect ground truth [11]. Anthropic's Opus 4.6 announcement reports 76 percent on the 8-needle 1M variant against 18.5 percent for Sonnet 4.5 [12]. That is a vendor-reported number, but the spread says effective context moves a lot between model versions, so rerun the battery below on every model swap.
HELMET
HELMET (Yen et al., ICLR 2025) bundles seven application-centric categories at controllable lengths up to 128K, scores with model-based metrics, and was run on 59 models. Two findings matter to a RAG team: synthetic tasks like NIAH do not reliably predict downstream performance, and the categories correlate weakly with each other, so no single long-context number stands in for the rest. The authors recommend their RAG tasks for fast model development because they are cheap to run and better predict the other categories [3]. Pair RULER's length cliff with a RAG-shaped test: HELMET's RAG tasks, or your own gold-chunk set from the plan below.
A test plan
The practical battery: run RULER at the deployed context length; sample a hundred production queries with labeled gold chunks; score Recall@K and MRR at production K; construct a position-stratified subset (gold at depth 0-10%, 40-60%, 90-100%) to check the lost-in-the-middle dip [13]; build a paraphrased-queries subset that strips lexical overlap with the gold chunk; build a distractor subset that adds one or four topically related chunks that do not answer the query. Chroma's context-rot study (18 models, July 2025) found that even a single distractor lowers accuracy against a needle-only baseline, four compound the loss, and the effect amplifies as input grows [14].
# position_strata.py
def stratify_by_depth(eval_set: list[dict], context_tokens: int) -> dict:
bands = {"early": [], "middle": [], "late": []}
for ex in eval_set:
depth = ex["gold_chunk_offset"] / context_tokens
if depth < 0.10:
bands["early"].append(ex)
elif 0.40 <= depth <= 0.60:
bands["middle"].append(ex)
elif depth > 0.90:
bands["late"].append(ex)
return bands
A clean stratification turns lost-in-the-middle into a number. An early-band score of 0.92 and middle-band of 0.74 reveals a position bias no headline metric will surface.
Figure: Position-stratified long-context RAG eval: bucketing queries by gold-chunk depth (0-10%, 40-60%, 90-100%) turns lost-in-the-middle into a number. Recall@K drops 0.18, from 0.92 in the early band to 0.74 in the middle band, a position bias that a single headline metric hides.
Stuff vs retrieve
The benchmarks settle this for most cases: retrieve. The effective window where a model preserves multi-hop accuracy is much shorter than the advertised window, and Anthropic's own context-window guide now says accuracy and recall degrade as token count grows [15]. Stuff context only when the application requires whole-document reasoning retrieval cannot decompose [16]. LongBench v2 is the realistic test for that case: 503 multiple-choice questions over contexts from 8K to 2M words, where human experts scored 53.7 percent under a 15-minute limit [16]. As fetched 2026-10-01, its public leaderboard puts Gemini-2.5-Pro first at 63.3 percent with chain of thought, but the newest entry is dated 2025-07-28, so run the questions yourself on the model you deploy [17].
Cost compounds the answer. As of October 2026, a stuffed 200K prompt carries 50x the input tokens of a retrieved 4K prompt. Anthropic bills the full 1M window at the standard per-token rate for Claude 4.6 and later, so the ratio stays at 50x there. Gemini 3.1 Pro Preview charges 4.00 for longer ones, so a stuffed prompt past 200K costs roughly 100x a retrieved 4K prompt [18][19]. Prompt caching narrows the gap when the same document repeats across queries: cache reads cost 10 percent of the base input price on most Claude models [18]. Run both arms, stuffed and retrieved, on the same gold set before you pick.
TIP
Run RULER at the actual context length you deploy, not at the model's headline max. The advertised 1M context is informational; the 32K, 64K, or 128K you push in production is the number that needs the benchmark.