Contamination is the case where a model has seen the eval data during training. The headline score becomes a measure of memorization, not capability. The discipline has three layers: prove contamination when you suspect it, mitigate by using held-out or rephrased variants, and report honestly when you cannot prove either way.
This page is the compressed version; the datasets section covers construction, versioning, and contamination at chapter length.
Why contamination matters
A benchmark score is a claim about generalization. Contamination breaks the claim. A model that memorized MMLU during pretraining produces high numbers without telling you anything about novel reasoning 1. Anthropic makes the same point about MMLU: it is so widely used that models are likely to have seen its questions in training, so scores need careful handling when compared across labs 2.
Known-contaminated benchmarks. MMLU shows signs of memorization: the MMLU-CF authors report that some LLMs, prompted with an MMLU question alone, reproduce the original answer choices 3. MMLU-Redux adds a second problem, estimating that 6.49% of MMLU questions contain ground-truth errors 4. HumanEval has near-verbatim solutions on the open web; HumanEval+ extends its tests by 80x and exposes wrong code that passed the original sparse tests, which fixes weak tests rather than contamination 5. GSM8K and MATH have published solutions; leaderboard scores are inflated. LiveCodeBench addresses contamination by stamping each problem with its release date and reporting per-cutoff scores 6.
The default position in 2026 is to treat any pre-2024 widely-cited benchmark as contaminated unless the model card explicitly demonstrates otherwise.
Five detection signals
Each has limits; the strongest evidence is concordance across two or more.
N-gram match. Compute 13-gram or 20-gram overlap between the eval set and the training corpus. Direct overlap above a few percent is hard evidence. Limit: most labs do not publish training corpora, so runnable only for open-data models (Pythia, OLMo, some Llama subsets).
Canary strings. The benchmark author plants a unique string in every benchmark file. BIG-bench did this with a canary GUID, meant both to let training pipelines filter the data out and to diagnose after the fact whether a model trained on it 7. If a probe finds the string in the model's outputs under a "complete this sequence" prompt, the model has seen the eval set.
Perplexity tests. Memorized eval items have lower perplexity than novel items from the same distribution. Compute perplexity on the eval and on a held-out reference set with matched distribution; a gap is contamination evidence.
Output-distribution tests. For multiple-choice items, memorized items show concentration on the correct answer above the model's calibration baseline.
Held-out variant comparison. Rephrase or freshly collect a variant; compare. A drop of more than 10 points on the variant relative to the headline indicates the original is contaminated. MMLU-CF applies this corrective: it rephrases questions, shuffles options, and keeps its test set closed 3.
The MMLU-CF rephrase pattern
The simplest mitigation a team can run. MMLU-CF, the benchmark it is named for, uses three decontamination rules: rephrase the question, shuffle the options, and replace one option with "None of the other choices" half the time. Its 10,000-question test set stays closed, with a separate public validation set 3. GPT-4o scores 73.4 percent (5-shot) on that closed test set against 88.0 percent on MMLU. That gap mixes contamination with a harder, differently sourced set, which is why rephrasing the same items, as below, is the cleaner test.
- Sample 100 items from the public benchmark.
- Use a frontier model to rephrase: change wording, reorder, paraphrase the options.
- Have the principal expert verify each rephrasing preserves the original answer.
- Score the candidate model on the rephrased subset.
- Compare to the score on the matched original subset.
A 10-point or larger drop on the rephrased version indicates the original is contaminated.
Figure: Contamination detection with the MMLU-CF rephrase pattern: sample 100 items from the public benchmark, rephrase them with a frontier model, expert-verify the answers, and score the rephrased subset against the matched originals. A rephrased score within 5 points of the headline means contamination is probably small; a drop of 10 points or more marks the headline number as contaminated and the rephrased score becomes the primary capability signal.
Canary insertion for private benchmarks
When building an internal benchmark, plant canaries before sharing it anywhere external.
import hashlib
def make_canary(secret: str, item_id: str) -> str:
h = hashlib.sha256(f"{secret}:{item_id}".encode()).hexdigest()[:16]
return f"CANARY_{h}_END"
def insert_canaries(items, secret):
return [{**item, "context": f"{item['context']} ({make_canary(secret, item['id'])})"}
for item in items]
def probe_for_canary(completion, secret, item_id):
return make_canary(secret, item_id) in completion
The probe prompts the model with the canary prefix and checks for the planted suffix. A hit is evidence the model has seen the benchmark.
Held-out sets as the only real defense
Detection is reactive. The proactive defense is a private held-out set the model has never been able to see. Private sets are expensive and decay with use, which is why Jacovi et al. also recommend encrypting any test data you do publish, under a license that disallows derivative distribution 1. Rules:
The set lives in a private repo. Access is logged. The set is never copied into a notebook, a slack message, a screenshot, or a publicly accessible chat.
Evaluations happen inside a sandboxed runner that returns only aggregate scores, not per-item completions. This prevents accidental exfiltration.
The set is rotated annually. Old items are retired and published; new items are collected and locked.
What MixEval and LiveCodeBench do
Two benchmarks are explicit about contamination. MixEval matches queries mined from the web to similar queries in existing benchmarks, so the mix follows real user query distributions, and it refreshes through a data update pipeline the authors call dynamic evaluation 8. LiveCodeBench publishes each problem with its release date; a model trained before cutoff X is evaluated only on problems released after X, eliminating contamination by construction 6. The shared pattern is structural changes that price contamination out.
"By construction" holds only while the benchmark keeps adding problems. As of October 2026 the latest public LiveCodeBench release is release_v6, covering problems from May 2023 to April 2025 (1,055 problems) 9. A model with a later training cutoff has no unseen problems left in the public set, so check the release version against the model's cutoff before calling a LiveCodeBench score contamination-free.
A practical detection pipeline
For a team reporting a defensible public-benchmark number:
- Take the public benchmark headline. Note it as the "potentially-contaminated baseline."
- Generate an MMLU-CF-style rephrased subset. Score the model on it.
- If the rephrased score is within 5 points of the headline, contamination is probably small. Report both.
- If the rephrased score is more than 10 points below, the headline is contaminated. Report the rephrased number as primary and the headline as baseline-for-comparison only.
Reporting template: "Model X scored 78% on MMLU (potentially contaminated baseline) and 65% on a 100-item rephrased subset constructed in October 2026; we report the rephrased number as the capability signal."
Saturation and contamination, both at once
Most pre-2024 widely-cited benchmarks are saturated and contaminated. HumanEval, MATH, MMLU, MGSM, BBH, HellaSwag, DROP, and NIAH show 90%+ from frontier models, and most show contamination or test-quality problems 5 4. The corrective for both is the same: switch to non-saturated, contamination-resistant alternatives. MMLU-Pro, GPQA Diamond, LiveCodeBench, FrontierMath, ARC-AGI-2 are the replacements usually named. One newer entry: ARC-AGI-3, launched on 25 March 2026 as interactive game-style environments with no instructions or stated goals, where frontier AI scored under 1 percent on the launch page 10. Check each candidate's headroom before adopting it; replacements saturate too. The foundations: why LLM evals are hard chapter covers the saturation question.
What good looks like
A team reporting public-benchmark scores responsibly ships the headline number labeled as potentially contaminated, a rephrased-subset number labeled as the capability signal, a canary-planted private held-out set used for internal release decisions and never published, and a model-card statement on contamination methodology with the date of the rephrased construction. The cookbook at /cookbook/contamination-detection walks the runnable version.