Public benchmarks contaminate. Any text published before a model's training cutoff has a non-zero chance of sitting in the training corpus, and a score on it is then a lower bound on memorization rather than an upper bound on capability 1. The two defenses that work without access to the training data are date-windowing (only score problems published after the cutoff, the LiveCodeBench pattern 2) and overlap detection (MinHash similarity between your problems and public mirrors of them 3). This recipe runs both on a seeded toy corpus so you can verify the mechanics, then tells you exactly what to swap to run it on your own set.
The hour breaks down as: ten minutes to run the toy pipeline and check your output against this page, twenty to wire in your own benchmark and mirror snapshots, thirty to read the flags and decide what the headline number is.
What you build
Two modules:
make_benchmark.py writes a 60-problem benchmark with publication dates, a fake "public mirror" (six pages containing 16 problems verbatim and 8 paraphrased), and a synthetic results file where the model scores 90% on leaked problems and 50% on clean ones. Seeded, so your numbers match this page.contamination_check.py indexes the mirror in a MinHash LSH table, flags every problem that collides, applies per-model date windows from a cutoff inventory, splits the score by flag status, and writes benchmark_clean.jsonl.
Step 1: the toy corpus
"""Build a toy benchmark plus a fake public mirror, with planted contamination.
Run:
pip install datasketch pandas numpy
python make_benchmark.py
"""
import json
import os
import numpy as np
rng = np.random.default_rng(7)
TEMPLATES = [
"A train leaves city {a} at {n} km/h while a second train leaves city {b} at {m} km/h; after how many hours do they meet if the cities are {d} km apart?",
"Given a list of {n} integers, return the length of the longest strictly increasing subsequence; the input may contain duplicates and negative values such as -{m}.",
"A retailer applies a {n} percent discount and then adds a {m} percent tax; what is the final price of an item listed at {d} dollars?",
"Implement a function that merges {n} sorted linked lists of total length {m} in O(total log {n}) time and returns the merged head node.",
"If a fair die is rolled {n} times, what is the probability of seeing at least one run of {m} consecutive sixes?",
"A tank fills through pipe A in {n} hours and drains through pipe B in {m} hours; with both open, how long until it holds {d} percent of capacity?",
"Write a SQL query that returns the top {n} customers by lifetime revenue, excluding orders refunded within {m} days of purchase.",
"A rectangle's length exceeds its width by {n} units and its area is {d} square units; find the perimeter rounded to {m} decimal places.",
"Design a rate limiter that allows {n} requests per minute per key with bursts of {m}, using a token bucket and O(1) memory per key.",
"Two cyclists start {d} km apart and ride toward each other at {n} and {m} km/h; a fly shuttles between them at twice the combined speed. How far does it travel?",
"Parse a log file where each line holds a timestamp and a status code, and report the longest window with more than {n} percent of {m}xx errors.",
"A bag holds {n} red and {m} blue marbles; drawing {d} without replacement, what is the chance of at least two reds in a row?",
]
def make_problem(i: int) -> str:
t = TEMPLATES[i % len(TEMPLATES)]
return t.format(a=chr(65 + i % 6), b=chr(75 + i % 6),
n=int(rng.integers(3, 60)), m=int(rng.integers(2, 40)),
d=int(rng.integers(50, 900)))
problems = []
for i in range(60):
date = f"{rng.choice(['2023-05', '2024-02', '2024-09', '2025-03', '2025-11', '2026-04'])}-15"
problems.append({"problem_id": f"p{i:03d}", "text": make_problem(i), "published": date})
# Plant contamination: 16 problems copied verbatim into mirror pages,
# 8 lightly paraphrased (word swaps), the rest absent from the mirror.
os.makedirs("mirror", exist_ok=True)
verbatim = problems[0:48:3][:16]
paraphrased = problems[1:48:6][:8]
filler = ("Unrelated forum chatter about keyboards, sourdough, and bike tires. " * 5).strip()
for page in range(6):
chunk = [filler]
for p in verbatim[page::6]:
chunk.append(p["text"])
for p in paraphrased[page::6]:
chunk.append(p["text"].replace("what is", "determine").replace("return", "compute"))
chunk.append(filler)
with open(f"mirror/page_{page}.txt", "w") as f:
f.write("\n\n".join(chunk))
with open("benchmark.jsonl", "w") as f:
for p in problems:
f.write(json.dumps(p) + "\n")
# Synthetic model results: 90% on planted items, 50% on clean ones.
# That gap is what contamination looks like when you split the score.
planted_ids = {p["problem_id"] for p in verbatim} | {p["problem_id"] for p in paraphrased}
with open("results.csv", "w") as f:
f.write("problem_id,correct\n")
for p in problems:
rate = 0.90 if p["problem_id"] in planted_ids else 0.50
f.write(f"{p['problem_id']},{int(rng.random() < rate)}\n")
print(f"wrote benchmark.jsonl ({len(problems)} problems), results.csv, mirror/ (6 pages)")
print(f"planted: {len(verbatim)} verbatim, {len(paraphrased)} paraphrased")
Step 2: the checker
"""Contamination check: date-windowing plus MinHash overlap against a mirror.
Run:
python contamination_check.py
"""
from __future__ import annotations
import glob
import json
import re
import pandas as pd
from datasketch import MinHash, MinHashLSH
# Training cutoffs you maintain by hand. Verify each against the vendor's
# model card before trusting a row; they drift across model versions.
CUTOFFS = {
"gpt-4o-2024-08-06": "2023-10-01",
"claude-sonnet-4-5": "2025-07-01",
}
SHINGLE = 5 # words per shingle
NUM_PERM = 128
JACCARD_FLAG = 0.35 # flag threshold; tune on known-clean pairs
def shingles(text: str, k: int = SHINGLE):
words = re.findall(r"[a-z0-9]+", text.lower())
if len(words) < k:
return {" ".join(words)}
return {" ".join(words[i : i + k]) for i in range(len(words) - k + 1)}
def minhash(text: str) -> MinHash:
m = MinHash(num_perm=NUM_PERM)
for s in shingles(text):
m.update(s.encode("utf8"))
return m
def main() -> None:
problems = [json.loads(line) for line in open("benchmark.jsonl")]
# Index mirror paragraphs in an LSH table.
lsh = MinHashLSH(threshold=JACCARD_FLAG, num_perm=NUM_PERM)
mirror_hashes: dict[str, MinHash] = {}
for path in glob.glob("mirror/*.txt"):
for j, para in enumerate(open(path).read().split("\n\n")):
if len(para.split()) < SHINGLE:
continue
key = f"{path}#p{j}"
mh = minhash(para)
mirror_hashes[key] = mh
lsh.insert(key, mh)
# Flag problems that collide with any mirror paragraph.
rows = []
for p in problems:
mh = minhash(p["text"])
hits = lsh.query(mh)
best = max((mirror_hashes[h].jaccard(mh) for h in hits), default=0.0)
rows.append({
"problem_id": p["problem_id"],
"published": p["published"],
"overlap": round(best, 2),
"flagged": best >= JACCARD_FLAG,
})
df = pd.DataFrame(rows)
n_flagged = int(df.flagged.sum())
print(f"{len(df)} problems, {n_flagged} flagged at jaccard >= {JACCARD_FLAG}\n")
# Date-windowing per model.
for model, cutoff in CUTOFFS.items():
after = df[df.published > cutoff]
print(f"{model} (cutoff {cutoff}): {len(after)}/{len(df)} problems post-cutoff")
# Score with and without the suspect subset.
scores = pd.read_csv("results.csv").merge(df, on="problem_id")
clean = scores[~scores.flagged]
flagged = scores[scores.flagged]
print(f"\nscore on full set : {scores.correct.mean():.3f} (n={len(scores)})")
print(f"score on flagged subset: {flagged.correct.mean():.3f} (n={len(flagged)})")
print(f"score on clean subset : {clean.correct.mean():.3f} (n={len(clean)})")
keep = df[~df.flagged].problem_id
with open("benchmark_clean.jsonl", "w") as f:
for p in problems:
if p["problem_id"] in set(keep):
f.write(json.dumps(p) + "\n")
print(f"\nwrote benchmark_clean.jsonl ({len(keep)} problems)")
if __name__ == "__main__":
main()
Expected output, exactly:
60 problems, 27 flagged at jaccard >= 0.35
gpt-4o-2024-08-06 (cutoff 2023-10-01): 53/60 problems post-cutoff
claude-sonnet-4-5 (cutoff 2025-07-01): 23/60 problems post-cutoff
score on full set : 0.583 (n=60)
score on flagged subset: 0.889 (n=27)
score on clean subset : 0.333 (n=33)
wrote benchmark_clean.jsonl (33 problems)
Step 3: read the report
The split is the finding. The model looked like a 58% model on the full set. On problems the mirror has seen it is an 89% model; on unseen problems it is a 33% model. When you see a gap like this on a real benchmark, the headline number is the clean-subset number, and the report should say so in the first line. LiveCodeBench institutionalized exactly this: scores reported per time window, with visible cliffs at each model's cutoff for the models that memorized 2.
27 flagged, but only 24 were planted. The three extra flags are problems built from the same template as a planted one, so they share 5-gram shingles. This false-positive mode is real on actual benchmarks too: problem families (LeetCode variants, contest archetypes) collide without being copies. Tune JACCARD_FLAG on pairs you know are clean, and read the overlap column before deleting anything. Verbatim copies score near 1.0; paraphrases land around 0.4 to 0.7; template cousins sit near the threshold.
Date-windowing and overlap detection answer different questions. Overlap detection catches your problems appearing in public text; it cannot see private training corpora. Date-windowing catches anything the model could have trained on, but only works when problems carry honest publication dates. Run both; trust the intersection. For the model rows, maintain the cutoff inventory by hand and re-verify it against each vendor's model card on every model bump; cutoffs are the most commonly stale line in any eval config.
Step 4: swap in your benchmark
Three substitutions:
- Replace
benchmark.jsonl with your problems. The published field should be the date the text first existed anywhere public (the GitHub issue date, the contest date), not the date you collected it.
- Replace
mirror/ with real snapshots: the benchmark's own GitHub repo, its Hugging Face dataset dump, top search hits for distinctive problem phrases. The most common leak path is the benchmark publishing itself; Jacovi et al. document how test sets uploaded in plain text end up in crawls, and propose the mitigations (canaries, encryption, held-out splits) worth adopting if you publish your set 1.
- Replace
results.csv with your model's per-problem outcomes from your harness.
For a 10k-problem set against a few GB of mirror text, this exact MinHash-LSH design runs in minutes on a laptop; that is the reason to prefer it over pairwise n-gram comparison, which is quadratic.
What this does not do
It does not detect contamination through private or licensed training data; nothing external can. It does not catch rephrased-beyond-recognition leakage; embedding-similarity retrieval is the next escalation, at the cost of a real false-positive problem. And a clean contamination report does not make a benchmark good: MMLU-Redux found 6.49% of sampled MMLU questions contain ground-truth errors, which no amount of decontamination fixes 4. The contamination technique reference covers the wider toolbox: canary strings, perplexity probes, and rephrase tests.
TIP
Re-run the checker on every benchmark version bump and every new model, and commit the flag report next to the scores. A score without its contamination report ages into a number nobody can defend.