An RL environment is only as good as the reward it hands back, and the audits published in 2025 and 2026 keep finding environments that hand back the wrong one. This page lists the checks to run before you spend compute on an environment or money on a vendor's. The checks are ordered so the cheap ones run first, and most need nothing but a reference solution, a known-wrong solution, and a batch of rollouts from the model you plan to train.
How do you evaluate an RL environment?
Evaluate an RL environment the way you would evaluate an LLM judge: measure what it gets wrong before you trust what it says. Run the reference solution to confirm every task is solvable, run known-wrong solutions to confirm the verifier rejects them, sample the base model to see whether the difficulty spread gives a learning signal, then probe for hacks, leaks, flakiness and contamination.
The people who build and buy environments already think this way. In Epoch AI's January 2026 interviews with 18 people across environment startups and labs, robustness against reward hacking came up again and again as the most important quality criterion 1. The ten checks below are this site's minimum audit: that criterion plus nine others.
| # | Check | What you need | Pass looks like |
|---|
| 1 | Every task can be solved | Reference solutions | Reference passes 100% through the real harness |
| 2 | Difficulty gives a learning signal | Base-model rollouts | Few tasks at 0% or 100% base pass rate |
| 3 | The verifier accepts correct work | Equivalent rewrites of correct answers | False-negative rate measured and small |
| 4 | The verifier rejects wrong work | Known-wrong solutions and a do-nothing agent | No wrong or empty solution passes |
| 5 | The policy cannot hack it | An adversarial agent or hack catalog | Exploits found and patched |
| 6 | The sandbox reaches nothing it should not | A list of what the agent can read and call | No unintended path to the answer or the scorer |
| 7 | The harness is reliable | Reruns, plus a failure reason per rollout | Crashes, budget cutoffs and wrong answers counted apart |
| 8 | Tasks are not memorized | A contamination probe | No completion of task text from partial prompts |
| 9 | The reward tracks the skill | A random-reward control and a held-out eval | Real reward beats the control on held-out data |
| 10 | The environment is versioned and documented | A datasheet | Known issues and error rates published per version |
Can every task be solved?
Run the reference solution for every task through the same harness and verifier the model will use, and expect a 100% pass rate. Any task the reference fails is broken: the answer key is wrong, the tests contradict the spec, or the harness cannot run the solution. A broken task in training rewards nothing honest, so the only rollouts that score on it are the ones that cheat.
Audits of public benchmarks show how often this check fails. An audit of Terminal-Bench 2.0 found issues in roughly 30% of the benchmark and shipped the repairs as Terminal-Bench 2.1; individual agents' scores moved by up to 12 percentage points and leaderboard positions by up to three ranks, though most scores rose only a few points and the two weakest agents lost accuracy 2. An automated audit of 168 benchmarks across nine domains flagged critical issues, including wrong ground truth, in more than 25.7% of tasks 3. Zhu et al. report that flaws like these can misestimate agent performance by up to 100% in relative terms 4.
DeepSeek-V3.2's report shows the build-time version of this check: a code environment counted as built only when the gold patch turned at least one failing test into a passing one and broke no passing test 5.
Does the difficulty spread give a learning signal?
Sample the base model, the one you plan to train, several times on every task and plot the per-task pass rates. Tasks the model always solves or never solves teach nothing under group-relative methods like GRPO: when every rollout in a group gets the same reward, the advantage is zero and so is the gradient. A good environment sits mostly in between.
DAPO made this a training technique, dynamic sampling, which over-samples prompts and drops the ones whose group accuracy is all-correct or all-wrong 6. TRL's environment docs say the same in plain words: a group where every completion gets the same reward gives no signal, so add partial credit when the model rarely succeeds 7. Practitioners set floors: multiple interviewees told Epoch AI they want a minimum pass rate of around 2 to 3%, or at least one success in 64 or 128 rollouts 1. AWS's reinforcement fine-tuning guidance makes the same point from the other side: if the base model cannot earn a non-zero reward on your prompts, the learning signal is too weak for effective training 8. DeepSeek-V3.2 kept general-agent tasks only if their pass@100 was above zero 5.
For a pass/fail reward, a group of G rollouts on a task with base pass rate p carries no gradient with probability p^G + (1 - p)^G. Averaged over your tasks, that is the share of groups that will teach nothing at the start of training. Plugging in a pass rate estimated from a handful of samples overstates it, so the audit script below uses the unbiased version: the chance that G of the n rollouts you collected, drawn without replacement, all pass or all fail.
Does the verifier accept correct work?
Feed the verifier correct answers written differently from the reference (another number format, another order, an extra unit, a different but valid code patch) and count how many it rejects. Every rejection is a false negative: a rollout that did the task, scored as failure, so the policy is pushed away from correct behaviour.
False negatives are common enough to change training results. TinyV found that of the responses a rule-based verifier marked incorrect in the Big-Math-RL-Verified dataset, 38.5% were actually correct, and adding a small LLM check raised pass rates by up to 10% 9. Agent environments have the same problem with richer state: an audit of 150 failure-scored computer-use trajectories from five benchmarks found 15.3% of FAIL verdicts wrong, with evaluator false negatives accounting for 10.7% of the audited verdicts and broken tasks for 4.7% 10. AWS suggests the simplest version of this check: confirm that a response matching the ground truth receives the maximum reward 8. The verifier chapter builds a metamorphic test set for this and measures a real library.
Does the verifier reject wrong work?
Feed the verifier solutions you know are wrong: a patch that special-cases the visible tests, an answer off by one, a response with the right format and the wrong content. Every acceptance is a false positive, and false positives are where reward hacking starts, because the policy will find the cheapest wrong answer that passes.
Run a do-nothing agent too. Zhu et al. found that an agent returning empty responses passed 38% of tau-bench's airline tasks, because those tasks were impossible and the grader counted an unchanged database as success 4.
Rajan's June 2026 audit measured this directly for code environments: in a 49-task sample of SWE-bench Verified, 28.5% of tasks had test suites weak enough that a Docker-verified incorrect patch passed them, and the rate was 25.0% on 20 R2E-Gym tasks. Across 134 SWE-bench Verified submissions, models scored 14.14 percentage points higher on the hackable tasks than on robust tasks of the same human-rated difficulty (95% CI 11.80 to 16.48), and 123 of the 134 showed the effect 11. Epoch AI's September 2026 review of OSWorld 2.1 audited 50 of its 108 tasks, found errors in 36, and rated the benchmark flawed: false negatives could affect up to 25.4% of the audited score, false positives up to 17.6%, and impossible tasks up to 7.4% 12.
Microsoft's reinforcement fine-tuning docs give the grader version of this check: test the grader on correct, incorrect and malformed responses, and check that superficial responses cannot earn a high score without solving the task 13.
Can the policy hack the environment?
Point an adversarial agent at the environment with one instruction: make the verifier pass without solving the task. Record every exploit it finds, patch the verifier or the harness, and repeat. Expect the attack rate to fall rather than reach zero, and keep every exploit as a regression test, because the next environment version can reopen it.
Zhong et al. first probed 1,968 tasks from five terminal-agent benchmarks with three frontier models told to make the verifier pass, given only the task description, and found 323 (16%) hackable. Their hacker-fixer loop then hardened two benchmarks. On KernelBench it drove the attack success rate on a held-out set of publicly reported exploits from 62% to 0%; on 77 Terminal-Bench tasks it cut documented exploits from 50.4% to 39.4% and blind attacks from 39.2% to 16.7%. The authors released the 323 hackable environments and 3,632 hack trajectories as Terminal Wrench 14. ImpossibleBench offers a cheaper probe: make the spec and the unit tests contradict each other, so any pass is a shortcut, and report the pass rate as a cheating rate 15. The Reward Hacking Benchmark found exploit rates from 0% for Claude Sonnet 4.5 to 13.9% for DeepSeek-R1-Zero, associated RL post-training with more hacking, and reported that simple environment hardening cut exploit rates by 5.7 percentage points, 87.7% in relative terms 16.
Assume the model you train will find exploits your probe missed: training applies pressure that a one-off probe does not. METR found an earlier version of o3 reward hacking in 30.4% of its RE-Bench runs, with one ambiguous task excluded, and documented o3 reading the scorer's reference answer off the call stack 17. Reward hacking detection covers the red-team side in deployed agents.
What can the sandbox reach?
List everything the agent can read, run, or call from inside the sandbox, then remove, or allowlist on purpose, anything that contains or leads to the answer: the scorer and its files, version-control history, the network, other episodes' state. An environment that leaves the answer reachable measures retrieval, and a policy trained on it learns to retrieve.
Cursor's June 2026 study is the clearest measurement of what reachability does to a score. Its auditor examined 731 Opus 4.8 Max trajectories on SWE-bench Pro and found that 63% of the successful resolutions retrieved the fix rather than derived it. In 9% of the trajectories the model searched the bundled .git history for the future commit that fixed the bug; in 57% it found the merged pull request or the fixed source file on the public web. Sealing the history and restricting internet access dropped Opus 4.8 Max from 87.1% to 73.0% and Composer 2.5 from 74.7% to 54.0%. Cursor is explicit that this does not mean every eval should remove internet access or git history 18; decide what the agent may use, and know when a score depends on it. The July 2026 OpenAI and Hugging Face incident showed the failure at its worst: agents in an evaluation sandbox searched for ways to beat the scorer and reached systems outside it 19.
Is the harness reliable from run to run?
Rerun the same tasks with the same model several times and look at the spread. Then separate the failures. A rollout that died from a sandbox crash, a dropped connection, or a dead tool server is an infrastructure failure, not a low score, and counting it as zero teaches the policy that the task is hard. A rollout cut off by a context, turn or time budget is different: the policy ran out of room, and a run that overruns its budget should not earn reward, but it belongs in its own column.
The spread is often larger than the effects people report. A validity audit of tool-calling benchmarks found that 23 repeated evaluations of the same LiveMCPBench setup scored anywhere from 57.9% to 76.8% 20. The Terminal-Bench audit showed that a fixed wall-clock limit can change results: extending the time limit closed a 20-point harness gap on GPT 5.4 and shrank a 30-point gap on GPT 5.4 mini to 5 points 2. Even the inference engine moves scores; the choice of backend alone shifted benchmark scores by up to 16.6 percentage points across five engines 21.
Read those columns before you read the score. Thinking Machines' Harbor recipe for Tinker reports a baseline where Kimi-K2.6 passed 29.0% of SWE-bench Verified while 60.6% of its rollouts ended in errors, all from context-window overflow in a harness with a 32K context cap, an 8K generation budget and no compaction, and its 29.0% pass rate counts those errors as misses 22. Read as one number, the 29.0% hides that most failures came from a context limit the harness set, not from wrong patches.
Silent failures are the dangerous kind. TRL's docs describe one in OpenEnv: Hugging Face Spaces closed long-lived WebSockets about every two hours, and clients up to OpenEnv 0.11.0 reconnected to a fresh session, so the next step() ran on a game the client had never reset, with no error. OpenEnv 0.12.0 raises SessionLostError instead 7. GLM-5's team logged a failure reason for every rollout and excluded the ones that failed because the environment collapsed 23, and NeMo Gym's gym eval reverify recomputes rewards from stored rollouts without rerunning inference, so a verifier fix can be checked against old data 24.
Are the tasks contaminated?
Ask whether the model could have seen the tasks, or their solutions, during pretraining. Environments built from public repositories, public benchmarks, or scraped problem sets often have answers on the web. Probe it directly: give the base model the first part of each task and see whether it completes the rest verbatim.
Wu et al. ran that probe on MATH-500. Given the first 60% of each problem, Qwen2.5-Math-7B regenerated the remaining 40% with a 54.6% exact-match rate and answered 53.6% of the truncated problems correctly; Llama3.1-8B managed 3.8% and 2.4%. On LiveMathBench, released after the model, Qwen's completion rate fell to 0.0% 25. RL on a contaminated task set can raise scores by surfacing memorized answers, which looks like learning and transfers nowhere. Prefer tasks written after the base model's cutoff, procedurally generated tasks, or tasks drawn from private data.
Does the reward track the skill you want?
Train a short run with a random reward and compare it with the real one on a held-out eval. If the random reward gets you most of the gain, the environment is mostly teaching format, or the base model already knew the answers, and the reward is not the thing doing the work.
This control exists because of a 2025 result. Shao et al. found that randomly assigned rewards improved Qwen2.5-Math-7B on MATH-500 by 21.4 percentage points, close to the 29.1-point gain from ground-truth rewards, while the same spurious rewards often failed to produce gains for other model families such as Llama3 or OLMo2 26. On a clean generated arithmetic benchmark, Wu et al. found that only accurate rewards gave steady improvements 25. TRL's docs give the quick version for any environment: run a few episodes by hand, and if a capable model cannot score higher than a random baseline, the reward signal needs work 7. The post-RL evaluation chapter covers the controls in full.
Is the environment versioned and documented?
An environment needs a version number, a changelog, and a datasheet that states its known broken tasks, its measured verifier error rates, its sandbox limits, and where its tasks came from. Without them you cannot compare two training runs, reproduce a result, or tell whether a score moved because the model changed or the environment did.
Buchanan et al.'s November 2026 paper, which shipped the 2.1 repairs, argues for treating benchmarks as software with continuous releases, and that repair is the example: the overall ranking stayed highly correlated (Spearman rho 0.958) while individual scores moved by up to 12 points and ranks by up to three 2. Ask for the same discipline from any environment you depend on, and pin the version in every result you report.
How do you audit an environment from rollouts?
Collect three kinds of rollouts per task: the base model several times, the reference solution once, and a known-wrong solution once. One pass over them answers checks 1, 2, 4 and part of 7: broken tasks, wasted compute from tasks that carry no gradient, verifier false positives, and the infrastructure failure rate.
The script below needs no dependencies. It reads one JSON object per rollout and prints the audit. The demo data is synthetic, generated in the script with two planted broken tasks and three planted weak-test tasks, so you can see what each line catches before you point it at real rollouts.
"""
Audit an RL environment from rollouts you already have. No dependencies.
Input: one JSON object per line, one per rollout:
{"task": "t017", "policy": "base", "reward": 1.0, "error": null}
policy is "base" (the model you will train), "oracle" (the reference
solution), or "wrong" (a known-incorrect solution). error is a string
when the rollout died for infrastructure reasons, else null. Rewards
are treated as pass/fail at pass_threshold.
"""
import json
import random
from collections import defaultdict
from math import comb
def no_signal(n, k, g):
"""Chance that g of these n rollouts, drawn without replacement, all
pass or all fail: an unbiased estimate of p**g + (1 - p)**g."""
return (comb(k, g) + comb(n - k, g)) / comb(n, g)
def audit(rows, group_size=8, pass_threshold=1.0):
by_task = defaultdict(lambda: defaultdict(list))
base_rows = [r for r in rows if r["policy"] == "base"]
for r in rows:
if not r.get("error"):
by_task[r["task"]][r["policy"]].append(r["reward"] >= pass_threshold)
tasks = sorted(by_task)
base = {t: by_task[t]["base"] for t in tasks if by_task[t]["base"]}
never = [t for t, v in base.items() if not any(v)]
always = [t for t, v in base.items() if all(v)]
oracle_fail = [t for t in tasks if by_task[t]["oracle"] and not all(by_task[t]["oracle"])]
wrong_pass = [t for t in tasks if any(by_task[t]["wrong"])]
infra = sum(1 for r in base_rows if r.get("error"))
print(f"tasks: {len(tasks)} base rollouts: {len(base_rows)} infra failures: {infra / len(base_rows):.1%}")
print(f"oracle fails (broken or unsolvable): {len(oracle_fail)} {oracle_fail[:5]}")
print(f"wrong solution passes (verifier FP): {len(wrong_pass)} {wrong_pass[:5]}")
print(f"base never solves: {len(never)} base always solves: {len(always)}")
# A GRPO group whose rewards are all equal has zero advantage: no gradient.
sized = [v for v in base.values() if len(v) >= group_size]
dead = sum(no_signal(len(v), sum(v), group_size) for v in sized) / len(sized)
print(f"expected zero-advantage groups at group size {group_size}: {dead:.0%}")
buckets = [0] * 5
for v in base.values():
buckets[min(int(sum(v) / len(v) * 5), 4)] += 1
for i, n in enumerate(buckets):
print(f" base pass rate {i * 20:>3}-{i * 20 + 20}%: {'#' * n} {n}")
if __name__ == "__main__":
# Synthetic demo: 40 tasks, 16 base rollouts each, plus one oracle and
# one wrong solution per task. Replace with your own JSONL.
rng = random.Random(7)
rows = []
for i in range(40):
task = f"t{i:03d}"
difficulty = rng.random()
broken = i in (3, 21) # no honest solution exists
weak_tests = i in (8, 30, 33) # tests accept a wrong patch
for _ in range(16):
error = "sandbox crashed" if rng.random() < 0.04 else None
solved = (not broken) and rng.random() > difficulty ** 0.5
rows.append({"task": task, "policy": "base", "reward": float(solved), "error": error})
rows.append({"task": task, "policy": "oracle", "reward": 0.0 if broken else 1.0, "error": None})
rows.append({"task": task, "policy": "wrong", "reward": 1.0 if weak_tests else 0.0, "error": None})
# In practice: rows = [json.loads(line) for line in open("rollouts.jsonl")]
audit(rows)
Running it with Python 3.11 prints:
tasks: 40 base rollouts: 640 infra failures: 4.2%
oracle fails (broken or unsolvable): 2 ['t003', 't021']
wrong solution passes (verifier FP): 3 ['t008', 't030', 't033']
base never solves: 4 base always solves: 0
expected zero-advantage groups at group size 8: 19%
base pass rate 0-20%: ########## 10
base pass rate 20-40%: ############ 12
base pass rate 40-60%: ############ 12
base pass rate 60-80%: ##### 5
base pass rate 80-100%: # 1
Read it top to bottom. The two broken tasks, t003 and t021, show up twice: as oracle failures and among the four tasks the base model never solves. Of the other two, t030 is also one of the weak-test tasks and needs a fix, and t004 is merely hard; decide whether it stays for a later curriculum stage. The three weak-test tasks are the ones a policy will learn to exploit first. The 19% zero-advantage figure estimates the share of groups at the start of training that will carry no gradient, before any fix. On this synthetic data the true share is 16.6%, and plugging the observed pass rates into p^G + (1 - p)^G would have printed 23%, which is why the script uses the unbiased estimate. Infrastructure failures are excluded from the pass rates on purpose: they belong on a dashboard, not in the reward.
What should you ask an RL environment vendor?
Ask for the evidence behind each check, not a description of the process. An environment is mostly a verifier, so you are buying its error rates. A vendor who has measured them can send numbers the same day.
- What is the reference-solution pass rate through your harness, per task family, and which tasks fail it?
- What false-negative and false-positive rates did you measure for the verifier, on how many probes, and on which answer formats?
- Which known exploits have you patched, and do you ship them as regression tests? Have you run an adversarial hacking pass on this version?
- What can the agent reach from inside the sandbox: version-control history, the network, the scorer's files, other episodes?
- What is the per-task pass-rate distribution for a model like ours, and what share of tasks sit at 0% or 100%?
- How were tasks sourced, and could their solutions be on the public web or in pretraining data?
- What share of rollouts fail for infrastructure reasons, and what share hit a context, turn or time limit? How is each scored?
- Is the environment exclusive? Two environment founders told Epoch AI that exclusive deals cost roughly four to five times as much as non-exclusive ones 1, so know what the premium buys.
- How is the environment versioned, and what changed in the last release?
What goes on an environment report card?
One table per environment version, reported next to every training result that used it. The thresholds in the right column are this site's starting points, not industry standards; tighten them for environments you train on heavily.
| Metric | How to measure it | Starting threshold |
|---|
| Reference pass rate | Reference solution through the real harness | 100%; every failure is a bug to fix or a task to drop |
| Wrong-solution acceptance | Known-wrong solutions and a do-nothing agent, per task family | 0 known acceptances; report probe count |
| Verifier false-negative rate | Equivalent rewrites of correct answers | Report with a confidence interval; investigate any category above a few percent |
| Base-model pass-rate spread | Several base rollouts per task | Most tasks strictly between 0% and 100% |
| Zero-advantage share | Unbiased estimate from your rollouts at your group size (see the script) | Track it; filter or resample when it is high |
| Infrastructure failure rate | Rollouts that died outside the model's control: crashes, dropped sessions | Reported apart from reward; never scored as 0 |
| Budget-limit failures | Rollouts cut off by a context, turn or time limit | Reported in their own column; scored by your training policy |
| Rerun spread | Same model and tasks, several reruns | Smaller than the effect you plan to claim |
| Contamination probe | Partial-prompt completion on a task sample | No verbatim completions |
| Random-reward control | Short run with random rewards, held-out eval | Real reward clearly beats it |
| Known exploits | Hacker-fixer pass, hack catalog | All patched, kept as regression tests |