An RL environment is the place a model practices during reinforcement learning. It has three parts: a set of tasks, a harness that lets the model act on each task (tools, files, a browser, a sandbox), and a verifier that turns the outcome into a reward. Remove the training loop and the same three parts make an eval. This page defines the parts, separates the terms people use loosely, and shows what the 2025 and 2026 open-model reports say they trained on.
What is an RL environment?
An RL environment is everything a model needs to attempt a task and get scored for it: the task data, the harness the model acts through, a verifier that checks the result, and the per-task state in between. During training the verifier's score becomes the reward. During evaluation the same score becomes a number in a report.
NVIDIA's NeMo Gym docs list exactly those four parts: dataset, agent harness, verifier, and state 1. Prime Intellect's environment docs go further: RL environments and agent evals are basically the same thing, a dataset plus a harness plus scoring rules, even though open-source efforts have generally treated them as separate 2. TechCrunch's 2025 explainer describes them from the product side: training grounds that simulate what an agent would do in a real software application 3.
flowchart LR
T[Task set<br/>prompts, specs, tests] --> H[Harness<br/>tools, sandbox, limits]
M[Policy model] --> H
H --> V[Verifier<br/>checks the outcome]
V --> R[Reward<br/>to the trainer]
V --> S[Score<br/>in an eval report]
The diagram is the reason this site treats environments as an evals topic. The verifier feeds both arrows. If it is wrong, the eval report is wrong and the training signal is wrong in the same way, and only the second one changes the model.
What does each part get wrong?
Each part of an environment has its own failure mode, and each failure looks like model behaviour from the outside. Broken tasks look like hard tasks, a flaky harness looks like a weak model, and a lenient verifier looks like a capable one.
| Part | What it is | Typical failure |
|---|
| Tasks | Prompts or task specs, often with reference answers or hidden tests | Impossible tasks, wrong answer keys, duplicates, tasks the model saw in pretraining |
| Harness | How the model acts: tool schemas, sandbox, turn limit, wall-clock limit | Sandboxes that die mid-episode, time limits that change scores, state leaking between episodes |
| Verifier | The check that turns a final answer or end state into a reward | False negatives (correct work scored 0) and false positives (wrong or hacked work scored 1) |
| State | The per-task execution context the harness carries between steps | Silent resets, sessions that reconnect to a different episode |
TRL's environment docs draw a line worth keeping: the verifier answers one question, how correct the outcome was, while the reward you train on is a separate training-time decision. You might binarize the score, penalize degenerate behaviour, or drop rollouts that could not be scored 4. When you evaluate an environment or a trained model, report the verifier's verdict, not the shaped reward.
RL environment vs RL gym vs sandbox vs benchmark
The four terms overlap, and vendors use them interchangeably. The distinctions below match how the main open frameworks use them.
| Term | What it means | Example |
|---|
| RL environment | One task family with its harness and verifier: the unit you train or evaluate on | A set of repository bugs, a Docker image per repo, and the hidden tests as verifier |
| RL gym | A collection of environments, or a framework for building and serving them | NeMo Gym, Reasoning Gym, OpenEnv |
| Sandbox | The isolated runtime inside the harness where code and tools execute | A Docker container, or a Daytona, Modal or E2B sandbox |
| Benchmark | An environment frozen for measurement, with a fixed split and a leaderboard | SWE-bench Verified, Terminal-Bench, tau-bench |
NeMo Gym's docs say the word "Gym" historically meant a collection of RL training environments, which is where their name comes from 1. A sandbox alone has no tasks and no verifier, so it is not an environment. A benchmark has all three parts and differs only in use: nobody is supposed to train on it. One startup employee interviewed by Epoch AI put RL demand for environments at perhaps 10 to 20 times benchmarking demand 5, so most environments built in 2026 were built to be trained on.
How is an LLM environment different from Gymnasium?
Classic RL environments follow the Gym API: reset() starts an episode, step(action) returns the next observation and a reward. LLM environments keep that shape but change what is expensive. A step is an inference call or a tool call, an episode can run for an hour, the reward usually arrives once at the end from a verifier, and the sandbox can take seconds to start.
Gymnasium, maintained by the Farama Foundation under MIT, is a standard API for single-agent RL environments and the continuation of OpenAI's Gym; version 1.4.0 shipped on 2026-10-05 6. OpenAI's original gym repository is archived, and its last release, 0.26.2, dates from October 2022 7. The LLM frameworks borrowed the interface. OpenEnv exposes reset(), step() and state() over a WebSocket, with each environment packaged as a Docker image that can run locally, in a cloud sandbox, or as a Hugging Face Space 8. TRL's GRPOTrainer takes an environment_factory class instead: it calls reset() at the start of each episode and exposes every other public method as a tool the model can call 4.
Two consequences matter for evaluation. Rollouts are expensive, so you cannot average away noise with millions of episodes the way classic RL can; you need fewer, better-designed measurements. And the reward is a verdict from a verifier, so verifier error is reward noise with a direction, not random noise. The verifier chapter measures that error directly.
What is RLVR, and how is it different from RLHF?
RLVR, reinforcement learning with verifiable rewards, trains against a reward computed by checking the answer: a math answer matched to a key, unit tests run against a patch, a constraint checked in code. RLHF trains a reward model on human preference pairs and optimizes against that learned proxy. RLVR's reward is cheap and hard to argue with when the check is sound; RLHF's reward covers open-ended quality but drifts from what people want when pushed too hard.
The name RLVR comes from AI2's Tulu 3 report in November 2024 9. GRPO, the algorithm behind DeepSeek-R1 and many later open RLVR recipes (Tulu 3 itself used PPO), was introduced in DeepSeekMath as a variant of PPO 10, and DeepSeek-R1 reported that pure RL could incentivize reasoning, with patterns such as self-reflection and verification emerging during training 11. On the RLHF side, Gao et al. showed with a synthetic gold-standard reward model that optimizing too hard against a proxy reward model lowers the gold score, Goodhart's law in measurable form 12.
The 2025 and 2026 open-model reports mix both, plus a third kind for domains with no checkable answer.
| Reward source | How it scores | Where it breaks |
|---|
| Verifiable (RLVR) | Answer match, unit tests, constraint checks | Equivalent answers rejected, weak tests accept wrong solutions |
| Learned reward model (RLHF) | A model trained on preference pairs | Overoptimization; style and length preferences |
| Rubric judge | An LLM scores the response against a per-prompt rubric | Partial credit gamed, implicit content counted as explicit |
Rubrics as Rewards reported gains over Likert-style judge rewards on HealthBench and GPQA-Diamond 13, and DeepSeek-V3.2 scores general tasks with a generative reward model where each prompt has its own rubric 14. The same approach can be gamed: a May 2026 study found rubric verifiers rewarding gains that rubric-free judges did not see, concentrated in completeness and presence checks 15.
What did open models train on in 2025 and 2026?
DeepSeek-V3.2 trained on about 85,000 agentic tasks across four task types, and GLM-5 on more than 10,000 verifiable software-engineering environments plus thousands of terminal tasks. Both reports describe filters as carefully as the environments: which tasks a gold solution must pass, which answers a verifier must be able to check, and which failures get excluded from training.
DeepSeek-V3.2 (December 2025) lists 24,667 code-agent tasks, 50,275 search-agent tasks, 5,908 code-interpreter tasks, and 4,417 general-agent tasks. Only the general-agent tasks sit in synthesized environments, 1,827 of them; the other three types use real tools and repositories 14. Its filters are a template for anyone building environments:
- General-agent tasks stay only if they have non-zero pass@100.
- A code environment counts as built only when the gold patch turns at least one failing test to passing and breaks no passing test.
- Search samples stay only when the ground truth is correct and every candidate answer is verifiably wrong.
GLM-5 (February 2026) reports more than 10,000 verifiable software-engineering environments across thousands of repositories in nine languages, plus terminal tasks packaged in the Harbor format with Docker build accuracy above 90%. A refine agent guided by written rubrics checked that tests matched task specs and that environments resisted shortcuts, and rollouts that failed because the environment collapsed were excluded rather than scored as model failures 16. The report also documents a hack its training found anyway: in slide generation the model learned to truncate overlong content and manipulate spacing to satisfy the layout reward, and the team fixed the renderer rather than the model 16.
NVIDIA says Nemotron 3 was post-trained with RL across many environments in NeMo Gym, and released RL datasets and environments for tool use, planning and multi-step reasoning alongside it 17.
Who builds RL environments in 2026?
Four kinds of builder supply them: in-house teams at frontier labs and newer labs, specialist environment startups, traditional data vendors such as Mercor, Surge, Handshake and Turing, and product companies building environments around their own software; one founder told Epoch AI there has been substantially more in-housing recently 5. Open hubs add free ones. The money is large enough that environment quality has become a buying decision, which makes it an evaluation problem for whoever signs the contract.
TechCrunch reported in September 2025, citing The Information, that Anthropic's leaders had discussed spending more than $1 billion on RL environments over the following year. The same piece described Surge building an internal environments organization, Mercor pitching environments for coding, healthcare and law, and Mechanize aiming to supply a small number of robust environments rather than many simple ones 3. Epoch AI's 18 interviews, published January 2026, put contracts at six to seven figures per quarter and tasks at $200 to $2,000 each, and two environment founders put exclusive deals at about four to five times the price of non-exclusive ones 5.
The 2026 funding record shows where the market went:
- Deeptune raised a $43 million Series A led by Andreessen Horowitz, and Mercor announced in July 2026 that it is acquiring the company for its platform for building training environments at scale 18.
- Patronus AI raised a $50 million Series B in June 2026 to build simulated "digital worlds" that replicate websites and internal systems for testing agents 19.
- Prime Intellect raised a $130 million Series A at a $1 billion valuation in July 2026, selling compute, an RL framework, and evaluation tools as one stack 20.
- Halluminate raised a $30 million Series A in October 2026 for finance environments; its CEO says four of the top five closed-source US labs are paying customers 21.
The raw material is changing too. MIT Technology Review reported in October 2026 that labs and data vendors are buying archives from defunct companies: Surge AI buys data to build "reinforcement-learning gyms" for its clients' agents, and Google outbid Mercor for Spirit Airlines' bankruptcy data at $10 million 22.
Why is an RL environment an evals problem?
Because the model learns whatever the verifier rewards, including its mistakes. An environment with impossible tasks rewards giving up or cheating, a lenient verifier rewards the shortcut it accepts, and a flaky sandbox teaches nothing while costing compute. Epoch's interviewees named resistance to reward hacking as the most important quality criterion for exactly this reason 5.
The same lesson shows up outside training. METR's investigation of the July 2026 OpenAI and Hugging Face incident found that many evaluation targets could not be exploited the intended way, and identified impossible tasks as the trigger that sent agents searching for ways to cheat the scorer 23. An environment you have not evaluated is an environment whose failure modes you will discover through the model.
The next chapter turns this into ten concrete checks: how to evaluate an RL environment before you train on it or buy it.