Six open projects cover most of what people mean by an RL gym for LLMs in November 2026: OpenEnv, Prime Intellect's verifiers library and Environments Hub, NVIDIA's NeMo Gym, Harbor, Gymnasium, and Reasoning Gym. They overlap more than their names suggest, and in 2026 they started reading each other's formats. This page compares them on what matters for evaluation: how an environment is defined, how results are scored and re-scored, and how much of the audit work they do for you. Versions, dates and licenses were read from GitHub and the project docs on 2026-11-18; this site has no commercial relationship with any of them.
Which RL gym should you use?
Pick by the trainer and the task you already have. OpenEnv if you train with Hugging Face TRL or want environments on the Hugging Face Hub; verifiers and the Environments Hub if you use Prime Intellect's stack; NeMo Gym if you train with NeMo RL or run large evaluations on NVIDIA infrastructure; Harbor for terminal and coding agents in containers; Gymnasium for classic control and games; Reasoning Gym for procedurally generated reasoning tasks with checkable answers. None of them audits your environment for you.
How do the main RL gyms compare?
All six define an environment as tasks plus a way to act plus a score; they differ in packaging (Docker images, Python packages, task directories, generators), in where environments run, and in how much evaluation tooling ships with them, from NeMo Gym's per-task profiling and re-scoring to none at all.
| Project | Maintainer and license | Latest release | How an environment is defined | Evaluation support |
|---|
| OpenEnv | Technical committee incl. Meta-PyTorch and Hugging Face; BSD-3-Clause | 0.13.0, 2026-11-12 | Docker image exposing reset(), step(), state() over a WebSocket; runs locally, in a cloud sandbox, or as a Hugging Face Space | README points to Inspect AI for evaluation; 40+ ready environments |
| verifiers + Environments Hub | Prime Intellect; MIT | 0.4.1, 2026-11-03 | A Python package with a pyproject.toml, distributed as a wheel and published with prime env push | vf-run for quick runs, prime-rl's uv run eval for full evaluations, Task.rescore for offline re-scoring |
| NeMo Gym | NVIDIA; Apache-2.0 | 0.7.0, 2026-10-12 | Dataset, agent harness, verifier and state; resources servers ship example data, configs and tests | gym eval run, gym eval profile for per-task pass rates, gym eval reverify to recompute rewards from stored rollouts |
| Harbor | Harbor Framework (the Terminal-Bench team); Apache-2.0 | 0.26.0, 2026-11-13 | A task directory with an instruction, an environment Dockerfile, and tests | Parallel trials across sandbox providers, regrading of recorded runs, and rollouts for RL optimization |
| Gymnasium | Farama Foundation; MIT | 1.4.0, 2026-10-05 | Python reset() and step() API, single-agent | None specific to LLMs |
| Reasoning Gym | open-thought; Apache-2.0 | 0.1.25, 2026-03-28 | Procedural generators for more than 100 tasks with adjustable complexity | Each dataset scores answers with its own score_answer method |
Sources for the rows: OpenEnv 1 2, verifiers 3 4 5, NeMo Gym 6, Harbor 7 8, Gymnasium 9, Reasoning Gym 10.
OpenEnv: one interface, many hosts
OpenEnv describes itself as an interface library for RL post-training with environments. Every environment is a Docker image behind the same reset(), step() and state() calls, sync or async, and the README warns that the project is in early development and its APIs may still change 1. It is governed by a technical committee whose members include Meta-PyTorch, Hugging Face, Nvidia, Prime Intellect, Mercor, Microsoft, Unsloth, Modal, Fleet AI, Reflection, RadixArk and Nebius 1.
The 2026 releases show where it is going: a tau2-bench customer-service environment in 0.9.0, a Kubernetes provider and openenv serve in 0.11.0, SessionLostError on dropped WebSockets in 0.12.0, and export of Harbor rollouts as SFT rows in 0.13.0 2. For evaluation, two operational details matter. OpenEnv servers allow one concurrent session by default, so a training or eval run that opens many sessions fails until you raise the limit, and clients before 0.12.0 could silently continue a session on an episode they had never reset 11.
verifiers and the Environments Hub
Prime Intellect's position is that an RL environment and an agent eval are basically the same thing, a dataset plus a harness plus scoring rules, and the Hub treats environments as Python packages that declare dependencies in a pyproject.toml and ship as wheels 5. The company reported more than 2,500 RL environments from over 400 contributors in June 2026, and integrated the verifiers framework and its Prime Sandbox runtime with NeMo Gym as part of NVIDIA's Nemotron coalition 12.
The library moves fast. Release 0.4.1 (2026-11-03) removed vf-eval, vf-replay, vf-init, the Rich dashboard and eval platform push; quick runs now use vf-run, full evaluations and uploads go through prime-rl's uv run eval, and Task.rescore(trace) scores a saved trace again offline 4. Offline re-scoring is the feature to care about: it lets you fix a verifier and measure the effect on old rollouts without paying for new ones. prime-rl, the trainer, is at 0.10.0 (2026-10-13) 13.
NeMo Gym and NeMo RL
NeMo Gym defines an environment as a dataset, an agent harness, a verifier and per-task state, and its resources servers ship example data, configuration files and tests 6. Its evaluation commands are the most complete of the group: gym eval run runs an agent over tasks and writes rollouts plus aggregate metrics, gym eval profile gives per-task pass rates, and gym eval reverify, added in 0.5.0, recomputes rewards from stored rollouts without re-running inference 6. NeMo RL 0.8.0 made NeMo Gym its main evaluation path and added grouped pass@k to GRPO validation 14.
NeMo Gym also reads Harbor's format. Task folders written for Harbor run in NeMo Gym as they are, with results in NeMo Gym's standard rollout format, and gym dataset publish --harbor converts Gym-only formats the other way when it can, refusing and naming any task it cannot convert 15.
Harbor
Harbor comes from the Terminal-Bench team and describes itself as a framework for evaluating and optimizing agents and language models; its README lists generating rollouts for RL optimization, and running experiments in thousands of environments in parallel through providers such as Daytona and Modal 7. A task is a directory with an instruction, a Dockerfile for the environment, and a test script that produces the reward. The Harbor review covers it as an eval runner, including the separate verifier that new tasks get by default since 0.25.0, which runs the tests outside the agent's sandbox 16.
Its task format is becoming a shared one. GLM-5's team packaged thousands of terminal training environments in the Harbor format 17, NeMo Gym runs Harbor task folders unchanged 15, and TRL's experimental harness trainer can train loop-owning agents such as OpenCode or Claude Code on Harbor tasks through openenv harbor serve 11. Thinking Machines' Tinker cookbook has an RL recipe for Harbor-formatted tasks whose reported baseline is itself an evaluation lesson: with a naive harness, a 32K context cap, an 8K generation budget and no context compaction, Kimi-K2.6 passed 29.0% of SWE-bench Verified, and 60.6% of its rollouts ended in errors that the recipe attributes entirely to context-window overflow 18. That 29.0% says as much about the harness's context budget as about the model.
Gymnasium and Reasoning Gym
Gymnasium is the maintained successor to OpenAI's Gym and a standard API for classic control, robotics and game environments; 1.4.0 shipped on 2026-10-05 9. Its API shaped the LLM frameworks, but nothing in it is specific to language models, so use it when the environment is a simulator rather than a sandbox.
Reasoning Gym, a NeurIPS 2025 Spotlight, generates reasoning tasks procedurally: more than 100 tasks with adjustable complexity, each with an algorithmic score_answer method 10. Procedural generation answers two audit questions at once. Fresh instances cannot be memorized from pretraining, and you can tune difficulty to keep base-model pass rates away from 0% and 100%. Its last release was 0.1.25 in March 2026, though the repository was still active in November.
Which trainers consume these environments?
Environments and trainers are separate choices, and most trainers now accept environments from more than one source. TRL's GRPOTrainer takes an environment_factory class, and its OpenEnv, OpenReward and Harbor integrations all follow that contract, so they are interchangeable at the TRL level 11.
| Trainer | Description from its repository | Latest release |
|---|
| TRL | Train transformer language models with reinforcement learning | 1.18.1, 2026-11-17 19 |
| verl | A flexible and efficient RL post-training framework | 0.9.1, 2026-09-20 20 |
| OpenRLHF | A Ray-based agentic RL framework | 0.11.2, 2026-09-14 21 |
| SkyRL | A modular full-stack RL library for LLMs | 0.5.0, 2026-11-17 22 |
| prime-rl | Agentic RL training at scale | 0.10.0, 2026-10-13 13 |
| NeMo RL | Scalable toolkit for efficient model reinforcement | 0.8.0, 2026-11-06 14 |
| rLLM | Reinforcement learning for LLMs | 0.3.0-pre, 2026-04-30 23 |
| OpenPipe ART | Train multi-step agents for real-world tasks using GRPO | 0.5.24, 2026-11-04 24 |
| Agent Lightning (Microsoft) | A trainer for AI agents | 1.0.3, 2026-11-03 25 |
| tinker-cookbook | Post-training recipes for the hosted Tinker API | 0.7.0, 2026-11-17 26 |
For evaluation the trainer matters in one way: what it logs. Prefer a setup that keeps every rollout with its verifier verdict, so you can re-score after a verifier fix and compute per-task pass rates instead of reading a mean reward curve.
Where does managed RL fine-tuning stand in November 2026?
OpenAI and Fireworks no longer offer managed reinforcement fine-tuning. Microsoft Foundry and Amazon Bedrock still run the RL loop and the graders for you, Google's version is Pre-GA and limited to testing, and Together AI, Thinking Machines' Tinker and Fireworks' Training API hand you the loop and the reward. Either way, evaluating the reward is your job.
| Provider | Status on 2026-11-18 | Reward and grader model |
|---|
| OpenAI | RFT is no longer among the fine-tuning methods OpenAI lists as supported (SFT, vision fine-tuning and DPO), and OpenAI says it is winding down the fine-tuning platform 27. The fine-tuned o4-mini snapshot shut down on 2026-10-23. Job creation for the remaining methods closed to organizations that had never run fine-tuning on 2026-05-07, to organizations without fine-tuned-model inference in the past 60 days on 2026-07-02, and closes to active customers on 2027-01-06 28 | No longer offered for new RFT jobs; Microsoft Foundry documents the same grader families |
| Microsoft Foundry | Available; docs updated October 2026 | The same grader families, with validate and run endpoints for testing a grader before training; reports training and validation reward means, per-grader rewards, grader error rates and reasoning tokens 29 |
| Amazon Bedrock | Available for Amazon Nova and supported open models | RLVR through custom AWS Lambda reward functions, or RLAIF with an LLM judge whose prompt acts as the reward 30 |
| Google | Pre-GA, for limited testing and evaluation under Google's Pre-GA terms; Gemini 3.5 Flash and Gemini 3.1 Flash-Lite, v1beta1 API only | Function-based, LLM-based, or custom reward functions 31 |
| Fireworks | Managed RFT removed, along with evaluators and evaluation jobs | RL runs on the Training API, where your rollout code computes the reward 32 |
| Together AI | Post-Training launched 2026-10-19; $9 per B200 GPU-hour for the first eight weeks | You define the training loop; standard losses include GRPO, CISPO and DPPO 33 |
Microsoft's docs are the most complete guide of the group to evaluating a managed RL job, and the advice transfers to any setup: test the grader on correct, incorrect and malformed responses, check that superficial responses cannot score high, compare checkpoints on validation reward and then on held-out tasks, and remember that high grader rewards alone do not establish that the model solves the task 29.
How do you evaluate a gym before adopting it?
Check what it lets you see and redo. A gym that hides rollouts, mixes infrastructure errors into reward, or cannot re-score old data makes every check in the environment checklist harder.
- Does it keep the verifier out of the agent's sandbox, or at least make that the default?
- Does it store full rollouts with verdicts, and can it re-score them after a verifier change (
Task.rescore, gym eval reverify, Harbor regrades)?
- Does it report per-task pass rates, not only a mean?
- Does it report infrastructure failures (crashes, dropped sessions) and budget cutoffs (time limits, context overflow) apart from task failures?
- Can you pin the framework version and the environment version in every result?
- What are its concurrency defaults, and what happens when a session drops mid-episode?
- How fast is the API changing? OpenEnv says its APIs may still change, and verifiers 0.4.1 removed several commands in a point release, so budget for upgrades.
What other gyms are worth knowing?
Several narrower projects cover single domains. BrowserGym, from ServiceNow, is a Gym environment for web task automation that wraps benchmarks such as WebArena and WorkArena; its latest release is 0.14.3 34. SWE-smith turns GitHub repositories into software-engineering tasks by synthesizing changes that break existing tests, and its authors built 50,000 instances from 128 repositories and trained SWE-agent-LM-32B to 40.2% pass@1 on SWE-bench Verified 35. Treat each as a source of environments and run the same audit before training on it.