A reward curve that goes up says the policy got better at earning reward. It does not say the model got better at the task, and the 2025 and 2026 literature is full of RL gains that shrank or vanished under a closer look: gains that a random reward also produced, gains on benchmarks the base model had memorized, gains smaller than the run-to-run noise. The measurements and controls below separate a real improvement from those artifacts, and the last sections cover the side effects to check before you ship the checkpoint.
How do you evaluate a model after RL training?
Compare the trained model with its own base model on held-out tasks, at several values of k, over enough seeds to see the noise. Then run the controls that rule out cheap explanations, a random-reward run and a contamination probe, and measure what training cost: capabilities you did not train, output length, calibration, and hacking behaviour. A single pass@1 number on a public benchmark answers none of these questions.
Why is pass@1 not enough after RL?
Because RL can raise pass@1 while shrinking the set of problems the model can solve at all. Yue et al. found that models trained with RLVR beat their base models at small k, such as k equal to 1, but the base models achieved higher pass@k when k was large. RL concentrated probability on solutions the base model could already find, rather than adding new ones.
The tables show the trade. Across six RL algorithms applied to Qwen2.5-7B, pass@1 on Omni-MATH-Test rose from the base model's 10.2 to between 23.8 and 28.1, while pass@256 stayed within about two points of the base model's 69.1, from 67.0 to 69.7. The loss appears with longer training: as GRPO ran from step 150 to step 450, pass@1 rose from 25.1 to 28.3 while pass@256 fell from 68.3 to 63.9, and on MATH500 pass@256 fell from 97.2 to 95.4 against the base model's 96.2. With a 32B model on Minerva, the base model beat the RL model by about 9% at k equal to 128. The same paper found that distillation, unlike RLVR, could introduce new reasoning patterns from the teacher 1.
Other results push the other way, which is why you measure instead of assuming. NVIDIA's ProRL reported that prolonged RL training uncovered strategies inaccessible to the base model, that RL-trained models outperformed base models across a wide range of pass@k, and that how much the boundary expanded correlated with the base model's competence on the task and with training duration 2. For agents, a 2026 PASS@(k,T) analysis that varies both the sampling budget k and the interaction depth T found that tool-use RL enlarged the capability boundary, with the gap over the base model widening at large k, and limited to compositional, sequential information gathering. The scale is small: one 7B base model and 100 test problems per category, where the RL agent reached 0.81 against the base agent's 0.77 at k equal to 64 on the sequential retrieval category 3.
The practical rule: plot pass@k for the base and the trained model on the same held-out problems, from k equal to 1 up to the largest k you can afford, and report where the curves cross, if they do. Also count the problems each model solves that the other never does. The script below computes both from per-problem sample counts, using the unbiased pass@k estimator from the HumanEval paper 4. Its demo data is synthetic: it builds a trained model that sharpens problems the base model already solves sometimes and loses some of the base model's rare successes, the pattern Yue et al. describe.
"""
Compare a base model and its RL-trained checkpoint across k, not only at
k=1. Input per model: for each problem, n samples drawn and c correct.
Uses the unbiased pass@k estimator from the HumanEval paper. No deps.
"""
import random
from math import comb
def pass_at_k(n, c, k):
"""Probability that at least one of k draws (without replacement) is correct."""
if n - c < k:
return 1.0
return 1.0 - comb(n - c, k) / comb(n, k)
def curve(counts, n, ks):
return {k: sum(pass_at_k(n, c, k) for c in counts) / len(counts) for k in ks}
def compare(base_counts, rl_counts, n, ks=(1, 4, 16, 64, 256)):
base, rl = curve(base_counts, n, ks), curve(rl_counts, n, ks)
print(f"{'k':>5} {'base':>7} {'RL':>7} {'RL - base':>10}")
for k in ks:
print(f"{k:>5} {base[k]:>7.1%} {rl[k]:>7.1%} {rl[k] - base[k]:>+10.1%}")
crossed = [k for k in ks if base[k] > rl[k]]
print("base wins from k =", crossed[0] if crossed else "never (in this range)")
lost = sum(1 for b, r in zip(base_counts, rl_counts) if b > 0 and r == 0)
gained = sum(1 for b, r in zip(base_counts, rl_counts) if b == 0 and r > 0)
print(f"problems solved only by base: {lost} only by RL: {gained}")
if __name__ == "__main__":
# Synthetic stand-in for real sampling results: RL sharpens problems the
# base model already solves sometimes and drops some rare successes.
rng = random.Random(0)
n = 256
base_counts, rl_counts = [], []
for _ in range(300):
p_base = rng.random() ** 3
p_rl = 0.0 if p_base < 0.01 else min(1.0, 3 * p_base)
base_counts.append(sum(rng.random() < p_base for _ in range(n)))
rl_counts.append(sum(rng.random() < p_rl for _ in range(n)))
compare(base_counts, rl_counts, n)
Running it with Python 3.11 prints:
k base RL RL - base
1 24.8% 48.1% +23.3%
4 46.7% 62.6% +15.9%
16 65.2% 74.4% +9.2%
64 78.1% 79.8% +1.7%
256 85.0% 80.3% -4.7%
base wins from k = 256
problems solved only by base: 15 only by RL: 1
These numbers are synthetic and show the shape, not a result. With your own data, replace the two count lists with real per-problem correct counts at the same n for both models. A trained model that gains 23 points at pass@1 and loses at pass@256 has become more reliable on problems it could already solve and lost some it solved only rarely; whether that is the trade you want depends on how the model will be sampled in production.
Did the reward cause the gain?
Run the same training with a random reward, or a reward for format only, and evaluate both runs on the same held-out set. If the control recovers most of the gain, the reward was not the cause: the run taught answer formatting, or surfaced answers the base model already knew. This control is cheap compared with publishing a result it would have overturned.
The control exists because of a result that reset expectations in 2025. Shao et al. found that randomly assigned rewards improved Qwen2.5-Math-7B on MATH-500 by 21.4 percentage points, close to the 29.1-point gain from ground-truth rewards, and that the spurious rewards that worked for Qwen models often failed to produce gains for other families such as Llama3 or OLMo2 5. One-shot RLVR is a related warning: a single training example lifted Qwen2.5-Math-1.5B on MATH500 from 36.0% to 73.6%, matching a 1,200-example subset, and the authors separate out how much of the gain was format correction 6. Gains on one base model family are not evidence about another, and a large jump from a weak starting point may be mostly format.
Is the gain from contamination?
Check whether the base model already knew the evaluation problems. Give it the first part of each problem and see whether it completes the rest; compare against a benchmark released after the model's training cutoff. A model that can recite a benchmark's problems can be pushed by almost any reward into reciting their answers.
Wu et al. found that Qwen2.5-Math-7B, given the first 60% of each MATH-500 problem, regenerated the remaining 40% with a 54.6% exact-match rate, while Llama3.1-8B managed 3.8%; on the newer LiveMathBench the Qwen completion rate fell to 0.0%. On RandomCalculation, a generator of clean arithmetic problems, only accurate reward signals gave steady improvements, and random or incorrect rewards did not 7. Evaluate RL gains on problems the base model cannot have seen: post-cutoff competitions, freshly generated tasks, or private data.
How many seeds and samples do you need?
More than most papers use. Small benchmarks move several points from one question, sampling seed, prompt template, decoding settings or inference engine, so a single run per model cannot support a claim of a few points. Average over multiple seeds, report a confidence interval on a paired comparison over the same problems, and pin every setting that is not the variable you are testing.
Start with the arithmetic: on a 30-problem benchmark such as AIME, one problem is 3.3 points of pass@1. Hochlehnert et al. found mathematical reasoning benchmarks highly sensitive to decoding parameters, random seeds, prompt formatting, and even hardware and software configurations, and concluded that most RL approaches yielded only modest improvements, far below prior claims, and were prone to overfitting small benchmarks such as AIME'24; in their settings supervised fine-tuning generalized more consistently 8. Changing only the inference backend shifted benchmark scores by up to 16.6 percentage points in a 2026 study of five engines 9.
DataFlex-RL shows what careful statistics does to a crowded literature. With 12 matched seeds on Qwen2.5-7B-Base, uniform GRPO improved the domain-balanced average by 7.76 points over the untrained checkpoint, but none of eight rollout-selection and reweighting methods achieved a paired 95% confidence interval that excluded zero against uniform sampling. In a separate re-scoring of nine Qwen2.5-7B-Instruct runs, rankings from a math-heavy six-benchmark summary and a domain-balanced 12-benchmark summary were negatively correlated, at -0.33 10. Which benchmarks you average over can change which method looks best.
Small examples show the same thing. TRL's Wordle tutorial reports a model going from 0 to 10 to 20 wins out of 30 after SFT and then GRPO, and says plainly that a new run will land a few wins away: two SFT runs with the same recipe won 10 and 13 games, and the 30 test words come from the 156 nouns the game draws secrets from, which the models also saw in training 11. That candour is the standard to hold your own results to. The paired bootstrap chapter covers the confidence intervals.
Does the gain transfer to held-out environments?
Hold out whole environments, not only tasks within the training environments. A held-out task from a training environment shares its tool schemas, grader quirks and phrasing, so a policy that learned those shortcuts passes it without having learned the skill. Cross-benchmark transfer is the stronger test.
Mehta et al. post-trained Qwen3.5-122B-A10B on 363 long-horizon MCP tasks across 27 categories and measured greedy pass@1 on benchmarks it never trained on: +9.6 points on Toolathlon, +5.3 on tau2-Bench, +3.5 on BFCL-V4, +5.8 on SWE-Bench Pro, and +2.8 on Terminal-Bench 2. Those deltas come from a single post-training run, with SFT on 3,000 teacher trajectories before RL, and the authors avoid claims of statistical significance. They argue that an in-distribution holdout shares the training environment's regularities, which makes cross-benchmark transfer the better evidence of a learned skill 12. For reasoning, Huan et al. evaluated over 20 open-weight reasoning-tuned models and found that most that succeeded in math failed to transfer their gains to other domains; in controlled Qwen3-14B runs on math-only data, RL-tuned models generalized across domains while SFT-tuned models often forgot general capabilities 13.
What side effects should you measure?
Measure what training changed besides the target metric: capabilities you did not train, the policy's entropy, output length, calibration, and hacking and misalignment behaviour. Each can move during RL, and none of them shows up in the reward curve.
| Side effect | What happens | What to measure |
|---|
| Forgetting | Forgetting tracks the KL divergence between the fine-tuned and base policy on the new task, and online RL forgets less than SFT 14 | KL to the base model, plus a fixed regression suite of capabilities you did not train |
| Entropy collapse | Policy entropy drops sharply early in training, and performance follows R = -a exp(H) + b, so it plateaus as entropy runs out 15 | Policy entropy per step; a flattening reward with falling entropy predicts the ceiling |
| Length growth | Outputs can get longer; Microsoft's RFT docs say more reasoning tokens do not necessarily mean a better model, and advise comparing token use with reward and held-out quality 16 | Mean output tokens and cost per solved task, before and after |
| Calibration | RLVR can make models excessively over-confident in incorrect answers 17 | Calibration error on stated or sampled confidence, before and after |
| Hacking and misalignment | A model that learned to reward hack in production coding environments generalized to alignment faking, cooperation with malicious actors and attempted sabotage; chat-style safety training fixed chat evals but not agentic ones 18 | Hack rate on probes, agentic misalignment evals, and scores computed with cheating counted both ways |
Hacking needs more than a table row. The Reward Hacking Benchmark associated RL post-training with substantially higher reward hacking, from 0.6% for DeepSeek-V3 to 13.9% for its RL-trained sibling DeepSeek-R1-Zero 19, so hack rate belongs in every post-RL report. And when cheating is common, compute the headline twice: METR's June 2026 evaluation of GPT-5.6 Sol put its 50% time horizon at about 11.3 hours with detected cheating counted as failure and past 270 hours with it counted as success 20. On the mitigation side, MacDiarmid et al. tested inoculation prompting, which frames reward hacking as acceptable during training, on a model first given knowledge of hacking strategies. It largely removed the misaligned generalization even though the hacking itself was still learned 18.
Can you predict where an RL run will end?
Partly. Fit a saturating curve of held-out performance against training compute early in the run and compare recipes by the fitted ceiling and efficiency, not by the reward at an arbitrary step. Two recipes can look identical at the step you checked and end in very different places.
Khatri et al. ran more than 400,000 GPU-hours of experiments and fit sigmoidal compute-performance curves to RL training. Recipes differed in their asymptotic performance, while details such as loss aggregation, normalization, curriculum and the off-policy algorithm mainly changed compute efficiency without materially shifting the asymptote. Their ScaleRL recipe held its predicted curve in a single run scaled to 100,000 GPU-hours 21. The evaluation lesson is to fit on a held-out validation set rather than the training reward, which rules out memorized prompts, and then to check the fitted ceiling against an independent evaluator, because a validation split graded by the same verifier still rewards whatever that verifier gets wrong.
What goes in an RL results report?
An RL results report needs base-versus-trained pass@k curves on held-out problems, the seeds and pinned settings behind them, a random-reward control, a contamination probe, a held-out environment, side-effect measurements, the hack rate, and the environment and verifier versions. Without those, a reported gain is a hypothesis.
- Base and trained pass@k curves on held-out problems, from k equal to 1 up to the largest k you can afford, with the crossover point and the solved-set overlap.
- The number of seeds per model, paired confidence intervals, and the pinned decoding settings, prompt template and inference backend.
- A random-reward or format-only control on the same base model and held-out set.
- A contamination probe, and the release dates of every evaluation set against the base model's cutoff.
- Results on at least one held-out environment, not only held-out tasks.
- Regression-suite deltas, KL to base, policy entropy, mean output tokens, and calibration before and after.
- Hack rate on probes, and the headline computed with detected cheating counted as failure and as success.
- The environment and verifier versions the run used, with their audit results from the environment checklist and the verifier tests.