A reward model is a learned judge whose verdicts become the training signal, so it inherits every problem an LLM judge has and adds one more: the policy is optimized to exploit it. Benchmarks score reward models on fixed preference pairs. What you need to know is whether optimizing against one improves the policy on something the reward model cannot see. Below: what each benchmark measures, how far its scores predict RL results, and the tests to run on your own data.
How do you evaluate a reward model?
Check three things in order. First, does it rank the better response above the worse one on data like yours, measured as pairwise or best-of-N accuracy with a confidence interval? Second, does that ranking survive shortcuts such as length, style and position? Third, does optimizing against it improve the policy on a held-out evaluator the reward model never scores? Public benchmarks answer the first question for someone else's data and the third only loosely.
The third check is the one that matters, because a reward model is a proxy. Gao et al. showed it with a synthetic setup in which a fixed gold-standard reward model plays the role of humans: optimizing hard against a proxy reward model eventually lowers the gold score, in line with Goodhart's law, and the shape of that curve depends on whether you optimize with RL or best-of-n sampling 1. Any evaluation that stops at accuracy on fixed pairs cannot see this.
What do reward model benchmarks measure?
Each benchmark measures ranking accuracy under a different stress: harder prompts, style interference, objective correctness, or step-level errors. None of them measures your domain. Scores track each other across them while absolute levels differ: over 17 reward models the RewardBench 2 authors trained, RewardBench 2 scores correlated at 0.96 with RM-Bench, 0.97 with RewardBench, and 0.89 to 0.95 with PPE's metrics 2.
| Benchmark | What it tests | Format and size | Headline finding |
|---|
| RewardBench (2024) | Chosen vs rejected accuracy on chat, safety and reasoning prompts | Pairwise | The baseline RewardBench 2 is compared against 3 |
| RewardBench 2 (2025) | Factuality, precise instruction following, math, safety, focus, ties | Best-of-4, 1,865 prompts | Models score about 20 points lower than on RewardBench 2 |
| PPE (2024) | Human preference and verifiable correctness | 12 metrics across 12 domains | Built against an end-to-end RLHF experiment 4 |
| RM-Bench (2024) | Subtle content differences while style varies (length, Markdown) | Pairwise, nearly 40 reward models tested | On the hard split, the top-ranked model overall scored 46.6% and most models failed to beat the 50% random level 5 |
| JudgeBench (2024) | Objective correctness on hard pairs | Pairwise | Strong judges are only slightly better than random 6 |
RewardBench 2 judges a reward model by whether it picks the correct response out of four completions per prompt, across six subsets: Factuality, Precise Instruction Following, Math, Safety, Focus and Ties 2. PPE combines a large human preference set with a verifiable correctness set, measures 12 metrics across 12 domains, and was designed by launching an end-to-end RLHF experiment on a crowdsourced preference platform to see which metrics track post-RLHF results 4. JudgeBench, an honest take covers how far to trust that benchmark's numbers.
Does a high benchmark score mean a better RL policy?
Only partly. RewardBench 2 correlates well with best-of-N selection, but its own authors found that reward models from a different model lineage than the policy produce worse policies under PPO, even when they top the benchmark. Choose a reward model by testing it in your own training loop, against a held-out evaluator.
The numbers come from the RewardBench 2 paper. Average downstream performance with best-of-N sampling had a Pearson correlation of 0.87 with RewardBench 2 accuracy. PPO was different: the paper found that using a reward model initialized from a different starting point had the strongest effect. Three reward models built on Llama 8B Instruct, a different base from the policy, scored 69.4 to 72.9 on RewardBench 2, the highest in that comparison, yet gave PPO scores of 54.5 to 56.4. Two built from the same Tulu 8B SFT base as the on-policy models but trained only on Skywork preference data scored 65.6 to 66.7 and gave 57.0 to 58.5. The on-policy reward models, scoring 49.8 to 68.7, gave about 59.5 to 60.7 2. In that study, a reward model's base and training prompts predicted PPO results better than its leaderboard score.
RM-Bench makes a related point about what accuracy hides. It tested nearly 40 reward models on pairs that differ in subtle content while the style varies in length and Markdown formatting. On the hard split, where the rejected response has the more favorable style, the top-ranked model overall scored 46.6%, worse than a coin flip, and most models failed to beat random 5. A reward model that prefers well-formatted wrong answers trains a policy to write well-formatted wrong answers.
How do you test a reward model on your own data?
Build a preference set from your own domain, with responses sampled from the policy you plan to train, and score the reward model on it with confidence intervals. Then split the results by every shortcut a policy could learn: length, formatting, position in the prompt, and confident tone. A reward model that looks fine overall can be reading the shortcut instead of the content.
Practical rules for the set:
- Sample both responses from your policy or a close relative: during training the reward model scores that policy's outputs, so test it on that distribution rather than on another model's style.
- Include pairs where the better response is shorter, plainer, or less confident than the worse one.
- Include near-ties and label them, so you can see whether the model invents preferences.
- For generative reward models that compare two responses in one prompt, run every pair in both orders.
- Label with your best grader: humans, or a verifier the reward model never sees.
The script below runs the length split. The data and the reward model are synthetic stand-ins, built so the reward model leans on length: the correct answer is long in half of the pairs and short in the other half.
"""
Pairwise accuracy of a reward model on your own preference pairs, split
by whether the preferred answer is the longer one. A model that leans on
length looks fine in aggregate and fails on the split. No dependencies.
"""
import random
from math import sqrt
def wilson(hits, n, z=1.96):
p = hits / n
centre = (p + z * z / (2 * n)) / (1 + z * z / n)
half = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
return (centre - half, centre + half)
def report(pairs, score):
# Score every pair once, then split, so the groups add up.
wins = [score(p["prompt"], p["chosen"]) > score(p["prompt"], p["rejected"]) for p in pairs]
longer = [len(p["chosen"]) > len(p["rejected"]) for p in pairs]
groups = {"all": range(len(pairs)),
"chosen is longer": [i for i, x in enumerate(longer) if x],
"chosen is shorter": [i for i, x in enumerate(longer) if not x]}
for name, idx in groups.items():
n = len(idx)
if n == 0:
continue
hits = sum(wins[i] for i in idx)
lo, hi = wilson(hits, n)
print(f"{name:>18}: {hits / n:.0%} (95% CI {lo:.0%} to {hi:.0%}, n={n})")
if __name__ == "__main__":
# Synthetic pairs: the chosen answer is correct; its length is random.
rng = random.Random(3)
pairs = []
for i in range(400):
short, long_ = "x" * rng.randint(20, 60), "x" * rng.randint(120, 400)
if rng.random() < 0.5:
pairs.append({"prompt": i, "chosen": "ok " + long_, "rejected": "bad " + short})
else:
pairs.append({"prompt": i, "chosen": "ok " + short, "rejected": "bad " + long_})
def toy_rm(prompt, response):
# Stand-in reward model: a weak correctness signal plus a length bonus.
return (1.0 if response.startswith("ok") else 0.0) + len(response) / 150 + rng.gauss(0, 0.4)
report(pairs, toy_rm)
Running it with Python 3.11 prints:
all: 62% (95% CI 57% to 66%, n=400)
chosen is longer: 100% (95% CI 98% to 100%, n=207)
chosen is shorter: 20% (95% CI 15% to 26%, n=193)
The aggregate 62% reads as a weak but usable reward model. The split shows what it is: near-perfect when the better answer is also longer, and wrong four times out of five when it is not. A policy trained against it would learn to write longer answers, the failure RM-Bench probes by varying length and Markdown formatting 5. Replace toy_rm with a call to your reward model and pairs with your labeled set; add splits for formatting, position and tone the same way.
How do you evaluate a process reward model?
A process reward model scores each step of a solution rather than only the final answer, so evaluate it on step-level labels: can it find the first wrong step, and can it tell a correct but unusual step from an error? Final-answer accuracy does not test this, because many solutions reach the right answer through a wrong step.
Lightman et al.'s PRM800K work established process supervision for math 7, and three benchmarks now test the step-level skill directly. ProcessBench has 3,400 test cases, mostly competition and Olympiad math, and asks a model to identify the earliest erroneous step or conclude that all steps are correct. Existing PRMs underperformed both prompted general models acting as critics and the authors' own PRM fine-tuned on PRM800K 8. PRMBench has 6,216 problems and 83,456 step-level labels, and its experiments found significant weaknesses in current PRMs 9. AgentProcessBench extends the test to tool-using agents with 1,000 trajectories and 8,509 human step labels at 89.1% inter-annotator agreement; it found that weaker policies show inflated correct-step ratios because they stop early, and that telling neutral actions from erroneous ones remains hard for current models 10.
The AgentProcessBench finding generalizes into a rule for your own evaluation: report step accuracy alongside trajectory length, or a policy that does less will look like a policy that errs less.
How do you evaluate a generative or rubric reward model?
Evaluate it as an LLM judge: calibrate it against human labels, measure its true-positive and true-negative rates, and attack it with responses that match the rubric's surface without meeting its intent. Generative reward models are now part of mainstream post-training, so this is no longer a niche check.
DeepSeek-V3.2 scores general tasks with a generative reward model in which every prompt has its own rubric 11. Microsoft's reinforcement fine-tuning docs give the minimum bar for any model grader: test the rubric on responses with known scores before training 12. The LLM-as-judge section covers calibration in depth, and the verifier chapter covers the documented ways rubric verifiers get exploited during RL.
What should a reward model evaluation report?
Report the reward model and its lineage, the preference set behind the numbers, accuracy with intervals and shortcut splits, a best-of-N curve graded by a held-out evaluator, and a short training run checked the same way. A single benchmark score does not tell anyone whether the model will work for their policy.
- The reward model, its version, and its lineage relative to the policy.
- The preference set: source, size, how responses were sampled, how they were labeled.
- Pairwise or best-of-N accuracy with a confidence interval, overall and split by length, formatting, position and tone.
- A best-of-N curve graded by a held-out evaluator, not by the reward model.
- A short training run against the reward model with the same held-out evaluator before and after, to catch overoptimization early.
- The date and the policy checkpoint the set was sampled from, so the evaluation can be redone as the policy drifts.