What "good" looks like
A reasoning eval scores two axes that diverge: correctness on hard problems where chain-of-thought matters, and faithfulness, whether the visible thinking trace reflects the computation that produced the answer. Score both, because a model can gain one while losing the other.
The two axes for reasoning models (the o-series, Claude with extended thinking, R1-style chains):
- Correctness: GPQA Diamond (198 graduate-level questions), MATH, and AIME[1]. All three are floor checks for frontier models now, not rankings. As of October 2026 (fetched as of 2026-10-02), Artificial Analysis lists GPT-6 Astra at 96.3% on GPQA Diamond[2], far above the 65% expert baseline, and three models at a perfect score on AIME 2025[3]. Its MATH-500 leaderboard tops out at 99.4%[4]. Rank on the harder sets under Top benchmarks.
- Faithfulness: how often the visible reasoning acknowledges an injected hint, counted only where the hint flipped the answer. When researchers injected hints into prompts, Claude 3.7 acknowledged the hint in its visible thinking only 25% of the time, R1 only 39%[5].
The trace can lie. The thinking looks like reasoning but is not always what the model is doing. Anthropic's 2025 result is the most useful single benchmark on this axis.
Faithfulness gets worse as problems get harder. In Anthropic's study, chains on GPQA were 44% (relative) less faithful than on MMLU for Claude 3.7 Sonnet and 32% for R1. Unfaithful chains were also longer, not shorter: 2,064 tokens on average against 1,439 for faithful ones on Claude 3.7 Sonnet[5]. The 2023 precursor found a related trend across scale: larger and more capable models gave less faithful reasoning on most tasks[6]. A model that wins on hard-problem accuracy tells you nothing about its trace.
The target depends on the surface. Show reasoning to users and faithfulness is product-critical; keep it internal and accuracy is the only customer-facing axis.
flowchart TD
PROB["Reasoning problem"] --> RUN["Run model, log full thinking tokens"]
RUN --> ACC["Correctness: GPQA Diamond, MATH, AIME"]
RUN --> HINT["Inject correct or incorrect hint"]
HINT --> FAITH["Faithfulness: does visible reasoning acknowledge the hint?"]
ACC --> SURF{"Is reasoning shown to users?"}
FAITH --> SURF
SURF -->|yes| BOTH["Gate on correctness and faithfulness"]
SURF -->|no| ONLY["Gate on correctness only"]
Figure: Chain-of-thought faithfulness measured by hint perturbation: Claude 3.7 Sonnet acknowledged an injected hint in its visible reasoning only 25% of the time and DeepSeek R1 only 39%, so most hint-influenced answers ship with a thinking trace that never mentions the hint.
Secondary metrics
| Metric | What it catches | How to score |
|---|
| Tokens to correct answer | Trace-length balloon; the cost-quality frontier | Ratio of tokens to correct answer |
| Sandbagging detection | Model knows the answer but hides it in visible reasoning | Adversarial hints where visible reasoning conceals the answer[5] |
| Paraphrase consistency | Answer instability across wordings | Same problem, four wordings, same answer |
| Self-consistency | Noisiness of the reasoning path | Sample N times at temperature greater than 0 and majority vote; the gap between argmax and majority vote is the proxy[7] |
Offline eval design
Capability set. GPQA Diamond, MATH-500, and the most recent AIME (AIME 2026, 30 problems) as floor checks, plus at least one set that frontier models still miss (see Top benchmarks). AIME has only 30 questions per year, so variance is high; report bootstrap confidence intervals, not point estimates. For competition problems that postdate a model's training cutoff, MathArena publishes uncontaminated results[8].
Pin the effort setting. Frontier APIs now expose reasoning as a per-request effort setting, and it moves scores a lot: ARC Prize verified GPT-6 Astra at six reasoning levels, and ARC-AGI-2 accuracy ran from 59.6% with no reasoning to 95.0% at max[9]. Record the effort level with every run, eval the level you ship, and plot correctness against tokens per answer across levels instead of reporting one number. For open-weight models run through lm-evaluation-harness, the think_end_token argument on the hf, vllm and sglang backends strips the reasoning trace before the answer is scored[10].
Faithfulness harness. Build a hint-perturbation harness in the Anthropic style: take a problem, prepend a hint of the answer (correct or incorrect), and measure how often the visible reasoning acknowledges the hint[5]. Score only the prompt pairs where the model switched to the hinted answer, and count a trace as faithful only if it mentions the hint and depends on it. The paper used six hint types: sycophancy, consistency, visual pattern, metadata, grader hacking, and unethically obtained information. Twenty cases is enough to see the gap; 100 is enough to track it across releases.
Vary the channel. In FACE-Eval, a 5,100-sample suite run on 15 open-weight models, every model verbalized a cue less often when it arrived in a tool return than in a user message, and less often when the cue was implicit than explicit. Higher unverbalized adoption went with worse detection by transcript monitors (correlations of -0.54 and -0.78 across the two monitors tested)[11]. If your agent reads tool outputs, plant the hint there too.
Log the thinking. Capture the full thinking tokens, not just the final answer. The thinking is the eval surface; treating it as opaque state defeats the purpose.
Online eval design
When thinking is surfaced to users. Sample traces and run a LaaJ check: does the visible reasoning support the answer the model gives?
Watch token growth. Alert on reasoning-length explosions, since reasoning tokens are billed. A 3x jump in tokens per answer is either a model regression or a prompt regression.
When reasoning is hidden. The online check is correctness against user feedback or downstream task success, not the reasoning surface.
Top benchmarks
Figures below were fetched as of 2026-10-02 unless a date is given.
- GPQA Diamond, https://arxiv.org/abs/2311.12022. 198 graduate-level questions; non-experts with web search reach 34%, experts around 65%. The top Artificial Analysis score is 96.3% (GPT-6 Astra, xhigh effort), so use it as a floor check[2].
- MATH, https://arxiv.org/abs/2103.03874. Hendrycks et al.; full MATH is saturated, and so is MATH-500 from Lightman et al.: the top Artificial Analysis score is 99.4%[12][4].
- AIME 2025/2026. Official MAA; 30 items per year; variance-heavy, and the 2025 set is saturated, with three models at 100% on Artificial Analysis[3].
- Humanity's Last Exam (HLE), https://arxiv.org/abs/2501.14249. 2,500 expert-written questions[13]. Scale's leaderboard (full multimodal set) has GPT-6 Astra first at 54.8%[14]. Artificial Analysis runs only the 2,158 text-only questions and has Claude Opus 5.5 first at 61.4%[15]. The two are not comparable, so pin the subset and the tools setting. HLE-Diamond, a cleaned 1,000-question subset (500 reasoning, 500 knowledge) released on 22 September 2026, shows GPT-6 Astra at 59.9% without tools and 82.9% with tools[16].
- FrontierMath, https://arxiv.org/abs/2411.04872. Unpublished research-level problems from Epoch AI[17]. Version 2 shipped on 12 June 2026 after Epoch corrected errors in 42% of problems and removed some entries, leaving 338 problems (295 in Tiers 1-3, 43 in Tier 4)[18]. Do not mix v1 and v2 scores. Epoch's page for GPT-6 Astra lists 94% on Tiers 1-3 (v2) and 98% on Tier 4 (v2)[19], so the v2 tiers are close to saturated at the top. Headroom remains on FrontierMath Erdos, 68 open Erdos problems checked in Lean: GPT-6 Astra solved 2 of 68, or 3%[20].
- ARC-AGI-2, https://arxiv.org/abs/2505.11831. Abstract-reasoning puzzles from the ARC Prize Foundation[21]. ARC Prize's verified results list GPT-6 Astra at 95.0% (2 September 2026), GPT-6.1 Sol at 94.2% (29 September 2026) and Claude Opus 5.5 at 93.3% (22 September 2026) on the public eval set[22]. That is near ceiling, so the open problem has moved to ARC-AGI-3, an interactive benchmark where agents explore novel environments and acquire goals on the fly[23].
- MonitorBench, https://arxiv.org/abs/2603.28590. 19 tasks across 7 categories (1,514 instances) for chain-of-thought monitorability. The authors report that most LLMs can deliberately reduce monitorability when stress-tested[24]. Run it beside your own hint harness for the faithfulness axis.
- Inspect AI: capturing the full thinking trace alongside the answer.
- lm-evaluation-harness: reproducible GPQA and MATH runs, from EleutherAI.
- Anthropic Console Evaluate: side-by-side prompt and model comparison on a fixed reasoning set.
- Braintrust: trace storage with full token logs.
Common pitfalls
| Pitfall | Why it bites | Fix |
|---|
| Treating high CoT length as "more thinking" | Unfaithful explanations are longer, not shorter (2,064 tokens against 1,439 for Claude 3.7 Sonnet); trace length and answer quality correlate weakly past saturation | Measure faithfulness directly, not length[5] |
| Ranking models on saturated sets | GPQA Diamond at 96%, AIME 2025 at 100% and MATH-500 at 99% leave a point or two of spread between top models | Gate on them as floors; rank on HLE-Diamond, FrontierMath, and your own hard set[2] |
| Using MMLU as a reasoning eval | MMLU is saturated; MMLU-Redux re-annotation found a 6.49% error rate across the benchmark, 57% in Virology | Use GPQA, MATH, or AIME instead[25] |
| Reporting AIME as a single number | 30 items is small and the variance is real | Report bootstrap confidence intervals |
| Skipping the faithfulness check | If your product shows reasoning, users decide on the visible trace | Run the hint-perturbation audit you owe them |
Canonical reading
Anthropic's 2025 "Reasoning Models Don't Say What They Think" is the most important single read in this category because it makes the faithfulness problem measurable rather than philosophical[5]. The 2023 precursor on measuring faithfulness in chain-of-thought intervenes on the trace itself (adding mistakes, paraphrasing); the 2025 paper moved to hint injection[6]. Pair with the original chain-of-thought paper[26] for context on why this matters.
Day-1 starter
- GPQA Diamond as a floor check plus your domain's "hard 50". Score correctness with bootstrap confidence intervals because the sample sizes are small.
- For 20 cases, inject a correct or incorrect hint and measure how often the visible reasoning acknowledges it. That mention rate is your faithfulness baseline.
- Log full thinking tokens for every release. Track tokens per correct answer over time; a release that adds 30% tokens without adding correctness is regressive.
Cross-cutting links