What "good" looks like
Autonomous coding is scored by execution: the agent edits files in a repo, and a hidden test patch either passes against its diff or does not. The long-running bar is resolved-issue rate on SWE-bench Verified, the 500-problem human-validated subset that OpenAI shipped in August 2024 to address noise and contamination concerns in the original SWE-bench[1]. It is now saturated (caveat below), so keep Verified as a baseline and read SWE-bench Pro as the harder public board, with a broken-task caveat of its own.
The ship targets:
- A resolved issue edits the right files and passes the hidden test patch against the model's diff. There is no partial credit.
- A minimal Bash+Edit agent reached 49% on Verified in Anthropic's January 2025 Claude 3.5 Sonnet report[2]. The top entry on the official board is now 79.2% (as of 2026-10-01)[3].
- Report the scaffold with the score: a 5% delta between two models can flip if you change the scaffold.
The score is a property of the agent, not the model. The Sonnet writeup is explicit that performance "varies significantly based on this scaffolding" around the model[2]. That single sentence is why this metric is hard to interpret in the wild.
So report the scaffold with the score: which tools the model has, what the system prompt says, whether the agent gets one shot or multiple turns with test feedback. Most public leaderboard numbers are not directly comparable because the scaffolds differ.
One caveat on Verified itself: it is saturated. As of 2026-10-01 the swebench.com Verified board tops out at 79.2% resolved, and its newest entry is dated 2026-02-26[3]. An independent analysis of that board found the top two entries both resolve 396 of 500 instances, and that none of the 29 adjacent pairs in the top thirty is separable under a paired McNemar test, so the order of the leaders carries no information[4]. OpenAI wrote in February 2026 that it had stopped reporting Verified because of flawed tests and training-data contamination, and recommended SWE-bench Pro instead[5]. Pro has the same problem in another form: OpenAI's own July 2026 audit of Pro estimated about 30% of its tasks broken, and Epoch AI's September 2026 review rates Pro Flawed[6]. Use Verified as a regression baseline, read a Pro score as a noisy signal rather than a ranking, put the ranking weight on your own golden 50, and discount any Verified number quoted without its scaffold.
flowchart TD
ISSUE["Issue + repo"] --> AGENT["Agent: model + scaffold"]
AGENT --> DIFF["Model diff"]
DIFF --> HIDDEN{"Hidden test patch passes?"}
HIDDEN -->|no| FAIL["Not resolved: no partial credit"]
HIDDEN -->|yes| REG{"Any previously passing test broken?"}
REG -->|yes| INCIDENT["Regression: a production incident, not a win"]
REG -->|no| REVIEW["Human-review 5 to 10% of resolved patches"]
REVIEW --> SHIP["Report resolved rate + scaffold version"]
Secondary metrics
| Metric | What it catches | How to score |
|---|
| Token cost per problem | Scores that climb only with longer trajectories | Ratio of tokens to resolved issues, the production signal |
| Trajectory length in turns | Fragility hidden by a 40-turn solution to a 5-turn problem | Count turns per resolved issue |
| Regression rate | A patch that fixes the target test but breaks a previously passing test elsewhere in the suite[7] | Re-run the full suite after the patch |
| Sampled patch quality | Passing patches that are wrong-shaped, monkey-patched, or undo a prior fix | Human review on 5 to 10% of resolved issues |
| Reliability over trials | Runs that succeed once but not repeatably | pass^k from tau-bench: probability all k independent attempts succeed[8] |
| Long-horizon capability | Whether the agent handles longer tasks over time | METR's 50%-task-completion time horizon. Original fit: doubling about every 7 months[9]. Time Horizon 1.1 (January 2026): about every 131 days since 2023 and every 89 days since 2024[10]. METR flags measurements above 16 hours as unreliable on its current task suite[11] |
Offline eval design
Run the benchmarks sealed. Run the SWE-bench Pro public set end to end and keep Verified as a regression baseline, then add SWE-Lancer Diamond from the OpenAI preparedness repo for freelance-style tasks. The paper's Diamond split is 502 Upwork tasks worth $500,800, the public part of the 1,488-task full set[12]. Since July 2025 the repo ships 463 of them (198 of the 237 implementation tasks, the ones that run offline, plus all 265 manager tasks, per its README), so a repo run is not comparable to the paper's Diamond numbers. No patches leak into the training pipeline.
Build a "golden 50" from your team's recently merged PRs. Capture the pre-merge state, the failing test, and the merge SHA so the harness can replay deterministically. The replay set is what makes the eval product-specific instead of leaderboard-shaped.
Score with the official SWE-bench harness for parity. Capture the trajectory for every attempt, not just pass/fail, so failures are debuggable later. Pin the model and the scaffold version; both contribute to the score and both move independently. When you compare two agents on the golden 50, run a paired test on per-task outcomes instead of reading the gap in headline score; at the top of the Verified board that test separates nothing[4].
Online eval design
Log every patch attempt with a fixed schema: diff, files touched, tests added or modified, retry count, latency, tokens. Human-review 5 to 10% of accepted patches each week.
Alert on three patterns:
- A regression spike, defined as a week-over-week increase in patches that pass the target test but fail any previously passing test.
- The "passes tests but reverts a previous fix" pattern, detectable by git-blame analysis of removed lines.
- Cost-per-resolved-issue drift greater than 20% week over week, which usually indicates either an agent regression or a distribution shift in incoming issues.
A trajectory viewer beats a metrics dashboard. When the resolved rate drops, the on-call's first action should be replaying ten recent failures with full bash and edit history visible, not a Grafana panel.
Top benchmarks
- SWE-bench Verified, https://www.swebench.com/verified.html. 500 human-vetted problems; saturated (the official board tops out at 79.2% as of 2026-10-01), so use it as a baseline rather than a ranking.
- SWE-bench (original), https://www.swebench.com/. 2,294 problems from 12 Python repos; noisier and contaminated, used only when historical comparison demands it.
- SWE-Lancer Diamond, https://github.com/openai/preparedness. The public split of SWE-Lancer. The paper's Diamond set is 502 Upwork-sourced tasks worth 453,800 in payouts), so report which version you ran. Harder distribution than Verified.
- SWE-bench Pro, https://labs.scale.com/leaderboard/swe_bench_pro_public. The contamination-resistant successor: 1,865 problems from 41 actively maintained repos, split into public, held-out, and commercial sets. Deliberately harder and less contaminated than Verified. Top public-leaderboard rows as of 2026-10-01: Muse Spark 1.1 at 61.5%, GPT-5.4 (xHigh) at 59.1%, Muse Spark at 55.0%, Claude Opus 4.6 (thinking) at 51.9%, with confidence intervals of roughly plus or minus 3 to 4 points, so the top two are not separable. Starred rows ran under the mini-swe-agent harness and the rest under an uncapped-cost, 250-turn configuration, so the board itself mixes scaffolds[13]. Audits have estimated 30% or more of Pro's tasks broken, and Epoch AI's September 2026 review rates it Flawed[6].
- Terminal-Bench, https://www.tbench.ai/. Agent tasks in a real terminal environment, a different tool surface than repo editing[14]. The task set is versioned and rolling: version 3.0 had 74 tasks, and version 4.0 (August 28, 2026) has 66 after removing eight for saturation, refusals, public solutions, and quality or platform issues. Scores from earlier versions need re-runs and are not comparable, so report the version with the number[15].
- Commit0, https://arxiv.org/abs/2412.01769. Greenfield generation rather than issue fixing: implement 54 Python libraries from scratch against API specs and interactive unit tests. No agent fully reproduces any library, and interactive test feedback measurably raises pass rates[16].
Figure: The autonomous coding benchmarks around SWE-bench Verified. Verified (500 human-vetted problems) was the ranking standard and is now saturated; SWE-Lancer Diamond (463 public freelance tasks) supplies a harder distribution and SWE-bench Pro (1,865 problems from 41 repos) a contamination-resistant one with broken tasks of its own, while Terminal-Bench and Commit0 (54 libraries built from scratch) stress different task surfaces.
- SWE-bench harness: the only way to get comparable Verified numbers.
- OpenHands: one of the two most common open-source agent scaffolds; worth comparing on your golden 50 because tool surfaces differ.
- Aider: the other common open-source scaffold to compare on your golden 50.
- mini-SWE-agent: the SWE-agent team's bash-only agent, about 100 lines for the agent class. Run every candidate model through it to hold the scaffold constant, which is what makes a model comparison mean something[17].
- Harbor: the evaluation framework Terminal-Bench 4.0 runs on; use it to reproduce the public Terminal-Bench numbers[15].
- Inspect AI: UK AISI's research-grade harness with strong sandbox primitives; the cleanest path for adding new evaluators alongside the standard ones.
- Braintrust or Langfuse: production trajectory logging, including per-step token accounting the official harness does not capture.
Common pitfalls
| Pitfall | Why it bites | Fix |
|---|
| Ranking agents on headline SWE-bench or Verified | The original set is contaminated and noisy, and Verified is saturated: the top entries are statistically tied | Rank on your golden 50; read SWE-bench Pro as a harder but noisy board and keep Verified as a regression baseline |
| Treating "tests pass" as ground truth | Hidden tests can be trivial; patches can monkey-patch tests or revert unrelated assertions | Sample for human review |
| Ignoring agent scaffolding when comparing models | The same model, different scaffold, swings the score meaningfully[2] | Report the scaffold and pin its version |
| Skipping the regression check | A patch that resolves the issue but breaks another test is a production incident, not a win | Re-run the full suite and alert on regressions |
Canonical reading
Anthropic's SWE-bench Sonnet report is the most useful single reading because it makes the scaffolding point explicit and shows a minimal Bash+Edit agent reaching competitive numbers without elaborate planning[2]. Read it alongside the original Jimenez et al. paper for the benchmark design[7] and the SWE-bench Verified site notes for what changed in the 500-problem subset[3].
Day-1 starter
- Pick 10 recently merged PRs from your repo. For each, save the pre-merge SHA, the failing test, and the human-authored diff as gold. Replay them as SWE-bench-style tasks via the official harness.
- Wire pass/fail to your CI on a small smoke subset (3 to 5 problems) per PR; run the full set nightly. Hamel's field guide framing applies directly: remove friction from looking at failed trajectories[18].
- Stand up a trajectory viewer. At minimum, the bash log and the unified diff for every attempt should be one click from the score. The viewer is the eval; the score is the headline.
Cross-cutting links