What "good" looks like
A spec generation eval scores two things. First, the artifact: does the agent catch the omissions, ambiguities, inconsistencies, and incorrect assumptions that real maintainers caught in review? Second, downstream success: hand the spec to a coding agent and measure what builds. The first tells you the spec looks right; the second confirms the score means something.
The targets and the state of the field:
- SpecBench's best agent (GPT-5.4 through Codex CLI) scores 44.4%, and no evaluated agent clears 45%: more than half of the weighted critiques experienced maintainers raised go unmatched[1].
- No canonical benchmark exists for free-form product specs. The reusable pattern is SpecBench's method: mine real review comments as ground truth and score recall against them.
SpecBench operationalizes the artifact half. SpecBench (2026) derives tasks from the RFC processes of Kubernetes, React, Rust, TVM, and vLLM. The agent receives an initial design proposal plus the codebase.
It is then judged against the actual expert critiques those proposals drew in RFC history[1].
Read the 44.4% for what v1 measures. The 28 May 2026 preprint scores only the first step, surfacing deficiencies in an initial proposal; revision tasks, where the agent turns findings into a corrected spec, are listed as next work. The agent also sees all past RFC discussions for the project, may submit up to 1.25 times as many deficiencies as the golden set holds, and earns credit only for matches, with consensus ("core") items weighted double[1]. Two caveats to carry into your own version: an LLM panel, not humans, rated the golden items, and matching used a two-judge ensemble (GPT-5.4 and Sonnet 4.6, accepted at 3 of 4 trials) with no human agreement number reported yet[1]. Treat 44.4% as a critique-recall ceiling, not as a measure of spec-writing quality.
This page owns the artifact between requirements and code. The implementation loop that consumes the spec is Autonomous coding; the quality of critiques on someone else's change is Code review.
flowchart TD
REQ["Requirements plus codebase"] --> AGENT["Spec generation agent"]
AGENT --> SPEC["Generated spec"]
REVIEWS["Real RFC review comments"] --> INTR["Intrinsic: recall of human-caught defects"]
SPEC --> INTR
SPEC --> EXTR["Extrinsic: hand spec to coding agent"]
EXTR --> TEST["Build and test pass rate"]
INTR --> GATE{"Rubric climbs and pass rate climbs together?"}
TEST --> GATE
GATE -->|yes| SHIP["Ship"]
GATE -->|no| GAMED["Rubric is being gamed: trust the extrinsic rate"]
Secondary metrics
| Metric | What it catches | How to score |
|---|
| Ambiguity detection P/R | Vague requirement sentences | ReqEval: 200 sentences, roughly half ambiguous, with a cross-domain test; detectors strong in-domain degrade when the domain shifts[2] |
| Acceptance-criteria testability | Criteria no test can target | Score the fraction of criteria a test can be written against; Kiro's requirements.md uses EARS-notation clauses that each map to a test[3] |
| Completeness, consistency, correctness | Weak sections of the draft | Rubric scoring; Krishna et al. found GPT-4 SRS drafts comparable to entry-level engineers and capable of constructive validation feedback on existing drafts[4] |
| Formal-spec quality | Code comprehension expressed through specs | SpecEval scores JML-style specification generation; six SOTA LLMs rate below satisfactory and are sensitive to semantics-preserving perturbations. Read it for what it measures, not requirements authoring[5] |
Offline eval design
The extrinsic metric is the headline: feed the generated spec to a coding agent and measure build and test pass rate.
Commit0 is the cleanest instrument. Agents implement 54 Python libraries from scratch from API specs plus interactive unit tests (the paper's count; the repo README now lists 57 core libraries, so state which split you ran). No agent fully reproduces any library, and interactive feedback measurably raises pass rates[6].
The direction of the arrow matters. SWE-bench measures issue-to-patch against an existing codebase[7]; spec2code puts the spec itself under test. If the coding agent fails on the generated spec but succeeds on the human-written one for the same feature, the spec is the defect.
For the intrinsic half, build a golden-spec set from shipped specs plus their real review threads. Start small and grow it as new specs ship.
Run both metric families and track their correlation on your own data. Intrinsic rubric scores are cheap and tell you which section of the spec is weak. Extrinsic pass rates are expensive and are the only ones that predict shipping. Where the intrinsic score climbs and the extrinsic rate does not, the rubric is being gamed.
Figure: How to read intrinsic versus extrinsic signals in a spec generation eval. Rubric scores against expert critiques (the SpecBench method, where the best agent reaches 44.4% defect recall) only count when the spec2code build-and-test pass rate climbs with them; a rising rubric score with failing builds means the rubric is being gamed.
Online eval design
Clarification rate first: does the agent flag underspecification instead of guessing? GitHub's Spec Kit operationalizes this with [NEEDS CLARIFICATION] markers in generated specs (the /speckit.specify command template caps them at three per spec[8]), plus these commands:
- /speckit.clarify, which asks up to five targeted questions per run and writes the answers back into the spec, and /speckit.checklist, which generates requirements-quality checklists.
- /speckit.analyze, a read-only consistency report across the spec, plan, and tasks[9].
Track marker density and resolution rate; an agent that never asks is guessing. With the cap at three markers, resolution rate per run says more than raw counts.
Convergence passes are a second signal. Spec Kit now ends its loop with /speckit.converge, which checks the implementation against the spec, plan, and tasks and appends any gaps as new tasks under a Convergence section of tasks.md; you rerun implement until it reports Converged (docs fetched as of 2026-10-01)[9]. Count appended tasks per feature, and label each one a spec defect or a coding defect before you call it spec rework.
Then measure downstream rework: the spec-change rate after implementation begins, and the share of implementation rework attributable to a spec defect rather than a coding defect.
Sample specs that produced high rework for human review. The defect taxonomy (omission, ambiguity, inconsistency, wrong assumption) mirrors the offline rubric, so the two datasets compound.
Top benchmarks
- SpecBench, https://arxiv.org/abs/2605.30314. RFC-derived design proposals from Kubernetes, React, Rust, TVM, and vLLM, judged against the real expert critiques from review history[1].
- Commit0, https://arxiv.org/abs/2412.01769. 54 Python libraries in the paper (57 in the current repo README), built from API specs plus interactive unit tests; the extrinsic spec2code instrument[6].
- ReqEval, https://nlp4re.github.io/2020/reqeval.html. 200 requirement sentences for ambiguity detection, with a cross-domain generalization split[2].
- GitHub Spec Kit: the operational loop; [NEEDS CLARIFICATION] markers, checklists, cross-artifact analysis, and convergence passes give you measurable events for free[8].
- Kiro: EARS-notation requirements that map one-to-one to tests, which makes the testability metric mechanical[3]. Its Analyze Requirements step reasons across the whole requirement set for inconsistencies, ambiguities, conflicting constraints, unstated assumptions, and missing edge cases, and streams clarifying questions you can log and score against your own labels[10]. Its property-based testing feature (IDE only) turns EARS requirements into executable properties, a requirement-to-test link you can audit[11].
- Commit0 harness: the spec2code extrinsic check.
- Inspect AI: rubric scoring with a judge from a different model family than the generator.
Common pitfalls
| Pitfall | Why it bites | Fix |
|---|
| No golden set | No canonical benchmark for product specs exists, so judge scores anchor to nothing | Build the shipped-spec-plus-review-thread set |
| Intrinsic scores standing in for downstream success | LLM-judge rubric scores do not predict build success | Always pair them with the extrinsic pass rate, and treat divergence as the finding |
| Self-evaluation bias | When the same model writes and reviews the spec, scores inflate | Judge from a different model family, plus periodic human calibration that is not optional |
| Assuming ambiguity detectors transfer | ReqEval's cross-domain result is that they degrade[2] | Revalidate on sentences from your own domain before trusting the precision number |
Canonical reading
SpecBench is the canonical read because it solves the ground-truth problem that blocks this whole category: instead of asking a judge what a good design doc looks like, it asks whether the agent found what Kubernetes and Rust maintainers actually found[1]. The 44.4% ceiling for the best agent is the honest baseline to beat, and the mining-real-reviews method is the part to copy for your own specs.
Day-1 starter
- Collect 15 shipped specs along with their real review comments; the human-caught defects are your ground truth, no synthetic labels needed.
- Have the agent critique or regenerate each spec, then score recall of the human-caught defects with a judge from a different model family.
- Run one spec2code extrinsic check: hand a generated spec to your coding agent and score it against a hidden test suite the spec author never saw.
Cross-cutting links