What "good" looks like
A spec generation eval scores two things. First, the artifact: does the agent catch the omissions, ambiguities, inconsistencies, and incorrect assumptions that real maintainers caught in review? Second, downstream success: hand the spec to a coding agent and measure what builds. The first tells you the spec looks right; the second confirms the score means something.
The targets and the state of the field:
- SpecBench's best agent scores 44.4%: more than half of what experienced maintainers caught goes unnoticed[1].
- No canonical benchmark exists for free-form product specs. The reusable pattern is SpecBench's method: mine real review comments as ground truth and score recall against them.
SpecBench operationalizes the artifact half. SpecBench (2026) derives tasks from the RFC processes of Kubernetes, React, Rust, TVM, and vLLM. The agent receives an initial design proposal plus the codebase.
It is then judged against the actual expert critiques those proposals drew in RFC history[1].
This page owns the artifact between requirements and code. The implementation loop that consumes the spec is Autonomous coding; the quality of critiques on someone else's change is Code review.
flowchart TD
REQ["Requirements plus codebase"] --> AGENT["Spec generation agent"]
AGENT --> SPEC["Generated spec"]
REVIEWS["Real RFC review comments"] --> INTR["Intrinsic: recall of human-caught defects"]
SPEC --> INTR
SPEC --> EXTR["Extrinsic: hand spec to coding agent"]
EXTR --> TEST["Build and test pass rate"]
INTR --> GATE{"Rubric climbs and pass rate climbs together?"}
TEST --> GATE
GATE -->|yes| SHIP["Ship"]
GATE -->|no| GAMED["Rubric is being gamed: trust the extrinsic rate"]
Secondary metrics
| Metric | What it catches | How to score |
|---|
| Ambiguity detection P/R | Vague requirement sentences | ReqEval: 200 sentences, roughly half ambiguous, with a cross-domain test; detectors strong in-domain degrade when the domain shifts[2] |
| Acceptance-criteria testability | Criteria no test can target | Score the fraction of criteria a test can be written against; Kiro's requirements.md uses EARS-notation clauses that each map to a test[3] |
| Completeness, consistency, correctness | Weak sections of the draft | Rubric scoring; Krishna et al. found GPT-4 SRS drafts comparable to entry-level engineers and capable of constructive validation feedback on existing drafts[4] |
| Formal-spec quality | Code comprehension expressed through specs | SpecEval scores JML-style specification generation; six SOTA LLMs rate below satisfactory and are sensitive to semantics-preserving perturbations. Read it for what it measures, not requirements authoring[5] |
Offline eval design
The extrinsic metric is the headline: feed the generated spec to a coding agent and measure build and test pass rate.
Commit0 is the cleanest instrument. Agents implement 54 Python libraries from scratch from API specs plus interactive unit tests. No agent fully reproduces any library, and interactive feedback measurably raises pass rates[6].
The direction of the arrow matters. SWE-bench measures issue-to-patch against an existing codebase[7]; spec2code puts the spec itself under test. If the coding agent fails on the generated spec but succeeds on the human-written one for the same feature, the spec is the defect.
For the intrinsic half, build a golden-spec set from shipped specs plus their real review threads. Start small and grow it as new specs ship.
Run both metric families and track their correlation on your own data. Intrinsic rubric scores are cheap and tell you which section of the spec is weak. Extrinsic pass rates are expensive and are the only ones that predict shipping. Where the intrinsic score climbs and the extrinsic rate does not, the rubric is being gamed.
Figure: How to read intrinsic versus extrinsic signals in a spec generation eval. Rubric scores against expert critiques (the SpecBench method, where the best agent reaches 44.4% defect recall) only count when the spec2code build-and-test pass rate climbs with them; a rising rubric score with failing builds means the rubric is being gamed.
Online eval design
Clarification rate first: does the agent flag underspecification instead of guessing? GitHub's Spec Kit operationalizes this with [NEEDS CLARIFICATION] markers in generated specs, plus these commands:
- /speckit.clarify and /speckit.checklist for resolving markers.
- /speckit.analyze for cross-artifact consistency checks[8].
Track marker density and resolution rate; an agent that never asks is guessing.
Then measure downstream rework: the spec-change rate after implementation begins, and the share of implementation rework attributable to a spec defect rather than a coding defect.
Sample specs that produced high rework for human review. The defect taxonomy (omission, ambiguity, inconsistency, wrong assumption) mirrors the offline rubric, so the two datasets compound.
Top benchmarks
- GitHub Spec Kit: the operational loop; [NEEDS CLARIFICATION] markers, checklists, and cross-artifact analysis give you measurable events for free[8].
- Kiro: EARS-notation requirements that map one-to-one to tests, which makes the testability metric mechanical[3].
- Commit0 harness: the spec2code extrinsic check.
- Inspect AI: rubric scoring with a judge from a different model family than the generator.
Common pitfalls
| Pitfall | Why it bites | Fix |
|---|
| No golden set | No canonical benchmark for product specs exists, so judge scores anchor to nothing | Build the shipped-spec-plus-review-thread set |
| Intrinsic scores standing in for downstream success | LLM-judge rubric scores do not predict build success | Always pair them with the extrinsic pass rate, and treat divergence as the finding |
| Self-evaluation bias | When the same model writes and reviews the spec, scores inflate | Judge from a different model family, plus periodic human calibration that is not optional |
| Assuming ambiguity detectors transfer | ReqEval's cross-domain result is that they degrade[2] | Revalidate on sentences from your own domain before trusting the precision number |
Canonical reading
SpecBench is the canonical read because it solves the ground-truth problem that blocks this whole category: instead of asking a judge what a good design doc looks like, it asks whether the agent found what Kubernetes and Rust maintainers actually found[1]. The 44.4% ceiling for the best agent is the honest baseline to beat, and the mining-real-reviews method is the part to copy for your own specs.
Day-1 starter
- Collect 15 shipped specs along with their real review comments; the human-caught defects are your ground truth, no synthetic labels needed.
- Have the agent critique or regenerate each spec, then score recall of the human-caught defects with a judge from a different model family.
- Run one spec2code extrinsic check: hand a generated spec to your coding agent and score it against a hidden test suite the spec author never saw.
Cross-cutting links