What "good" looks like
A structured-output eval scores two things at once: does the output validate against the schema, and are the field values actually right. The primary metric is the product of those two rates, because either alone lies about production quality.
The ship targets on your domain set:
The two numbers track different problems, so report both. Validity alone is necessary-not-sufficient; semantic correctness alone is unmeasurable when validity is 60%.
No standalone benchmark predicts your fields because the failure modes are domain-specific: which fields, which types, which enums. Two public sets are useful scaffolding. JSONSchemaBench, about 10,000 real-world JSON schemas, compares constrained-decoding stacks (Guidance, Outlines, llama.cpp, XGrammar, OpenAI, Gemini) on efficiency, schema coverage, and output quality, so use it to choose an engine, not to estimate your field accuracy[1]. The BFCL v4 format-sensitivity subset is the closest public proxy for value correctness: 200 single-turn function-call entries rerun under 26 prompt variants, scored with BFCL's AST-based checker[2]. The leaderboard (last updated 2026-04-12, fetched as of 2026-10-02) states that format-sensitivity cases are only supported for prompt (non-FC) models, so it does not score models through their native function-calling interface[3].
flowchart TD
IN["Input"] --> GEN["Model generates JSON"]
GEN --> V{"jsonschema validates?"}
V -->|no| LOG["Log raw output, retry or reject"]
V -->|yes| SEM{"Field values correct?"}
SEM -->|no| FIELD["Per-field failure: enum, date, currency, ID"]
SEM -->|yes| GATE{"Validity above 99% and per-field in the 90s?"}
GATE -->|no| FIX["Fix the weaker axis"]
GATE -->|yes| SHIP["Ship"]
Figure: The primary metric for structured output and JSON mode multiplies schema validity by per-field semantic accuracy. A system at 99% validity and 94% field accuracy scores 93% and ships, while perfect JSON with wrong values (100% x 60%) and correct values inside invalid JSON (60% x 94%) both collapse to 60% or below.
Secondary metrics
| Metric | What it catches | How to score |
|---|
| Field omission rate | Optional fields that should have been populated but were left out | Fraction of expected fields present |
| Extra-field rate | Fields the model added that are not in the schema | Count of out-of-schema keys per output |
| Enum hallucination | An enum value not in the allowed set | Membership check against the allowed set; log case-only mismatches separately, because Anthropic does not guarantee enum capitalization[4] |
| Format adherence | Malformed dates, currencies, and identifiers | Format checks for each typed field |
| Retry count | Constrained decoding looping | Per-output retry budget, a cost signal |
| Refusal and truncation | A safety refusal or a max_tokens cut-off returns a body that does not match the schema, even in strict mode[4][5] | Count these stop reasons separately from schema failures |
| Cold-start latency | The first request with a new or changed schema pays a grammar-compile cost[4][5] | Report cold and warm P95 separately |
Offline eval design
Build the fixture set. Assemble 100 inputs with gold JSON. Score with jsonschema validate plus per-field accuracy:
- String equality for IDs and enums.
- Type-and-range checks for numbers.
- ISO-format checks for dates.
Compare strict against unconstrained. Run strict mode (constrained decoding via the provider's schema enforcement) against unconstrained (prompt-engineered JSON). Strict mode usually wins on validity and loses a few points on semantic correctness, because the constraint can force the model into a local optimum that satisfies the schema but is wrong. The eval should make both visible.
Read the provider's schema limits before trusting a green validity number. Anthropic's structured outputs do not support numeric bounds (minimum, maximum) or string length limits, and its SDK helpers strip them from the schema sent to the model, then enforce them only if you validate the response through the helper[4]. Gemini lists minimum and maximum as supported, and still tells you to always validate values in your application[6]. Keep type-and-range checks in your scorer, not only in the schema.
Test for format sensitivity. Sclar et al. found open-source LLMs extremely sensitive to subtle prompt-format changes in few-shot settings, with differences of up to 76 accuracy points on LLaMA-2-13B[7]. BFCL's study of 39 models across 26 prompt variants shows format choice matters for tool calls too: models generally scored higher with Python and JSON return formats than with either XML variant[2]. Format sensitivity is a real and underestimated source of variance. Run multiple seeds and multiple plausible prompt phrasings, not just one.
Online eval design
Validate at the boundary. The boundary is the system contract: an invalid JSON should never propagate. Log parse failures with the raw output so the failure mode is debuggable.
Alert on the right signals. Alert on validity-rate drops greater than 1% and on any field's per-value accuracy regressing. The "extra fields" alert is often the leading indicator of a model swap or prompt regression that does not affect validity.
Go per-field for critical products. If you ship a critical structured-output product, build a per-field accuracy dashboard. Per-field beats global because the per-field failure modes (date parsing, enum drift, currency formatting) are independent, and the global number aggregates them away.
Top benchmarks
- BFCL v4 format-sensitivity subset, https://gorilla.cs.berkeley.edu/leaderboard.html (last updated 2026-04-12, fetched as of 2026-10-02). The closest public proxy; AST-based scoring, prompt (non-FC) models only.
- JSONSchemaBench, https://github.com/guidance-ai/jsonschemabench. About 10,000 real-world schemas; compares decoding engines on efficiency, coverage, and quality.
- Instructor and Outlines test suites. Domain-specific; useful as scaffolding but not benchmarks per se.
- Your domain fixture set. The eval that actually predicts production quality.
- Instructor: Pydantic-based output validation with auto-retry on validation error[8]. About 8.4 million PyPI downloads in the last 30 days (pypistats, fetched as of 2026-10-02)[9].
- Outlines: JSON Schema, regex, or context-free-grammar constraints applied during decoding, on self-hosted backends such as vLLM, transformers, and llama.cpp[10].
- Provider strict mode (Anthropic, OpenAI, Google): schema-shape guarantees during generation[11]. Anthropic's structured outputs are generally available with no beta header: JSON outputs go through output_config.format, strict tool use through strict: true, and the older output_format parameter is deprecated[4].
- jsonschema: the Python library for the validity check.
- Inspect AI: sandboxed, reproducible runs. The built-in scorers cover text match, pattern, and model grading, so field-level assertions on parsed JSON are a custom scorer[12].
Common pitfalls
| Pitfall | Why it bites | Fix |
|---|
| Measuring only "is JSON" without measuring values | The most common single mistake; validity is necessary but not sufficient | Score per-field semantic correctness alongside validity |
| Letting the model choose the schema | The schema is the contract; if the model can vary it, downstream code cannot rely on anything | Fix the schema and bind both prompt and eval to it |
| Not testing constrained and unconstrained side by side | Strict mode wins on validity and sometimes loses on value correctness; reporting only one hides the trade | Run both and report both numbers |
| Ignoring format sensitivity | Findings show large variance across plausible prompt rewordings; a single-prompt eval undersells the noise | Test multiple seeds and phrasings[7] |
Canonical reading
The Outlines documentation is the most useful single reading, because it frames the structured-output problem as a decoding-time constraint problem rather than a prompt-engineering problem[10].
The framing is the operational insight: if you want guaranteed validity, constrain decoding; if you want maximum flexibility, accept retries. Pair with Instructor's docs for the Pydantic-plus-retry pattern that most teams use in production[8].
Day-1 starter
- Pydantic models for every output type. The model is the contract; both the prompt and the eval bind to it.
- 50 golden input-to-output pairs in CI. Per-field accuracy, not aggregate; the per-field view is what catches enum drift and date-format regressions.
- Per-field accuracy dashboard with alert thresholds. The dashboard is the production eval; the CI run is the pre-deployment gate.
Cross-cutting links