What "good" looks like
A text-to-SQL eval runs the generated query against a real database and checks the rows it returns, not the query text. Correctness is measured by execution accuracy (EX): do the returned rows match the gold query's rows? Efficiency is measured alongside it, because a correct-but-slow query is broken in production.
The targets on BIRD-bench:
- Human EX: 92.96%.
- Top systems: 82.95% test EX (GrainSQL, Purdue University, submitted 2026-09-26; leaderboard fetched as of 2026-10-01). The next three entries sit between 81.95% and 82.39%, and the gap to humans is about 10 points.
- Bare frontier models score well below the top of the board. The same leaderboard lists GPT-5.5-xhigh at 72.55% (2026-04-28) and a Claude Opus 4.6 baseline at 70.15% (2026-02-21). Compare your own stack against rows like those, not against the top entry.
- Report both EX and R-VES (Reward-based Valid Efficiency Score), which jointly rewards correctness and query efficiency. The top test R-VES is 84.10% (GrainSQL) against 83.26% for the human row.
The primary metric is EX on BIRD-bench, the 12,751-pair benchmark across 95 databases (33.4 GB total) that introduces realistic dirty-data and efficiency considerations beyond Spider[1]. Reporting both EX and R-VES is the responsible default: R-VES exists precisely because a correct-but-slow query is broken in production.
Read the leaderboard with one caveat. Every row above 67.41% test EX uses oracle knowledge, the per-question evidence hint that ships with each BIRD question, and 67.41% is the best score on the board from a row without it. Your users do not type evidence. In November 2025 the BIRD team said it would open an interactive track that drops oracle evidence and has the model ask the user instead[1]. The board fetched on 2026-10-01 shows no such track, so expect production EX below the leaderboard figure.
Spider remains useful for cleaner ablation testing but is saturated for frontier models: its leaderboard tops out at 91.2% execution accuracy, last updated in November 2023[2]. BIRD is where the discrimination lives in 2026.
Spider 2.0 is the enterprise-workflow version of the problem. It has 632 real-world workflow problems over databases that often exceed 1,000 columns on BigQuery, Snowflake, and similar systems, with SQL that often runs past 100 lines[3][4]. As of 2026-10-01 the leaderboard's top scores are 96.70% on Spider 2.0-Snow (547 tasks), 76.23% on Spider 2.0-Lite (547 tasks), and 65.6% on Spider2-DBT (68 repository-level tasks). Snow is close to ceiling for the best agent stacks, so Lite and DBT are where the headroom is. Reach for Spider 2.0 when your warehouse has hundreds of tables and the hard part is finding the right ones.
flowchart LR
Q["NL question"] --> SQL["Generated SQL"]
SQL --> VALID{"Parses and executes?"}
VALID -->|no| FAIL["Invalid: fix generation"]
VALID -->|yes| ROWS["Row-set hash vs gold"]
ROWS --> EX{"EX: rows match?"}
EX -->|no| FAIL2["Wrong answer"]
EX -->|yes| EFF{"R-VES: efficient? explain-plan clean?"}
EFF -->|no| SLOW["Correct but slow: regression"]
EFF -->|yes| SHIP["Ship"]
Figure: Execution accuracy (EX) on BIRD-bench: top text-to-SQL systems reach 82.95% on the test set (leaderboard fetched as of 2026-10-01) while humans score 92.96%, a 10.01-point gap. BIRD's 12,751 question-SQL pairs across 95 databases (33.4 GB) are why it discriminates where Spider is saturated.
Secondary metrics
| Metric | What it catches | How to score |
|---|
| Query validity rate | SQL that fails to parse or execute | Did the generated SQL parse and execute, regardless of correctness? |
| Execution time and cost | Correct rows at a runaway price | A 10x cost increase versus the gold query is a regression even if the rows match |
| Row-set equivalence | ORDER BY ambiguity and column-order differences that exact-match scoring penalizes | Compare row sets, not query text |
| Explain-plan red flags | Valid SQL that triggers a full table scan or an exploding join | Inspect the explain-plan for the pattern |
Offline eval design
Public sets first. Spider for breadth, BIRD for the harder discriminator. If the full 33.4 GB BIRD set is too heavy for CI, BIRD's Mini-Dev (500 examples) ships in SQLite, MySQL, and PostgreSQL[1]. In November 2025 BIRD released a cleaner dev split (bird-sql-dev-1106) after a quality-control pass, and the leaderboard tags submissions scored on it as New Dev. Check which split a published dev number used before you compare it with yours.
Then build a warehouse-specific set of your top 100 ad-hoc queries with gold SQL and expected row-set hashes. The row-set hash is the artifact that makes scoring deterministic across ORDER BY ambiguity.
Run on a copy of the production schema, sanitized of PII. The dirty-data behavior in BIRD is what makes the score predict production quality, and you only get that if your eval schema looks like production.
Capture four things per query: EX (correctness), R-VES (efficiency), execution time, and the explain-plan. The composite tells you whether the model is shipping correct, performant queries or correct queries that will melt the warehouse at 9 AM.
Online eval design
Log the full funnel: every natural-language question, the generated SQL, the execution result, and the user's next action.
User-edit rate is the most useful single online signal. If users edit the generated SQL before running it, the model failed even if the SQL was technically correct.
Alert on schema-drift breakages. A column rename in the warehouse silently invalidates the prompts.
Track per-table accuracy. The model usually understands the tables it has seen often and degrades on rarely-queried tables.
Top benchmarks
- BIRD-bench, https://bird-bench.github.io/. 12,751 pairs, 95 databases; human EX 92.96%, top systems 82.95% test EX as of 2026-10-01.
- Spider, https://yale-lily.github.io/spider. 10,181 questions, 200 databases; cleaner, saturated[2].
- Spider 2.0, https://spider2-sql.github.io/. 632 enterprise workflow problems in three settings: Snow, Lite, and DBT[4].
- LiveSQLBench, https://livesqlbench.ai/. Contamination-resistant by design: each release has an open dev set and a hidden test set, and the hidden set becomes the next release's dev set. Large-v1 (2026-03-02) has 18 databases of about 1,000 columns each and 480 tasks, and it covers CRUD, not just SELECT[5].
- BIRD-Interact, linked from the BIRD site. An interactive setting (ICLR 2026 oral) where the model works through ambiguity with a user simulator, the direction BIRD says it is moving instead of oracle evidence[1].
- Your warehouse top-100. The eval that predicts whether your data team will ship the model.
- OpenAI cookbook SQL eval: the
How_to_evaluate_LLMs_for_SQL_generation.ipynb notebook, a good template for the execution-accuracy harness[6].
- Defog sql-eval: an open-source harness that runs the gold and generated queries and compares the result sets (exact and subset match) across Postgres, Snowflake, BigQuery, MySQL, SQLite, and SQL Server[7].
- Vanna: an open-source text-to-SQL agent framework. Its repository was archived read-only on 2026-03-29, so read it for ideas rather than depend on it, and note that its README documents no eval harness[8].
- Inspect AI: sandboxed deterministic runs against a reference schema.
Common pitfalls
| Pitfall | Why it bites | Fix |
|---|
| Exact-match scoring | Multiple valid SQLs exist for almost every non-trivial question; exact match penalizes the model for choosing a different correct path | Use execution accuracy |
| Ignoring efficiency | A correct query that costs 10x is broken in production | Report R-VES, which BIRD defines to fix this[1] |
| Not testing on dirty data | If your eval schema is clean, your score does not predict production quality | Use BIRD-style realistic schemas with messy values[1] |
| Treating user-edit rate as a vanity metric | It is the most actionable online signal in this category | Track and alert on it, then read the edited queries: error analysis on real data is the highest-ROI step[9] |
Canonical reading
The BIRD-bench paper and site are the canonical reading because BIRD is explicitly designed as "the first text-to-SQL benchmark designed to encourage semantic parsers to produce SQL queries that are not only correct but also efficient"[1].
The efficiency framing is the operational thesis: in production, the metric is execution-accuracy-conditioned-on-acceptable-cost, not pure correctness. Pair with the OpenAI cookbook for the practical harness implementation[6].
Day-1 starter
- 50 natural-language questions to gold SQL to expected row-set hash. The hash is the artifact that makes scoring robust to ORDER BY ambiguity.
- Execution harness on a sanitized copy of the production schema. The schema shape is what makes the eval predict production quality.
- Track EX and R-VES per release. EX is correctness; R-VES is correctness-given-acceptable-cost; reporting both keeps the model from optimizing for the headline at the expense of the warehouse bill.
Cross-cutting links