Text-to-SQL Is Not Business Understanding: Why Valid SQL Can Still Be Wrong

A query gets generated from a plain-English question. It parses. It runs. It returns a number in under two seconds. It goes into a deck, and then a finance lead asks which definition of revenue it used — and nobody in the room can say.

That moment is the whole problem with text-to-SQL accuracy in one scene. The system did not fail. The SQL was valid. The result was wrong.

Two contracts, and only one of them is being tested

Every natural-language database query has to satisfy two separate contracts, and they are not the same contract.

The syntactic contract asks: is this valid SQL that executes against this schema without error? This is a translation problem, and models have gotten genuinely good at it.

The semantic contract asks: does this query mean what the business means by the question that was asked? Which metric definition, which join path, which aggregation level, which exclusions, which time window, which organizational scope.

Execution success measures the first contract. It says nothing about the second. When a system reports a 90% success rate, the number usually describes the first contract while the reader hears the second.

The published failure data makes the split concrete. Omni Analytics analyzed 4,602 incorrect SQL queries and found that schema-level errors — wrong column selection, misinterpreted meaning — account for 81.2% of failures, while syntax errors account for only 18.8%. Roughly four out of five failures are not the model failing to write SQL. They are the model writing SQL that means something different from what the business meant.

For the platform leader, that reframes the entire evaluation. You are not buying a translator. You are buying an interpreter, and interpreters need to know the domain.

The core distinction: a valid query answers a question. A correct query answers your question. Execution success only proves the first.

What text-to-SQL failure modes actually look like — and why they hurt text-to-SQL accuracy

Failure taxonomies are usually written as symptoms. The more useful version maps each symptom to the missing business meaning underneath it.

The EntSQL benchmark, built to test grounding in long-context enterprise knowledge, published a distribution across 982 failed predictions: wrong filter at 54.6%, wrong scope at 14.4%, wrong aggregation at 10.0%, with the remaining 21.1% covering joins, calculations, classification, and missing metrics. Read that distribution carefully. The largest single failure class is not a broken query. It is a working query with the wrong constraint.

Beneath all of this sits a question of business meaning, not a question of SQL syntax. The model is filling a gap with the most statistically likely interpretation from its training data, and that interpretation is rarely how a specific business defines a specific term. Collate states the mechanism plainly: when meaning is missing, the model interpolates the hole with the most statistically likely interpretation available to it, not with the organization's actual convention.

The consequence is what Omni names metric drift — queries returning different numbers for the same metric, such as revenue or churn. Its description of how drift spreads is worth sitting with: every ad-hoc query is a fork, the metric shifts between runs, and the business user sees a number without seeing the SQL that produced it. The drift is invisible until someone reconciles two dashboards in a meeting.

Why these errors survive every review gate

This is the part that makes text-to-SQL qualitatively different from other reliability problems in a data platform.

There is no error to catch. A query with a missing exclusion clause returns a smaller number that looks entirely reasonable. A query aggregating at customer grain when the question implied order grain returns a defensible-looking average. Nothing in the pipeline objects, because nothing in the pipeline knows what the question meant.

The dbt Labs benchmark team framed this as the difference between silent and loud failure: text-to-SQL will cheerfully produce a wrong number, whereas a governed semantic layer "tells you it can't answer. It never returns invalid data." Their conclusion is the sentence to bring to a design review — with text-to-SQL, failure looks like a plausible but incorrect answer; with a semantic layer, failure looks like an error message. For a board deck, an auditor, or a KPI dashboard, that difference is operational risk.

Now compare the two failure signatures from an incident-response standpoint:

Silent failure is the expensive one. It is also the default.

What the evidence says about closing the text-to-SQL accuracy gap

The interesting question is not whether models are improving. They clearly are. The question is where the improvement lands.

Raw generation has improved substantially. In the same benchmark lineage, raw text-to-SQL moved from 32.7% at the GPT-4 era to 64.5% in 2026 across the full question set. That is real progress, and it is still roughly three in ten answers wrong on a small, well-documented schema.

Compare that to the same models routed through a governed semantic layer on the same questions. The April 2026 dbt Labs benchmark — 11 business questions against a 15-table insurance dataset, each question run 20 times, four configurations — is a well-documented reference point:

The benchmark is vendor-published, and that matters for how you read it — the methodology is disclosed, but the conclusion favors the vendor's product category. Treat it as one well-documented data point rather than a neutral verdict.

Independent results point the same direction. Snowflake's own BIRD-SQL testing showed accuracy moving from 57% to 78% on the identical LLM once a semantic model was supplied — a 21-point lift from context, not from a model swap. A peer-reviewed study found that adding a semantic-layer document lifted accuracy by +17.2 to +23.2 percentage points across every model tested, and also reduced hallucination rates.

Two details from the dbt benchmark are more instructive than the headline numbers. First, increasing reasoning effort changed latency, not accuracy — GPT models on the highest effort setting averaged over 20 seconds per query instead of 8, with no meaningful accuracy gain. Thinking harder does not supply missing business meaning. Second, when an LLM was asked to build the minimum set of models needed to answer every question, it created three — none hand-written — and the governed layer could then answer the full question set.

Three definitions. That is the size of the intervention relative to the size of the problem.

Where the gap is widest

Benchmark rankings are not the same as deployment outcomes, and the gap between them is measured in tens of points.

BEAVER, an enterprise benchmark built from three private data warehouses — 9,128 question-SQL pairs across 812 tables and 19 domains — reports that the strongest agentic method, ReFoRCE with Claude-4.5-Sonnet, reaches 11.4% accuracy. The same method scores 62.9% on Spider 2.0, a benchmark already considered substantially harder than its predecessors.

The reason is structural, and it has nothing to do with model quality. Enterprise warehouses average 101.5 tables and 869.4 columns per database, with cryptic column names and thin documentation. Join relationships are frequently implicit — there are no foreign key constraints to read. The data is private, so no public model has seen the domain's entity codes or conventions. And the queries are analytical: averaging 5.7 joins, 5.6 levels of nesting, and 3.7 CTEs.

A demo on a clean schema tells you almost nothing about what will happen on yours.

What to do instead of waiting for a better model

The practical answer is a sequence of gates between the generated query and the executed number. Treat generated SQL as untrusted input, and treat NL2SQL reliability as a property of the pipeline rather than a property of the model.

1. Parse gate. Reject anything that fails to parse, and reject disallowed statement types outright — no writes, no DDL, no CTAS on the analytical path.

2. Schema gate. Verify every referenced table, column, join key, and dialect-specific function actually exists before execution.

3. Semantic gate. Check the query against the governed definitions: which metric, which time window, which tenant scope, which exclusions. This is the gate that catches the 81.2%.

4. Access gate. Enforce row-level and column-level policy so generation cannot expose data the asker is not entitled to see.

5. Safety gate. Run EXPLAIN, cap cost, inject row limits, set timeouts, and block Cartesian joins and unbounded scans.

6. Execution gate. Execute under read-only, sandboxed credentials with full logging.

7. Result gate. Compare returned values against trusted references or known invariants and route anomalies to human review.

The gate sequence is documented production practice rather than a novel proposal; Promethium's evaluation guidance makes the same point that evaluation must test business correctness, not syntax, and AI Workflow Lab's production guide describes the cheap-to-expensive gate ordering applied above.

Gates three and four are the ones most teams skip, and they are the ones that require the semantic substrate. You cannot check a query against governed definitions if governed definitions do not exist anywhere in machine-readable form. The good news is that the substrate does not have to start large — the dbt result suggests three careful models can cover a meaningful question set.

This is the layer where a governed semantic view does its work: definitions, relationships, and metrics expressed once and exposed to every consumer — dashboards, analysts, and agents alike — instead of being re-derived inside each prompt. When natural-language interfaces sit on top of that layer, the practical pattern is mapping natural-language intent to governed metrics rather than generating SQL against raw tables. Teams that have consolidated fragmented engines into a single governed platform have made this transition as part of a production lakehouse migration rather than as a separate project.

Pro Tip: before adding a single validation gate, count how many of your organization's top 20 business terms have a written, machine-readable definition with an owner. That number predicts your text-to-SQL accuracy ceiling more reliably than any model benchmark.

Where this argument is wrong

An honest version of this position has to concede three things, and the published evidence supports all three concessions.

First, a governed layer cannot answer what has not been modeled. The dbt benchmark's own historical comparison shows the semantic layer scoring 0.0% on questions requiring too many entity hops, in both 2023 and 2026 — the required joins simply cannot be expressed without additional modeling. On those same questions, raw text-to-SQL reached 70.0% in 2026. Generation is more flexible than a governed layer, and there is a real category of question where generation wins.

Second, more context is not the same as curated meaning. EntSQL's results are a useful corrective to the "just add more context" instinct. Adding long-form domain documents lifted the best system from 6.8% to 15.9%. Expert-curated evidence snippets lifted it to 21.4% — better than full documents by 5.5 points. Human experts with the same evidence reached 84.0%. The bottleneck is knowledge selection and grounding, not context window size.

Third, some arguments go further than this piece does. One published position contends that semantic layers are a patch and ontologies are the structural fix; practitioner discussion in the field pushes back that genuinely ambiguous prompts cannot be resolved by context alone. Both are worth taking seriously. If your business vocabulary is contested enough that even humans disagree on what a term means, no layer will rescue the query — the ambiguity has to be resolved upstream, by people, before it becomes a definition.

So the accurate claim is narrower than "semantic layers solve this." It is this: for questions your organization has defined, a governed layer makes the answer correct and its absence loud. For questions it has not defined, you still need generation — and you still need a human or a workflow in the loop.

What to change on Monday

The uncomfortable implication of the failure data is that most text-to-SQL accuracy figures you will be shown describe a different problem than the one you have.

Four moves follow from that.

Stop accepting execution success as an accuracy metric. Report two numbers, separately: execution success rate, and business-correct answer rate against a golden set of your own questions with analyst-verified answers. The second number is the one that survives an audit.

Measure your ceiling before you scale. Take 30 real questions from your own workload, have analysts write the correct answers, run them 20 times each, and count how many were answered correctly and consistently. This finds your actual production accuracy rather than a vendor's benchmark accuracy. Then decide what to route where.

Route by question type. Governed metrics, recurring KPIs, anything customer-facing, and anything headed to a board deck should go through defined semantics. Exploratory, long-tail, and multi-hop analytical questions can go through generation with gates in front of them. This is a routing decision, not a religious one.

Give the model the definitions, not just the columns. The most reproducible finding in the published record is that the same model, on the same questions, performs dramatically better when business meaning is supplied explicitly. That is a data-modeling task, and it is where the compounding return sits.

The version of this that most teams arrive at is not a replacement decision. It is an ordering decision: generate for discovery, govern for the numbers that matter, and make sure the system says "I don't know" out loud instead of inventing a number that looks right.

Frequently asked questions

Does a higher text-to-SQL accuracy benchmark mean the problem is solved? Not necessarily. Benchmark accuracy and production accuracy measure different conditions. Enterprise warehouses average around 100 tables and 870 columns per database with implicit join relationships — a very different setting from a clean academic schema. Evaluate against your own schema and business questions.

Is text-to-SQL useless, then? No. It remains the right tool for exploratory questions, long-tail analysis, and queries a governed layer has not modeled — the dbt benchmark showed raw generation reaching 70.0% on multi-hop questions where the semantic layer scored 0.0%. It is unreliable as the authority for recurring, governed metrics.

How long does it take before accuracy improves? It depends on definition coverage, not implementation effort. The most instructive published result is small: an LLM built three models — none hand-written — and the governed layer could then answer an entire 11-question benchmark set. Start with your highest-stakes terms rather than your full vocabulary.

Do we have to abandon our current warehouse? No. A semantic layer sits above storage and expresses meaning; it does not require moving data. That matters if you are bound by compatibility requirements, existing pipelines, or a phased migration plan.

What is the single highest-value first step? Write down and version the definition of your five most-contested business terms, with an owner for each. This is cheap, it is unblocking, and it is the prerequisite for every gate described above.

Next steps

If your team is evaluating natural-language analytics, the first artifact worth requesting is not a demo — it is the definition coverage. Ask which metrics are governed, who owns each definition, and what happens when the system cannot answer. That conversation tells you more than any accuracy figure in a slide.

A useful reference to bring into that discussion is the platform architecture overview, which shows how definitions, storage, and consumers are separated — the structural precondition for everything described above.

Valid SQL was never the goal. A number your business can defend is. Singdata builds the governed layer that makes the difference visible — including when the honest answer is that the question has no approved definition yet.