Why Text-to-SQL Is the Wrong Layer: The Case for Text-to-Metric

Two dashboards. Two revenue numbers. Both trace back to the same warehouse, both marked "certified," and both defensible when someone asks how they were calculated. If you have sat in that meeting, the problem was not the SQL. It was that nobody had defined what "revenue" resolves to before a query was ever written.

That distinction is the whole argument. Text-to-SQL optimizes query generation: it turns a question into executable syntax. But the unsafe step in an analytics pipeline is metric resolution, the process of deciding which governed definition, grain, filters, and joins a business term maps to before any SQL runs. Generation is a translation problem. Resolution is a governance problem, and a better model does not solve it. The fix is architectural: move the resolution step out of the model and into a semantic layer that defines metrics once, so Text-to-Metric replaces Text-to-SQL as the layer that answers a business question.

Singdata is a Singapore-headquartered company that builds data agents on top of its cloud-agnostic, AI-native data infrastructure, and it applies those agents across data engineering and analytics processes. Its work on natural-language-driven AI agents and metric-oriented analytics is the basis for this article's argument about resolving questions against defined metrics rather than generating open-ended SQL.

The Conventional View: Better Models Will Fix Text-to-SQL

The mainstream position is straightforward: natural-language analytics is a capability problem, and each model generation narrows the gap. On the BIRD leaderboard's current top score, DataGallery-Text2SQL from Huawei 2012 Labs reached 82.39% Test EX as of 7 September 2026, against a human baseline of 92.96%. That is roughly ten points of headroom, and the trend line has been climbing steadily. If the curve continues, the argument goes, the remaining gap closes on its own.

The Spider benchmark tells a similar story in reverse. The Spider 1.0-to-2.0 drop runs from roughly 91% on the original benchmark to 21.3% on Spider 2.0, where the best o1-preview-based code-agent framework scored 21.3% execution accuracy and the strongest text-to-SQL parser managed just 5.7% on Spider 2.0-Lite. Treat this as a single-source comparison rather than a settled consensus, but the direction is hard to dismiss: harder, more realistic tasks expose how much of the earlier progress depended on clean schemas.

The view is popular because it is measurable. What BIRD's execution accuracy actually measures is result equivalence on one database instance, not semantic equivalence. A query that returns the right rows on the test instance passes, even if it would diverge on a different instance with different grain or null handling. Benchmarks reward that narrower standard because it is reproducible, and reproducible scores are what drive leaderboard progress.

Why Execution Accuracy Hides the Failure That Matters

A syntax error ends in a message you cannot miss. A semantic error ends in a number, and that is why a semantic error ends in a number, not an error message rather than a stack trace: the query runs, the result looks reasonable, and someone puts it in a board deck. Ask "what was our revenue in June" in a single warehouse and you can get three defensible figures, invoices issued, orders fulfilled, and revenue recognized, each correct from some department's perspective. The model picks the first plausible field and reports it without qualification.

The failure classes behind that are all about meaning, not syntax. Grain inflation and other silent failure classes include ambiguous joins across two foreign-key paths, NULL handling where COUNT(col) and COUNT(*) diverge, time-window inference that reads "last quarter" as a calendar quarter rather than a trailing 90 days, and categorical resolution that fills a string matching zero rows. Join customer grain to invoice-line grain and the query still runs; the number is wrong by a factor of seven. This is metric drift: the same metric name returning different values depending on who asked and how.

Benchmarks do not surface it. Where BIRD scores collapse on hard questions shows the top leaderboard model at roughly 79% on simple queries but about 64% on challenging ones, with multi-hop reasoning and deep joins as the primary failure points.

The Step Text-to-SQL Leaves Undefined

Text-to-SQL leaves one step undefined: metric resolution, the act of binding an ambiguous question to a single governed definition before any SQL exists. Generation is the visible work; resolution is the work that decides whether the generated query is the right one.

The resolution pipeline runs as an operational sequence, and each stage removes a specific failure mode. Intent classification separates a lookup from a comparison from a trend, removing the failure where a "why did revenue drop" question gets answered with a row count. Metric candidate matching finds the governed definitions that could satisfy the intent, removing the failure where a plausible column name stands in for a business concept. Disambiguation resolves the remaining candidates against grain, filters, and time window, removing the failure where two teams' revenue definitions silently diverge. Definition binding locks the chosen metric to its versioned definition, removing the failure where a calculation drifts between runs. Query assembly emits the SQL last, once the target is fixed.

dbt Labs' own framing of MetricFlow states the requirement plainly: metrics "should not be probabilistic or depend on an LLM guessing each calculation. They should be deterministic." How MetricFlow resolves a metric request shows the mechanism, with metrics and semantic models declared in YAML and a semantic graph that computes the best join path before constructing the query.

That ambiguity is a first-class obstacle rather than a prompting problem is the premise of the ambiguity taxonomy proposed in AmbiSQL, which catalogs ambiguity across both database elements and LLM reasoning, and ships 40 ambiguous queries drawn from two real-world benchmarks.

What the Data Actually Shows About Text-to-Metric

The evidence does not support waiting for a better model. In a 2026 arXiv preprint on what predicts correctness in text-to-SQL, one reflection round moved execution accuracy only from 0.451 to 0.468, while the model assigned confidence of at least 0.9 to 99% of its revised queries, with mean confidence 0.97, AUROC 0.637, and expected calibration error 0.501 (What Predicts Correctness in Text-to-SQL?, arXiv 2607.06799, 2026). Self-correction without calibrated abstention does not close the gap, which means abstention has to be a separate mechanism rather than a byproduct of scale.

A small, non-peer-reviewed trial points the same way. On its compute-tail query set, the weakest model tier failed loudly on 1.00 of trials and silently on 0.00, while the strongest tier was silent on 0.96 (Security Data Works, 2026-06-17). Capability can make a wrong answer quieter, not rarer.

That is the case for Text-to-Metric. Resolution turns an open generation problem into a bounded selection problem over governed metrics, so the failure mode changes shape instead of shrinking.

How to Apply Text-to-Metric: Acceptance Criteria and Scope

Start by inventorying the metrics that actually drive decisions, not every column in the warehouse. A catalog of forty certified metrics that finance, operations, and product all recognize beats four hundred that nobody owns.

1. Define each metric in the semantic layer. Record grain, filters, time logic, exclusions, and a named owner. dbt Labs explains how MetricFlow resolves a metric request, including how definitions are versioned and who can access them.

2. Certify join paths. Ambiguous relationships between tables are where silent errors enter. Approve one path per question type.

3. Add a disambiguation step that asks rather than guesses. When intent is unclear, the system should request clarification instead of picking a plausible interpretation.

4. Gate on lineage and auditability. Every answer should trace back to a governed definition, with the version and owner visible.

Singdata's Data Analytics Agent supports this pattern: it maps intent to centrally defined metrics instead of inferring from raw schema, so business users query governed metrics without touching tables directly.

Pro Tip: Acceptance checklist

• Every returned number traces to a certified metric with a named owner

• Ambiguous questions trigger a clarifying prompt, not a guess

• Lineage and metric version are visible on the answer

• Long-tail exploratory questions stay on text-to-SQL by design

Track two numbers: the share of natural-language answers traceable to a governed definition, and the rate of silent semantic corrections, where someone quietly fixes a number after the fact. Expect the catalog to cover decision-critical metrics within weeks; the long tail is deliberately out of scope.

The Strongest Counterargument: Better Models Will Close the Gap

The objection deserves a straight answer: if frontier models keep improving, metric resolution is scaffolding you can remove in a year or two. That argument assumes the failures are capability failures. The evidence points elsewhere. Security Data Works makes the case that open-ended text-to-SQL is a verification-and-security problem, not a capability problem: why a stronger model can make the wrong answer quieter is that the more capable model did not get the answer right, it got the query right enough to execute, which removed the only thing that had been catching the error. Weaker models were accidentally safe because a strict engine refused their broken queries. The same source found that a deterministic OBDA/OWL2QL arm declines loudly (1.00 loud, zero silent) where every LLM tier was silently wrong.

The measurement story has the same shape. What BIRD's execution accuracy actually measures is narrower than it sounds: a 2025 study recap reports that execution accuracy agrees with human experts only 62% of the time, with roughly 32% of training problems carrying annotation errors. A benchmark that disagrees with experts a third of the time cannot certify that a better model fixed the problem.

The honest limitation: catalog coverage gaps and long-tail exploratory questions still belong on text-to-SQL, and some answers will keep requiring human review. That is bounded scope, not a retreat from the thesis.

Frequently Asked Questions

Doesn't text-to-SQL work fine on small, well-documented schemas?

Often it does, and that is the honest scope limit. The problem is what the benchmark measures. BIRD's execution accuracy compares a generated query against a reference query on one database instance, so a system can score well while returning a result that answers a different question than the one asked. Small schemas reduce the search space; they do not remove the ambiguity.

We have already invested in a text-to-SQL deployment. Is that wasted?

No. Keep it for exploratory, long-tail questions where no governed metric exists. Add metric resolution in front of it for the recurring business questions, so those resolve against centrally defined metrics rather than a fresh interpretation of the schema each time.

What happens when no governed metric covers my question?

That question stays on text-to-SQL, and it should. The failure mode to watch for is the opposite case: a question that sounds covered but is not. Grain inflation and other silent failure classes describe queries that run cleanly and return a number at the wrong aggregation level, which is why coverage gaps need to be visible rather than inferred.

Who owns a metric when two teams disagree on its definition?

Ownership is a governance decision, not a modeling one. The ambiguity taxonomy proposed in AmbiSQL separates ambiguity that comes from the question from ambiguity that comes from the schema, and the second kind is resolved by naming one definition as canonical and versioning it, with a named owner accountable for changes.

Conclusion

The unsafe step was never generation. It was resolution: the point where a question becomes a specific, governed metric with a definition, an owner, and a version. Text-to-SQL optimizes the layer above that gap and inherits every ambiguity underneath it, which is why two correct queries can still return two defensible numbers.

Treating metric definitions as prompt context keeps that gap open. Treating them as infrastructure closes it. Governed metrics belong in a versioned semantic layer with named owners, review gates, and an audit trail that records which definition answered which question, under whose access rights, on what date. That is what makes metric resolution reviewable rather than merely plausible, and it is the part of the stack an auditor can actually inspect.

The shift is organizational as much as technical. Someone has to own each definition, and someone has to approve changes to it. Teams that assign that ownership early spend less time reconciling dashboards later.

If you want to see the mechanism rather than the argument, see how metric resolution works or read the architecture note behind it.