Blog · AI and self-service analytics
What text-to-SQL is, how accurate it is on the Spider and BIRD benchmarks, and a worked query on two tables, revenue by region for customers with more than 10 orders, with the check that proves it. Then the same question going wrong three ways, the alternative of mapping questions to tested calculations, and when text-to-SQL is the right tool.
Text-to-SQL turns a question in plain English into a SQL query, runs it against a database and returns the result. A language model writes the query from the question and the table schema; the database does the arithmetic. It works well on clean, well-named tables and goes wrong on ambiguous words, joins that duplicate rows, and business definitions the schema does not hold, so someone has to read the query.
Text-to-SQL, also called NL2SQL or natural language to SQL, is one way to build a natural language query: a question typed in ordinary words that software answers from a database.
Question + table schema → generated SQL → database runs it → result
The language model sees the question and a description of the tables (names, columns, sometimes sample rows). It writes a query. The database executes it and returns rows. The figures themselves are computed by the database, which is the right place for them: why the AI must never do the arithmetic explains why. The risk sits one step earlier, in whether the query asks the database the right question.
Two academic benchmarks dominate:
Two cautions. Scores change month to month, so read the leaderboard, not an article. And a benchmark query is either right or wrong; on your own schema, a wrong query usually still runs and returns a believable number. Test on your own tables, with questions whose answers you already know.
The question: "What was revenue by region last quarter for customers with more than 10 orders?" Two tables:
invoices(invoice_id, customer_id, invoice_date, amount), one row per invoicecustomers(customer_id, region)"Last quarter" is April 1 to June 30, 2026. The generated query, using a common table expression (the WITH clause) to total each customer first:
WITH q AS (
SELECT customer_id, COUNT(*) AS orders, SUM(amount) AS revenue
FROM invoices
WHERE invoice_date >= '2026-04-01' AND invoice_date < '2026-07-01'
GROUP BY customer_id
)
SELECT c.region, SUM(q.revenue) AS revenue
FROM q
JOIN customers c ON c.customer_id = q.customer_id
WHERE q.orders > 10
GROUP BY c.region;
The quarter's data by customer:
| Customer | Region | Invoices | Revenue (USD) | More than 10? |
|---|---|---|---|---|
| C01 | Northeast | 14 | 48,200 | Yes |
| C02 | Northeast | 9 | 22,500 | No |
| C03 | South | 22 | 91,400 | Yes |
| C04 | South | 11 | 30,100 | Yes |
| C05 | West | 12 | 40,700 | Yes |
| C06 | West | 6 | 12,900 | No |
The result:
| Region | Revenue (USD) |
|---|---|
| Northeast | 48,200 |
| South | 121,500 |
| West | 40,700 |
| Total | 210,400 |
South is C03 plus C04: 91,400 + 30,100 = 121,500.
Filtered total + excluded total = unfiltered total
The excluded customers, C02 and C06, had 22,500 + 12,900 = $35,400. 210,400 + 35,400 = $245,800, which must equal the quarter's total revenue from a query with no customer filter at all. Then count: four customers passed the filter, two did not, six invoiced in the quarter. If either check fails, the query dropped or duplicated something.
The query above is correct. Three small changes make it wrong, and each still runs without an error.
A fan-out join. Suppose the schema also has invoice_lines, and the generated query joins invoices to their lines before summing invoices.amount. Each invoice amount is now repeated once per line. If C03's 22 invoices have 3 lines each, C03 shows 274,200 instead of 91,400, South shows 304,300 and the total 393,200. The test before any sum of a header-level amount:
-- after the join, these two must be equal
SELECT COUNT(*), COUNT(DISTINCT i.invoice_id)
FROM invoices i JOIN invoice_lines l ON l.invoice_id = i.invoice_id;
"Orders" read as invoice lines. Count lines instead of invoices and C02's 9 invoices, at 2 lines each, become 18 "orders": C02 crosses the threshold and $22,500 moves into the result.
"Last quarter" read as the calendar quarter. If the fiscal year starts in February, last quarter is May through July, not April through June. The schema does not know the fiscal calendar; see fiscal calendars and period cuts.
The alternative is to stop generating queries at all. Each business question is mapped to one of a fixed set of tested calculations, each with an agreed definition of "order," "revenue" and "quarter" kept in versioned metric definitions. The language model's job shrinks to choosing the calculation and its options; the calculation itself never changes between questions. That is the architecture described in deterministic first.
The trade-off is real. Free SQL can answer a question nobody anticipated; a fixed set of calculations cannot, and should say so. In return, the same question on the same data gives the same answer every time, and no answer depends on a join someone forgot to check.
It is the wrong tool for figures that go to a board or a customer without anyone reading the query.
amt2 and flag_x give the language model nothing to go on.Covirage answers questions without free SQL. In its workspace the external AI model is given the question and the list of tools that workspace holds, with the options each accepts, and returns only which tool to run and with which of those options; it does not write a query. The tool computes from your uploaded data, every figure cites the rows behind it, and the external AI model then explains the tool's result. If no tool fits, the answer says so, and if the external AI model cannot be reached, a keyword match picks the tool and the figures are shown without commentary. See AI analytics, and AI data analyst for the same approach on sales data. For the queries analysts write by hand, see SQL for data analysis and SQL PIVOT; for what AI should and should not do with numbers, AI data analysis.
On academic benchmarks such as Spider, recent systems score highly; on harder benchmarks with realistic, messy databases such as BIRD, accuracy is noticeably lower. Scores change quickly, so check the current leaderboards and test on your own schema before trusting it.
A question typed in ordinary language, such as 'sales by region last month', that software translates into a database query or a defined calculation. Text-to-SQL is one way to do it; mapping the question to pre-built, tested measures is another.
Not to get an answer, but you need someone who can read SQL to trust it. Without that, you cannot tell whether the query filtered, joined and grouped the way you meant.
Only with read-only credentials, row limits, query timeouts and access limited to the tables needed. Generated queries can be expensive or touch data the user should not see, so use a reporting replica or a governed semantic layer.