text to sql

Text-to-SQL's nightmare isn't syntax errors, it's the query that runs, returns a confident number, and is silently wrong. The defense is a semantic layer plus transparency UX. Here's the full design.

A query that errors is a bug. A query that runs and returns a plausible wrong number is a product teaching a finance team, one figure at a time, not to trust it. The diagnosis is a classification exercise, and the fix is almost never a better prompt.

Three correct queries scored 0 of 3 on exact match and 3 of 3 on execution. The golden set is question, reference SQL and result set, stratified by the ways queries go wrong, with abstention as its own column. A hundred cases gives plus or minus eight points; four hundred gives four.