arXiv：cs.AI· Raghu Venkat, Tricha Anjali·· 6 小时前
置信度评分能否判断 Text-to-SQL 答案对错？基于 BIRD 的探索性研究与 Spider 上的预注册验证
Does a Governed Confidence Score Know When a Text-to-SQL Answer Is Right? An Exploratory Study on BIRD and a Pre-registered Confirmation on Spider
AI 导读
一项研究审计了生产级对话分析系统为 Text-to-SQL 答案附带的置信度评分：在 BIRD dev 的 1534 道题上（由 GPT-5.1 作答、每题五次采样），该设计评分区分对错的能力较弱（AUROC 0.624），并扣留了 35.7% 的正确答案。
正文
Abstract:Conversational analytics lets people ask questions of records spread across many systems and get answers in plain language; it is useful only if each answer can be trusted and explained. A production system of this kind attaches a confidence score to every answer it writes over a database and uses it to show the answer, show it with a warning, or withhold it. The score's weights, caps and thresholds were set by design. This paper audits that design. On the 1,534 questions of BIRD dev, answered by GPT-5.1 with a benchmark-adapted production prompt and five samples per question, the designed score separates right from wrong answers weakly (AUROC 0.624, 95% CI 0.595-0.652) and withholds 35.7% of correct answers. Nearly all of its ranking signal comes from agreement among the samples, and most of it is lost where agreement is combined by a minimum with a checklist that drops to zero on any unsupported claim, including claims in commentary. We froze a rule in which agreement with the shown answer is the score and deterministic checks act as vetoes and caps, and pre-registered a test on Spider dev before any Spider answer existed. The primary hypothesis held: AUROC rose from 0.566 to 0.725 (paired difference +0.159, 95% CI +0.097 to +0.221), and correct answers withheld fell from 49.0% to 8.8% with PASS accuracy 95.7%. Under one matching policy the choice of agreement statistic matters little; the gain comes from how agreement enters the score. Whether added checks help depends on how correctness is defined, while they always cost unqualified coverage. A pre-registered empty-result check did not meet its effect-size criterion on Spider. Correctness here means the core answer; labels are model-judged, and a blinded two-person audit, not yet complete, will be reported in a revised version.
| Comments: | 23 pages, 4 figures, 9 tables. Code, frozen scorer, pre-registration and data: this https URL |
| Subjects: | Artificial Intelligence (cs.AI) |
| ACM classes: | H.2.3; I.2.7 |
| Cite as: | arXiv:2610.17256 [cs.AI] |
| (or arXiv:2610.17256v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.17256 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Raghu Venkat [view email]
[v1]
Mon, 12 Oct 2026 18:14:43 UTC (35 KB)
来源：arXiv：cs.AI · arxiv.org