arXiv：cs.CL· Liang-Hsun Huang, Min-Yi Chen, Chao-Chun Chuang, Yu-Wei Liu, Che-Yu Wu, Po-Feng Lee·· 16 小时前
OpenTWBench：面向大语言模型的台湾领域知识评测基准，以及测量装置如何影响分数
OpenTWBench (Open Taiwan Benchmark): Taiwan-Domain Knowledge Evaluation for Large Language Models, and How the Measurement Apparatus Shapes the Score
AI 导读
研究者发布 OpenTWBench 开放评测套件，含 255,854 道台湾考试四选一试题（148 个学科，来自考试院十五年开放数据）与 1,696 道台湾日常知识题 tw-formosa-bench。
正文
Abstract:We present OpenTWBench, an open evaluation suite for Taiwan-domain knowledge in large language models, built around two benchmarks. tw-exam-bench contains 255,854 four-option questions across 148 academic subjects, from fifteen years of Taiwan's national examinations published as open data by the Ministry of Examination; answer keys agree across 31,312 duplicated question groups at 99.91%, with all 27 disagreements accounted for. A reviewed second pass withdrew 2,180 items whose stems depend on a figure the text does not carry; all scores use the remaining 253,674. tw-formosa-bench contains 1,696 difficulty-curated questions about everyday and contemporary Taiwan - the Taiwan that licensing examinations never ask about - generated from a community-curated corpus under a discipline in which no model writes a ground truth, and curated against a frozen seven-model pool so that the pool scored at the passing margin (46.9-74.6%, median ~54%); a leave-one-out experiment shows the selection preserves model ranking (near-zero bias). Within six weeks the frontier had cleared that band: of 108 models now on the public board, 22 score above its ceiling, the best 91.6%.
We also document how much of a score comes from the apparatus rather than the model. On identical questions, switching answer extraction from pattern matching to an explicit \boxed{} answer moves accuracy from 20.8% to 93.9%; a single placeholder character in a system prompt moved one model from last (58.2%) to first (83.9%) on the same benchmark; doubling the thinking budget from 8k to 16k tokens bought 0.2% while shuffling which questions failed; and a three-judge validation panel proved to be one judge in disguise, two members approving 92-99% of everything shown. We release all datasets, per-item difficulty data, curation manifests and a leaderboard at this https URL
| Comments: | 19 pages, no figures. Datasets, per-item difficulty data and a public leaderboard at this https URL |
| Subjects: | Computation and Language (cs.CL) |
| ACM classes: | I.2.7; I.2.6 |
| Cite as: | arXiv:2610.19848 [cs.CL] |
| (or arXiv:2610.19848v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19848 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Liang Hsun Huang [view email]
[v1]
Wed, 14 Oct 2026 09:18:47 UTC (30 KB)
来源：arXiv：cs.CL · arxiv.org