arXiv：cs.AI· Le Qiu, Zhuan Anne Zhang, Hok Ming Adeline Chan, Emmanuele Chersoni·· 22 小时前
香港建筑业消防安全规范解读基准测试：12 款 LLM 表现评估
Towards a Benchmark for Fire Safety Code Interpretation in the Hong Kong AEC Industry
AI 导读
研究团队与领域专家合作构建了面向香港消防安全规范的本地化基准数据集，包含多选题与开放性问题，用于评估 12 款主流闭源与开源 LLM。表现最优的闭源模型在多选题上达到 80-90% 准确率，开源模型普遍落后；但开放性问题经专家人工评判后准确率仅为 50-70%，表明 LLM 在专业合规任务上仍有较大提升空间。
正文
Abstract:Large Language Models (LLMs) are increasingly explored in the Architecture, Engineering, and Construction (AEC) domain, yet their reliability in regulatory compliance remains uncertain. In this study, we evaluate the performance of LLMs on Hong Kong fire safety codes. In collaboration with domain experts, we create a localized benchmark dataset comprising both multiple-choice questions (MCQs) and open-ended questions (OEQs) to evaluate 12 widely used proprietary and open-source models. Our findings show that top-performing proprietary models achieve 80-90% accuracy on MCQs, whereas open-source models generally lag behind. However, on open-ended evaluation, expert human assessments reveal an accuracy range of only 50-70%, demonstrating that LLMs still have substantial room for improvement in specialized compliance tasks.
| Subjects: | Computational Engineering, Finance, and Science (cs.CE); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.16127 [cs.CE] |
| (or arXiv:2610.16127v1 [cs.CE] for this version) | |
| https://doi.org/10.48550/arXiv.2610.16127 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Le Qiu [view email]
[v1]
Mon, 12 Oct 2026 03:45:26 UTC (237 KB)
来源：arXiv：cs.AI · arxiv.org