arXiv：cs.AI· Abdul Basit, Muhammad Abdullah Hanif, Muhammad Shafique·· 19 小时前
MS-Exam-Gen：面向多发性硬化MRI知识评测LLM的自动基准构建工具包
MS-Exam-Gen: An Automatic Benchmark Construction Toolkit for Evaluating LLMs on Multiple Sclerosis-MRI Imaging Knowledge
AI 导读
MS-Exam-Gen 是一个可复现框架，用于构建和审计评估 LLM 多发性硬化MRI（MS-MRI）文本知识的多选题基准，不涉及直接MRI图像解读。
正文
Abstract:Biomedical large language model (LLM) evaluation requires auditable assessment of narrow, evolving, source-grounded subspecialty knowledge. Multiple sclerosis MRI (MS-MRI) provides a high-stakes textual-knowledge test case because correct reasoning requires current diagnostic criteria, standardized acquisition and reporting knowledge, longitudinal monitoring concepts, lesion morphology, and recognition of difficult mimics. We present MS-Exam-Gen, a reproducible framework for constructing and auditing a text-based multiple-choice question (MCQ) benchmark for MS-MRI knowledge; it does not evaluate direct MRI image interpretation. MS-Exam-Gen targets source-grounded criteria, protocols, reporting, and differential diagnosis. The framework combines expert-source indexing, exam-oriented topic induction, evidence-grounded MCQ generation, automated quality audits, a same-family consistency screen, and empirical calibration. From a 66-source corpus indexed into 4,289 retrieval chunks, the pipeline produced a locked 3,058-item candidate benchmark spanning 16 topics and 53 subtopics. Evaluation across 12 primary LLM endpoints yielded 36,696 item-level predictions and separated performance over a 42.8-percentage-point accuracy range (89.7% to 46.9%). Across these endpoints, 25.5% of items were missed by at least four. Post-generation audits showed that refreshed construction reduced measurable answer cues, while option-order testing showed that absolute MCQ scores remain position-sensitive. Generated construction labels remain metadata rather than validated psychometric categories. Because expert adjudication and full option-order counterbalancing remain future work, MS-Exam-Gen is not a clinically certified examination. It should be interpreted as an automatically filtered, source-grounded candidate benchmark and reproducible audit workflow for item-level and topic-specific LLM evaluation.
| Comments: | 7 pages, 3 figures. Accepted for publication to BHI 2026 |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| MSC classes: | 68T01, 68T50 |
| ACM classes: | I.2.7; H.3.3 |
| Cite as: | arXiv:2610.06170 [cs.AI] |
| (or arXiv:2610.06170v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.06170 arXiv-issued DOI via DataCite |
Submission history
From: Abdul Basit [view email]
[v1]
Mon, 5 Oct 2026 11:41:57 UTC (240 KB)
[v2]
Sat, 10 Oct 2026 12:55:19 UTC (240 KB)
来源：arXiv：cs.AI · arxiv.org