arXiv：cs.LG· Chantal Pellegrini, Adrian Delchev, Ege \"Ozsoy, Nassir Navab, Matthias Keicher·· 6 小时前AI 评分32
ProtoSR：面向细粒度结构化放射报告的原型知识引导方法
Prototype-Based Knowledge Guidance for Fine-Grained Structured Radiology Reporting
AI 导读
ProtoSR 通过指令微调 LLM 挖掘 80k+ MIMIC-CXR 研究，构建与结构化报告模板对齐的多模态知识库，用视觉原型表示每个答案选项。模型检索与当前图像-问题对相关的原型，并通过原型条件残差对预测进行选择性校正。在 Rad-ReStruct 基准上取得 SOTA，尤其在细粒度属性问题上提升最大。
正文
Abstract:Structured radiology reporting promises faster, more consistent communication than free text, but automation remains difficult as models must make many fine-grained, discrete decisions about rare findings and attributes from limited structured supervision. In contrast, free-text reports are produced at scale in routine care and implicitly encode fine-grained, image-linked information through detailed descriptions. To leverage this unstructured knowledge, we propose ProtoSR, an approach for injecting free-text information into structured report population. First, we introduce an automatic extraction pipeline that uses an instruction-tuned LLM to mine 80k+ MIMIC-CXR studies and build a multimodal knowledge base aligned with a structured reporting template, representing each answer option with a visual prototype. Using this knowledge base, ProtoSR is trained to retrieve prototypes relevant for the current image-question pair and augment the model predictions through a prototype-conditioned residual, providing a data-driven second opinion that selectively corrects predictions. On the Rad-ReStruct benchmark, ProtoSR achieves state-of-the-art results, with the largest improvements on detailed attribute questions, demonstrating the value of integrating free-text derived signal for fine-grained image understanding.
| Subjects: | Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG) |
| Cite as: | arXiv:2603.11938 [cs.AI] |
| (or arXiv:2603.11938v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2603.11938 arXiv-issued DOI via DataCite |
Submission history
From: Chantal Pellegrini [view email]
[v1]
Thu, 12 Mar 2026 13:51:13 UTC (250 KB)
[v2]
Wed, 7 Oct 2026 09:16:13 UTC (250 KB)
来源：arXiv：cs.LG · arxiv.org