arXiv：cs.AI· Tiago Martinelli, Jo\~ao Paulo Papa, Adriano Jos\'e Pereira·· 1 天前
蒸馏前先审核：将推理能力迁移到紧凑医疗语言模型的受控工作流
Audit Before You Distill: A Controlled Workflow for Reasoning Transfer into Compact Medical Language Models
AI 导读
一项针对巴西葡萄牙语住院医师考试题的研究发现，教师模型的完整推理链在教师答案正确时可将学生模型选对答案的概率提升 21 个百分点，教师答错时则降低 14 个百分点，打乱推理内容后这一模式消失。
正文
Abstract:Teacher rationales can help a smaller model answer health professions questions, but they can also reinforce errors. We examined when they help and how this evidence can guide training using Brazilian Portuguese questions from medical and multiprofessional residency examinations. First, we measured how a teacher rationale changed the student's probability of selecting the correct answer and compared this effect with rationales drawn from other questions. The complete rationale increased this probability by 21~p.p. when the teacher was correct and decreased it by 14~p.p. when the teacher was wrong. By intervening on the reasoning content, we showed that shuffled rationales did not produce the same pattern. Guided by this result, we trained only with rationales attached to correct teacher answers and compared answer-only training, step-by-step distillation, and our proposed Distill-SFT method. Distill-SFT showed a modest advantage over step-by-step distillation on previously missed questions under reasoning, while using fewer training instances by combining the answer and rationale into a single target. We also investigated catastrophic forgetting using the out-of-domain ENEM dataset. Answer-only training reduced reasoning accuracy by 12~p.p. relative to baseline, compared with approximately 4~p.p. for either rationale-based training method. Together, these findings support a workflow that audits rationale content before distillation, verifies teacher answers before supervision, and trains compact models efficiently for health professions question answering.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.16713 [cs.CL] |
| (or arXiv:2610.16713v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.16713 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tiago Martinelli [view email]
[v1]
Mon, 12 Oct 2026 12:47:07 UTC (97 KB)
来源：arXiv：cs.AI · arxiv.org