arXiv：cs.CL· Aarav Singh, Animesh Pathak, Navyansh Singh·· 4 小时前AI 评分37
面向罕见病诊断的后见之明引导推理链蒸馏
Hindsight-Guided Rationale Distillation for Rare Disease Diagnosis
AI 导读
研究在 ZebraMap 上以 8B 教师模型的思维链蒸馏 1.5B 学生模型，教师生成时可看到真实诊断。过滤版学生 StudentF 准确率小幅但显著超过教师（p < 0.001），未过滤版则无显著优势（p = 0.129），说明增益来自污染过滤而非后见蒸馏本身。原因是教师会把"真实答案是 X"写入推理链并被 SFT 复制，未过滤学生在 33.9% 的案例中复现该表述，标签错误时准确率严重下降。
正文
Abstract:We study hindsight-guided distillation for rare disease diagnosis on ZebraMap: a 1.5B student is fine-tuned on chain-of-thought traces from a 8B teacher that observes the ground-truth diagnosis during generation. Absolute accuracy remains low for all models - the task is hard at this scale - but within this ceiling a filtered variant (StudentF) achieves a small, statistically significant accuracy advantage over the teacher (p < 0.001), concentrated in better-represented diseases. The unfiltered student does not significantly outperform the teacher (p = 0.129), establishing that contamination filtering - not hindsight distillation alone - drives the gain. The gap traces to an artifact we term GT hallucination. Label-visible generation causes the teacher to embed "ground truth is X" phrases in its reasoning chain; SFT copies the pattern. At inference, the unfiltered student reproduces the phrase in 33.9% of cases, with severe accuracy degradation when the hallucinated label is wrong. A regex filter removing these slots reduces contamination to near-zero, producing the observed gain - though the effect remains small. We precisely quantify this gain-cost tradeoff, document frequency-dependent knowledge transfer absent from the RL-trained teacher, and characterize a calibration gap that SFT does not close - identifying both as directions for future work.
| Comments: | 15 pages, 4 figures, Github: this https URL, Accepted at AACL-IJCNLP SRW 2026 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.03176 [cs.CL] |
| (or arXiv:2610.03176v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.03176 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Aarav Singh [view email]
[v1]
Fri, 2 Oct 2026 11:52:17 UTC (65 KB)
来源：arXiv：cs.CL · arxiv.org