arXiv：cs.CL· Huan Li, Zhe Cao, Qinlei Xie, Fushun Cui, Xuechen Liang·· 6 小时前AI 评分35
条件锚定蒸馏如何稳定语言模型的持续学习
Stabilizing language models under continual learning via condition-anchored distillation
AI 导读
研究者提出条件锚定生成蒸馏（CAGD），通过保留少量旧提示词、用冻结的上一版模型重建补全与生成状态，并在学习新任务时匹配其预测分布，从而稳定语言模型的持续学习。
正文
Abstract:Continual adaptation of language models can change their output distribution on prompts learned earlier, while retaining every old prompt-answer pair may be undesirable or impossible. We study condition-anchored generative distillation (CAGD): retain a small set of old prompts, use a frozen previous model to reconstruct completions and generation states, and match its predictive distributions while learning the next task. The formulation separates three roles that ordinary replay conflates: conditions select the behavior to protect, teacher generations locate relevant states, and soft targets specify how predictions may change. For autoregressive language generation, teacher-rollout distillation admits an exact chain-rule decomposition of sequence divergence. For masked-diffusion language modeling, our implementation directly controls local denoising drift on teacher-generated completions. In continual adaptation of a 219M masked diffusion language model, CAGD reduces four-task final held-out loss from 2.927 to 1.114 in one task order and from 2.168 to 0.891 in exact reverse. The same soft targets lower final average loss by 0.055 over hard replay when teacher-generated support is held identical. The direction persists on fresh facts and natural instructions across SMDM and Qwen3. On GSM8K, Qwen adaptation preserves answer-format compliance, but exact-match retention is seed-mixed at 0.6B and worsens at 1.7B. These results support condition-anchored functional preservation as a common design principle across the tested language-generation objectives.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.06940 [cs.CL] |
| (or arXiv:2610.06940v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.06940 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhe Cao [view email]
[v1]
Sat, 3 Oct 2026 07:22:11 UTC (5,116 KB)
来源：arXiv：cs.CL · arxiv.org