arXiv：cs.LG· Anat Kleiman, Gintare Karolina Dziugaite, Jonathan Frankle, Sham Kakade, Mansheej Paul·· 3 小时前
无需回放或辅助目标：SFA 与 Lookahead 缓解持续学习灾难性遗忘
Forget About It: Mitigating Forgetting Without Replay or Auxiliary Objectives
AI 导读
Qwen3-14B 适配器持续学习实验显示，周期性参数插值方法 SFA 与免回放的 Lookahead 在平均最终准确率上超过所测试的回放与自适应合并配置，任务训练时间最多减少 36.3%。在语言和视觉任务上，这两种固定配置的方法结果可与更复杂的持续学习方法相比，表明训练中简单插值在引入更复杂的遗忘缓解机制前值得考虑。
正文
Abstract:Have we made mitigating catastrophic forgetting more complicated than it needs to be? Continual adaptation often degrades performance on earlier tasks, motivating rehearsal buffers, auxiliary objectives, and adaptive merging procedures. We investigate how far a simpler alternative-periodic parameter interpolation during fine-tuning-can go on its own. We introduce Sequential Fine-tuning with Averaging (SFA) and evaluate it alongside replay-free Lookahead, originally designed for optimization stability to find compelling results. In Qwen3-14B adapter continual learning, fixed configurations of both methods outperform the tested replay and adaptive-merging configurations in mean final accuracy using up to 36.3% less task training time. Furthermore, across language and vision tasks, our results compete with more complex continual-learning methods. Together, these findings suggest that simple in-training interpolation deserves consideration before introducing more elaborate forgetting-mitigation mechanisms.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2501.05559 [cs.LG] |
| (or arXiv:2501.05559v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2501.05559 arXiv-issued DOI via DataCite |
Submission history
From: Anat Kleiman [view email]
[v1]
Thu, 9 Jan 2025 20:11:08 UTC (3,166 KB)
[v2]
Wed, 14 Oct 2026 01:34:00 UTC (861 KB)
来源：arXiv：cs.LG · arxiv.org