arXiv：cs.LG· Hwiyeong Lee, Hyelim Lim, Ingyu Bang, Hoki Kim, Taeuk Kim·· 4 小时前AI 评分39
学习如何决定遗忘：记忆-泛化谱系下的机器遗忘研究
How Learning Governs Unlearning across the Memorization-Generalization Spectrum
AI 导读
研究揭示模型的学习方式会决定其后续遗忘效果：通过 modular addition 中的 grokking 区分记忆型与泛化型模型后，发现泛化型模型在遗忘时 retain set 性能下降更大。作者进一步提出 bucketed modular addition 精细控制两种策略贡献，证实该趋势在整个记忆-泛化谱系上近乎单调成立，并在 LLM 的逐字与事实回忆遗忘场景中同样得到验证。
正文
Abstract:While unlearning seeks to negate undesired capabilities acquired through learning, little research has examined how the way models learn shapes their subsequent unlearning. In this paper, we investigate this connection from the perspectives of memorization and generalization, the two most representative yet competing strategies that models employ during training. We first classify memorization- and generalization-heavy models using grokking in modular addition and compare their responses to unlearning, showing that the latter suffer greater retain damage, i.e., a larger performance drop on the retain set. Furthermore, we conduct a finer-grained analysis by introducing bucketed modular addition, in which the respective contributions of the two strategies can be explicitly controlled across the memorization-generalization spectrum. In this setup, we reaffirm that the same trend persists and is nearly monotonic. We further demonstrate that this relationship also holds in LLM unlearning across verbatim and factual recall settings. Finally, we provide two practical insights for developing better unlearning methods, highlighting the importance of accounting for learning dynamics in unlearning.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.08577 [cs.LG] |
| (or arXiv:2610.08577v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.08577 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hwiyeong Lee [view email]
[v1]
Tue, 6 Oct 2026 15:50:51 UTC (744 KB)
来源：arXiv：cs.LG · arxiv.org