arXiv：cs.LG· Zinan Ling, Yang Xiao, Ruimeng Ye, Sijia Liu, Bo Hui·· 3 小时前
RA²：Retain-Anchored Attraction 在 LLM 遗忘中保住遗忘邻近效用
RA$^2$: Retain-Anchored Attraction Preserves Forget-Adjacent Utility in LLM Unlearning
AI 导读
针对 LLM 遗忘中标准评估遗漏的"遗忘邻近效用损失"——模型在标准 retain 提示上表现正常、但在靠近被遗忘领域的良性推理上退化，作者提出隐空间遗忘方法 RA²（Retain-Anchored Attraction），把 forget token 状态重定向到邻近的 retain 支持表征上，同时保留更高层在 retain 数据上的行为。
正文
Abstract:Large Language Model (LLM) unlearning is often evaluated primarily by target suppression and clean retain utility. These metrics can miss a local failure mode: a model may perform well on standard retain prompts, yet degrade on benign reasoning when prompts lie near the forgotten domain. This phenomenon is termed forget-adjacent utility loss. A fundamental question is: where should forget representations go after they leave the removed behavior? Successful unlearning depends not only on whether forget-conditioned representations depart from the harmful computation, but also on the direction in which they are redirected afterward. We propose RA$^2$ (Retain-Anchored Attraction), a latent-space unlearning method that redirects forget-token states toward nearby retain-supported representations while preserving higher-layer behavior on retain data. On WMDP and MUSE, RA$^2$ yields higher forget-adjacent utility than strong unlearning baselines at similar broad-utility levels. These results show that the migration direction of forget representations materially affects benign behavior near the forget boundary. Our code is available at this https URL.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.19385 [cs.LG] |
| (or arXiv:2610.19385v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19385 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zinan Ling [view email]
[v1]
Wed, 14 Oct 2026 01:31:40 UTC (451 KB)
来源：arXiv：cs.LG · arxiv.org