arXiv：cs.AI· Peizhi Niu, Evelyn Ma, Boya Hou, Huiting Zhou, Duo Zhou, Huan Zhang, S. Rasoul Etesami, Olgica Milenkovic·· 17 小时前
GUARD：基于数据归因的大语言模型引导式遗忘与保留框架
GUARD: Guided Unlearning and Retention via Data Attribution for Large Language Models
AI 导读
研究者提出 GUARD 框架，通过轻量级代理数据归因指标量化遗忘集与保留集之间的“对齐”程度，并据此为样本分配随归因分数指数衰减的自适应非均匀权重，以缓解 LLM 遗忘中的意外遗忘问题。
正文
Abstract:Unlearning large language models (LLMs) is important due to copyright and privacy concerns. A key challenge is unintended forgetting, where removing data inadvertently impairs model utility. While prior work has focused on architectural innovations, the influence of data-level factors remains underexplored. We introduce GUARD, a framework for **G**uided **U**nlearning **A**nd **R**etention via **D**ata Attribution. GUARD uses a lightweight proxy data attribution metric tailored to LLM unlearning, which efficiently quantifies the "alignment" between the Forget and Retain (or synthetic supplement) Sets. This metric leads to a new unlearning objective that assigns adaptive, nonuniform weights to samples, decaying exponentially with their proxy attribution scores, and thereby provably mitigates unintended retention loss. On the TOFU benchmark with a 10% forget set, applying GUARD to Gradient Ascent (GA) improves Retain Set ROUGE-L from 0.18 to 22.40 (a 124x increase) and Retain Probability from 0.06% to 17.50%, a substantial recovery from the GA collapse. Furthermore, GUARD combined with NPO (Negative Preference Optimization) outperforms Representation Misdirection for Unlearning (RMU) with 33.80% and 53.10% reductions in the Retain Set Sacrifice Rates computed from ROUGE-L and Probability ($\rho_r$ and $\rho_p$), and a 16.01% improvement in knowledge retention on the MUSE NEWS Retain Set. These findings persist when attribution scores are computed using synthetic or highly subsampled (20%) Retain Sets, where NPO+GUARD still reduces $\rho_r$ and $\rho_p$ by up to 43.89% and 48.13% over NPO. Similar improvements are observed on MUSE, with a tolerable absolute privacy leakage degradation of 13.7 compared to the best-performing existing method in the privacy category.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2506.10946 [cs.LG] |
| (or arXiv:2506.10946v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2506.10946 arXiv-issued DOI via DataCite |
Submission history
From: Peizhi Niu [view email]
[v1]
Thu, 12 Jun 2025 17:49:09 UTC (429 KB)
[v2]
Wed, 22 Oct 2025 02:25:01 UTC (539 KB)
[v3]
Tue, 13 Oct 2026 03:11:51 UTC (245 KB)
来源：arXiv：cs.AI · arxiv.org