arXiv：cs.CL· Oscar Chew, Po-Yi Lu, Yi-Jie Cheng, Kunal Jain, Hsuan-Tien Lin, Zhengzhong Tu, Kuan-Hao Huang·· 16 小时前
我们真的在测量遗忘吗？多模态 LLM 机器遗忘评估的陷阱与 GEMMU 基准
Are We Really Measuring Forgetting? Pitfalls in Evaluating Machine Unlearning for Multimodal LLMs
AI 导读
针对多模态大语言模型机器遗忘评估，研究指出三大陷阱：缺少留出评估导致无法区分真正的概念级遗忘与实例级过拟合；依赖合成微调知识而非预训练获得的真实知识；对不同遗忘方法套用相同训练设置造成不公平比较。为此作者提出基于真实知识与原则性评估协议的基准 GEMMU，重新评估代表性遗忘方法后揭示出显著的泛化差距。
正文
Abstract:As machine unlearning for multimodal large language models (MLLMs) rapidly advances, rigorous evaluation of its effectiveness becomes increasingly important. However, existing benchmarks largely inherit evaluation protocols from text-only unlearning and overlook the unique challenges of unlearning in multimodal settings. In this work, we identify three major pitfalls that may lead to misleading conclusions: the lack of heldout evaluation to distinguish genuine concept-wise forgetting from instance-wise overfitting, the reliance on synthetic, fine-tuned knowledge rather than real-world knowledge acquired during pre-training, and unfair method comparisons caused by applying the same training settings to different unlearning methods. To address these challenges, we introduce GEMMU (Generalizable Evaluation for Multimodal Machine Unlearning), a benchmark built on real-world knowledge and a principled evaluation protocol. Re-evaluating representative unlearning methods under GEMMU reveals substantial generalization gaps and challenges conclusions drawn from previous benchmarks. Our findings highlight the importance of rigorous evaluation for advancing multimodal machine unlearning and provide guidelines for future benchmark design.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.19613 [cs.CL] |
| (or arXiv:2610.19613v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19613 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Oscar Chew [view email]
[v1]
Wed, 14 Oct 2026 05:36:03 UTC (10,828 KB)
来源：arXiv：cs.CL · arxiv.org