arXiv：cs.AI· Lisong He, Yunxuan Jiang, Haishan Ye, Xiangyu Chang·· 20 小时前
ZOIF：仅用前向传播的零阶影响力函数估计，用于 LLM 数据选择
ZOIF: Zeroth-Order Influence Function Estimation for LLM Data Selection Using Only Forward Passes
AI 导读
研究者提出 ZOIF，一种完全通过前向传播完成影响力估计的零阶框架，用于 LLM 数据选择。ZOIF 将影响力估计拆解为三个无需反向传播的阶段：零阶估计代理梯度、把逆 Hessian 向量积重构成辅助零阶优化以规避显式求逆、再沿参数更新方向做前向有限差分得到单样本影响力分数。实验显示其选出的数据子集下游效用与基线相当甚至更优，峰值内存大幅降低，总计算时间相近。
正文
Abstract:Influence functions enable principled data selection for large language models (e.g., identifying influential training samples for model editing, debugging, or efficient fine-tuning). However, scaling IFs to LLMs is hindered by the intractability of billion-scale Hessian matrices and the prohibitive memory overhead of backpropagation (BP). While many classical IF approximations reduce Hessian complexity, they still rely on BP for gradient computation, requiring massive storage for intermediate activations that impedes practical data selection at scale. To address this, we propose ZOIF, a zeroth-order framework for influence estimation that operates entirely via forward passes, enabling memory-efficient data selection under resource constraints. ZOIF decomposes influence estimation into three BP-free stages: (1) approximating the average proxy gradient via zeroth-order estimation; (2) reformulating and solving the inverse Hessian-vector product as an auxiliary zeroth-order optimization to bypass explicit inversion; and (3) deriving per-sample influence scores via a forward finite difference along the previously computed parameter update direction. Experiments show that ZOIF selects data subsets with downstream utility competitive with, and in some cases exceeding, the evaluated baselines, yet at substantially lower peak memory cost and with comparable overall computation time.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.15336 [cs.LG] |
| (or arXiv:2610.15336v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.15336 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Lisong He [view email]
[v1]
Sun, 11 Oct 2026 04:25:40 UTC (249 KB)
来源：arXiv：cs.AI · arxiv.org