arXiv：cs.LG· Haiyuan Wan, Yanyi Li, Weicheng Lin, Cong Fang·· 11 小时前
SAZO：样本自适应零阶后训练方法
Sample-Adaptive Zeroth-Order Post-Training
AI 导读
研究者提出样本自适应零阶优化方法 SAZO，通过在线学习样本重要性并自适应分配查询来加速大语言模型微调收敛。该方法从现有方向差分中提取梯度范数信息估计样本重要性，仅比 MeZO 多 O(N) 标量状态且无需额外模型评估。在四个模型族上的实验显示，相同训练损失下查询次数收敛速度最高提升 6 倍。
正文
Abstract:Zeroth-order (ZO) optimization offers a memory-efficient alternative to backpropagation for fine-tuning large language models. However, existing ZO methods typically sample training examples uniformly, overlooking their evolving importance and leaving avoidable estimation variance. To address this limitation, we propose Sample-Adaptive Zeroth-Order Optimization (SAZO), which learns sample importance online and adaptively allocates queries to accelerate convergence. SAZO extracts gradient-norm information from existing directional differences to estimate sample importance without access to true gradients. Furthermore, to track evolving importance from sparse minibatch feedback, SAZO maintains per-example exponential moving averages that aggregate observations across updates. The method requires no additional model evaluations and only $O(N)$ additional scalar state compared with MeZO. We prove that SAZO asymptotically matches the leading convergence term of the minimum-variance sampling oracle, and achieves a strictly faster convergence rate than standard methods. Extensive experiments across four model families demonstrate improved average downstream performance and up to $6\times$ faster convergence to the same training loss in terms of query count.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.12566 [cs.LG] |
| (or arXiv:2610.12566v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12566 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Haiyuan Wan [view email]
[v1]
Thu, 8 Oct 2026 18:00:37 UTC (305 KB)
来源：arXiv：cs.LG · arxiv.org