arXiv：cs.LG（机器学习，全量分类）· Cristian McGee, El Houcine Bergou, Aritra Dutta·· 14 小时前AI 评分36
ZFO：面向 LLM 微调的零阶与一阶混合优化方法
Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning
AI 导读
研究者提出 ZFO 框架，将方向选择与步长解耦：由可信的一阶优化器确定方向，仅沿该一维子空间做零阶评估来选取曲率感知步长，成本低于完整线搜索。该方法在语言模型和数据集上常优于固定步长的一阶基线，论文已被 NeurIPS 2026 接收，代码已公开。
正文
Abstract:Step-size selection remains a central challenge in large-scale neural network optimization; conservative steps slow convergence, while aggressive steps can destabilize it. We combine \textbf{Z}ero-and-\textbf{F}irst-\textbf{O}rder optimization~(ZFO) and propose a lightweight framework that decouples direction selection from step-size. ZFO uses a trusted first-order optimizer to determine the direction and performs zeroth-order evaluations only along this one-dimensional subspace to choose how far to move. Using the current {gradient information} and two additional objective function evaluations, ZFO instances construct a local model of the objective function along the proposed direction and select a curvature-aware step within a bounded search interval. This yields an adaptive step-selection mechanism that costs less than a full line search. We provide theoretical guarantees to show that shared-sample evaluations produce reliable finite-difference curvature estimates, that the induced local model selects a near-optimal step along the search interval, and that ZFO converges to a neighborhood of a stationary point. Across the evaluated settings, language models and datasets, ZFO frequently improves optimization and final performance relative to fixed-step first-order baselines, with the magnitude and preferred local model depending on the objective. Our code is publicly available at: this https URL.
| Comments: | Accepted to 40th Conference on Neural Information Processing Systems (NeurIPS 2026). Code: this https URL |
| Subjects: | Machine Learning (cs.LG); Optimization and Control (math.OC) |
| ACM classes: | G.1.6; I.2.6; F.2.1 |
| Cite as: | arXiv:2610.02190 [cs.LG] |
| (or arXiv:2610.02190v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.02190 arXiv-issued DOI via DataCite (pending registration) |
|
| Journal reference: | 40th Conference on Neural Information Processing Systems (NeurIPS 2026) |
Submission history
From: Cristian McGee [view email]
[v1]
Thu, 1 Oct 2026 17:59:28 UTC (3,292 KB)
来源：arXiv：cs.LG（机器学习，全量分类） · arxiv.org