GAC：面向混合 SFT-RL 后训练的噪声感知自适应混合方法

View PDF HTML (experimental)

Abstract:Hybrid post-training combining supervised fine-tuning (SFT) and reinforcement learning (RL) is the standard paradigm for aligning large language models, yet fixed mixing schedules cannot adapt when the relative noise of the two signals evolves. We derive a noise-aware mixing weight mu* by minimizing an MSE upper bound on the mixed stochastic gradient, yielding a closed-form that balances gradient noise variance and SFT-RL disagreement. Building on the token-wise reweighting of CHORD, the practical Guided Adaptive Controller (GAC) adds EMA smoothing, a schedule prior, and capped updates around this estimator, with all statistics estimated online from existing training tensors. The noise-aware controller alone outperforms the best rule-based controller by +3.0 percentage points on AMC; the full system reaches +3.8 points over HPT across math, code, science, and logic benchmarks, while reducing KL-drift area by 28% and large absolute mixing-weight changes by more than 70%, at less than 1% overhead. Gains grow with model size from 1.5B to 14B. Resources: Qwen3.5-4B compact-backbone release: this https URL reference implementation: this https URL GAC project page: this https URL

Comments:	17 pages, 3 figures, 22 tables. Camera-ready update adds links to the released Qwen3.5-4B compact-backbone checkpoint, reference implementation, and project page; experimental results and conclusions are unchanged
Subjects:	Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2605.26184 [cs.LG]
	(or arXiv:2605.26184v2 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2605.26184 arXiv-issued DOI via DataCite

Submission history

From: Yuelin Hu [view email]
[v1] Mon, 25 May 2026 07:52:29 UTC (1,327 KB)
[v2] Sat, 10 Oct 2026 16:23:42 UTC (1,341 KB)