多目标对齐中的跨目标干扰：LLM 标量化训练的失败模式与 COVER 方法

View PDF HTML (experimental)

Abstract:We study a persistent failure mode in multi-objective alignment for large language models (LLMs), in which scalarized training improves only some objectives while the others degrade. We formalize this phenomenon as cross-objective interference and, to our knowledge, conduct the first systematic study of scalarization algorithms for multi-objective LLM alignment. The study shows that interference is pervasive across algorithms yet strongly model-dependent. To understand how interference arises, we derive a local covariance law stating that an objective improves or degrades at first order according to the sign of the covariance between its reward and the scalarized score. We extend this law to the clipped surrogate objectives of modern reinforcement fine-tuning and show that it still holds under mild conditions. Building on this law, we propose COVariance-floor Enforced Reweighting (COVER), a one-sided controller that raises an objective's weight only when the covariance between its reward and the clipped advantage weight falls below a target. Through extensive experiments, we find that COVER can mitigate cross-objective interference while matching linear scalarization when objectives already co-improve. Finally, to explain why interference is model-dependent, we complement the local covariance law with a global convergence analysis. This analysis gives sufficient conditions for the non-convex scalarized objective to satisfy the Polyak--Łojasiewicz condition and relates interference to model geometry.

Subjects:	Computation and Language (cs.CL); Machine Learning (cs.LG)
Cite as:	arXiv:2602.06869 [cs.CL]
	(or arXiv:2602.06869v3 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2602.06869 arXiv-issued DOI via DataCite

Submission history

From: Yining Lu [view email]
[v1] Fri, 6 Feb 2026 16:55:27 UTC (402 KB)
[v2] Wed, 6 May 2026 17:20:23 UTC (525 KB)
[v3] Tue, 6 Oct 2026 05:01:41 UTC (643 KB)