arXiv：cs.LG· Yuxuan Wang, Guangxiang Zhao, Xinyu Tan, Yahui Han, Xinxiang Yin, Zhonghai Wu, Xiangzheng Zhang, Lin Sun, Tong Yang·· 4 小时前
SRP-SFT：面向监督微调的选择性边界保持方法
Beyond Distribution Matching: Selective Margin Preservation for Supervised Fine-Tuning
AI 导读
研究者提出 Selective Rank-Preserving SFT（SRP-SFT），一种软边界目标，对参考模型 top-K 非目标 token 对的参考缩放下界施加 hinge 约束，达标即停止，从而在不冻结参考分布的前提下保留选定的局部非目标边界关系。
正文
Abstract:Supervised fine-tuning (SFT) changes logit margins among non-target tokens at teacher-forced prefixes. Matching a local reference distribution keeps those probabilities and also penalizes margin expansions. We ask whether a soft auxiliary can preserve selected local non-target margin relations without freezing that distribution. We introduce \textbf{Selective Rank-Preserving SFT (SRP-SFT)}, a soft margin target: a hinge on reference-scaled lower bounds for pairs from the reference's top-$K$ non-target tokens, applied at fixed teacher-forced prefixes. The hinge stops once the requested bound is met. We derive the KL cost of violating a margin, characterize the feasible region and its conditional utility, and give a finite-step condition for reducing aggregate violations. In one-epoch experiments on Qwen3-4B-Base and UltraFeedback, this target raises MATH-500 and code-generation scores over CE and LP-SFT at all three training seeds. At seed 1234, MATH-500 pass@1 rises from 49.56\% to 54.56\% and the four-metric code mean rises from 65.91\% to 69.50\% relative to LP-SFT. Ranking controls, held-out cache diagnostics, sensitivity sweeps, and six additional backbones show where these score changes appear.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.19685 [cs.LG] |
| (or arXiv:2610.19685v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19685 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yuxuan Wang [view email]
[v1]
Wed, 14 Oct 2026 07:09:26 UTC (440 KB)
来源：arXiv：cs.LG · arxiv.org