arXiv：cs.LG· Haodong Liang, Yanhao Jin, Krishnakumar Balasubramanian, Lifeng Lai·· 3 小时前AI 评分38
SFT 与 RFT 在分类任务上的对比研究：学习风格，遗忘语义
Learning Style, Forgetting Semantics: A Case Study of SFT and RFT on Classification Tasks
AI 导读
一项 arXiv 研究用可解析的线性-softmax 策略，对分类任务中 SFT 与 RFT 的更新做了语义与风格分量的精确分解。在共同策略与提示词下，两者的语义更新平行，但风格动态不同：RFT 用精确策略梯度可保持类内风格对称，SFT 在非均匀教师下会产生偏离轴的风格漂移。
正文
Abstract:Why does supervised fine-tuning (SFT) lead to more forgetting than reinforcement fine-tuning (RFT), even when all teacher demonstrations are semantically correct? We study this question on classification tasks where tokens within each semantic class express the same semantic answer in different styles. The tasks share an underlying semantic rule but differ in their prompt distributions and teachers' stylistic preferences. Using a tractable linear-softmax policy, we derive an exact decomposition of the updates into semantic and style components. We show that, at a common policy and prompt, SFT and RFT have parallel semantic updates but differ in their style dynamics. Starting from a policy with no within-class style preference, RFT with exact policy gradients preserves this symmetry, whereas SFT with a nonuniform teacher develops off-axis style drift along a nonzero task mean under population updates. We use this drift to establish a separation under explicit conditions: for population updates from a common perfectly fitted checkpoint, SFT forgetting admits a strictly positive lower bound over a finite training interval, while RFT retains zero semantic error. Simulations over task sequences support these theoretical predictions.
| Comments: | 43 pages, 7 figures |
| Subjects: | Machine Learning (stat.ML); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.02437 [stat.ML] |
| (or arXiv:2610.02437v1 [stat.ML] for this version) | |
| https://doi.org/10.48550/arXiv.2610.02437 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Haodong Liang [view email]
[v1]
Thu, 1 Oct 2026 20:03:07 UTC (171 KB)
来源：arXiv：cs.LG · arxiv.org