HuggingFace Daily Papers·· 2 天前
用单一算法训练基础模型：统一训练框架将预训练、SFT 与 RL 合并
Training Foundation Models with a Single Algorithm
AI 导读
研究者提出“统一训练”框架，通过将 RL 目标推导为对模型生成目标分布的最大似然拟合，把预训练、midtraining、SFT 和 RL 全部纳入同一个基于似然的训练算法。该框架在单一训练器中混合静态数据与 RL 生成，在受控和真实实验中改善了精度-算力权衡与任务覆盖，同时保留更广泛能力，并为持续学习提供更好的起点。论文共 67 页、31 图、22 表。
正文 · 原文
Abstract:Foundation models are typically trained in stages: pretraining, midtraining, supervised fine-tuning (SFT), and reinforcement learning (RL), with changes in both data and learning algorithms. We show that this algorithmic separation is not fundamental. By deriving suitable RL objectives as maximum-likelihood fitting to model-generated target distributions, we place all of these stages under a single likelihood-based training algorithm, unified training. This perspective turns the foundation-model training lifecycle into a problem of choosing which data to train on, in what proportions, and when, with conventional staged training as a special case. We instantiate this framework in a unified training system that mixes static data and RL generations within a single trainer. Across controlled and real-world experiments, unified training improves the accuracy-compute tradeoff and task coverage while preserving broader capabilities, and provides a stronger starting point for continual learning. This work lays the groundwork for a unified science of foundation-model training, connecting mathematical principles, training infrastructure, and the development of model capabilities.
| Comments: | 67 pages, 31 figures, 22 tables |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.40564 [cs.LG] |
| (or arXiv:2610.40564v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.40564 arXiv-issued DOI via DataCite |
Submission history
From: Yuda Song [view email]
[v1]
Thu, 29 Oct 2026 17:42:05 UTC (836 KB)
来源：HuggingFace Daily Papers · arxiv.org