arXiv：cs.AI· Chun-Hao Yang, Bo-Han Feng, Tzu-Yuan Lai, Yan-Yu Chen, Yin-Kai Dean Huang, Shou-De Lin·· 22 小时前
训练 LLM 超越下一 token 预测：填补互信息鸿沟
Training LLMs Beyond Next Token Prediction - Filling the Mutual Information Gap
AI 导读
论文提出 Max(MI(S;t)) 方法，用基于互信息的确定性启发式重新组织目标 token 的训练顺序，优先预测信息量大的 token。在算术推理、多标签文本分类和文本生成任务上，GPT 风格模型及预训练 Llama/Qwen 变体使用该方法后常优于标准下一 token 预测及 reverse order、TF-IDF、条件概率选择等基线，且不改变模型架构与自回归解码。
正文
Abstract:Autoregressive language models are usually trained with next-token prediction under a fixed left-to-right target order. This paper studies target-token organization as a training-time design choice across arithmetic reasoning, multi-label text classification, and text generation. Controlled order studies show that the default NTP order can be far from the best-performing tested organization, while fixed hand-crafted orders such as reverse order do not generalize across tasks. To address this, Max(MI(S;t)) is introduced as a deterministic mutual-information-based heuristic that prioritizes information-rich target tokens. For fixed-length outputs, it constructs task-specific prediction orders; for open-vocabulary text generation, the same principle is applied through lightweight prefix augmentation. Experiments with GPT-style models and pretrained Llama/Qwen variants show that Max(MI(S;t)) often improves over standard NTP and simple baselines such as reverse order, TF-IDF, and conditional-probability selection, without changing model architecture or autoregressive decoding.
| Comments: | AACL-IJCNLP 2026 |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2511.00198 [cs.CL] |
| (or arXiv:2511.00198v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2511.00198 arXiv-issued DOI via DataCite |
Submission history
From: Chun-Hao Yang [view email]
[v1]
Fri, 31 Oct 2025 18:59:29 UTC (189 KB)
[v2]
Tue, 13 Oct 2026 06:48:26 UTC (415 KB)
来源：arXiv：cs.AI · arxiv.org