arXiv：cs.LG· Xiaocong Yang, Yuren Hao, ChengXiang Zhai·· 6 小时前
DAGFormer：用全局拓扑预测规划 Transformer 跨层连接
DAGFormer: Global Topology Prediction for Transformer Cross-Layer Connections
AI 导读
DAGFormer 是一种用联合训练的拓扑预测器规划跨层连接的 Transformer 语言模型，可一次性规划整个模型的路由，而非逐层单独设置。
正文
Abstract:We propose DAGFormer, a Transformer language model whose cross-layer connections are planned by a jointly trained topology predictor. Beyond the residual connections in a standard Transformer, DAGFormer adds direct connections from earlier layers to later ones, and the predictor controls the strength of every connection based on input tokens. Unlike existing cross-layer routing architectures, which set each layer's connections separately, DAGFormer plans the routing of the whole model at once. On matched OLMo-2 pairs from 75M to 1B parameters, DAGFormer lowers WikiText-2 perplexity by 17% to 25% over the vanilla Transformer, and a 1B DAGFormer trained on 5B tokens outperforms the vanilla model trained on 10B. Against four routing baselines with the same data and budget, it reaches the lowest perplexity at 75M, 300M and 1B. The gain carries over to 13 commonsense, science, reading and math evaluation tasks (2.0 to 2.5 points higher mean accuracy at 300M and 1B) and to instruction finetuning (63% to 70% AlpacaEval win rate against the vanilla Transformer, leading every routing baseline), and under structured pruning DAGFormer degrades the least in five of six settings.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.19625 [cs.CL] |
| (or arXiv:2610.19625v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19625 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xiaocong Yang [view email]
[v1]
Wed, 14 Oct 2026 05:50:21 UTC (184 KB)
来源：arXiv：cs.LG · arxiv.org