arXiv：cs.LG· Jingwen Liu, Alexandr Andoni, Daniel Hsu·· 5 小时前AI 评分51
arXiv 论文提出 Fixed Universal Transformers：固定参数即可模拟任意 transformer
Fixed Universal Transformers
AI 导读
Jingwen Liu、Alexandr Andoni 与 Daniel Hsu 在 arXiv 论文（arXiv:2605.31423，NeurIPS 2026）中提出 universal transformers：通过合适的输入嵌入编码目标模型描述，参数完全固定的 transformer 可模拟给定类别中的任意 transformer，类比通用图灵机。
正文
Abstract:We introduce \emph{universal transformers}: fixed transformers that can simulate any transformer in a given class via a suitable input embedding. Analogous to a universal Turing machine, the input embedding encodes a description of the target model while all internal parameters remain fixed. We provide explicit sparse constructions achieving universality when the embedding dimension is sufficiently large, and further show that universality is generic: randomly initialized transformers are universal almost surely, which aligns with recent empirical results of Zhong and Andreas (2024). We empirically validate our theory on the algorithmic tasks of parenthesis balancing and multi-hop reasoning. Our results suggest that much of a transformer's expressive power may reside in its input representation rather than its learned weights.
| Comments: | NeurIPS 2026 |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.31423 [cs.LG] |
| (or arXiv:2605.31423v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.31423 arXiv-issued DOI via DataCite |
Submission history
From: Jingwen Liu [view email]
[v1]
Fri, 29 May 2026 15:22:06 UTC (179 KB)
[v2]
Thu, 1 Oct 2026 21:28:25 UTC (180 KB)
来源：arXiv：cs.LG · arxiv.org