arXiv：cs.LG· Darshil Doshi, Wenjie Zhou, Corinna Elena Wegner, Daniel J. Korchinski, Santiago Acevedo, Matthieu Wyart·· 4 小时前AI 评分43
语言模型中的柏拉图式表示理论：多语言模型跨语言相似性为何在中层达到峰值
A theory of platonic representations in language models
AI 导读
该研究为多语言语言模型中层跨语言表示相似性现象提供了理论解释。作者用共享上层、不共享下层产生式规则的概率上下文无关文法生成合成语言，证明贝叶斯最优下一 token 预测器为信念传播（BP），将其消息编码进逐层结构所得预测与同数据训练的 Transformer 高度吻合。
正文
Abstract:Representations of translated sentences are similar in the inner layers of multilingual language models -- an observation connected to the platonic representation hypothesis, yet unexplained theoretically. We provide an explanation based on the assumption that data have a hidden hierarchical structure whose abstract levels are shared across languages while surface levels are modality- or language-specific. Concretely, we generate synthetic languages from probabilistic context-free grammars sharing upper-level but not lower-level production rules. In this setting the Bayes-optimal next-token predictor is belief propagation (BP); encoding its messages in successive layers yields analytical predictions that agree well with transformers trained on the same data. The framework explains why cross-lingual similarity peaks in middle layers, coexists with language-specific structure, and strengthens with language proximity, model quality and data exposure. It distinguishes similarity (shared neighborhood geometry) from alignment (shared coordinates), showing that the latter occurs when code-switched data, i.e. mixed-language sentences, are abundant enough. It further predicts that subtracting from each layer the component linearly predictable from the preceding one increases cross-lingual similarity, which we confirm in pretrained LLMs.
| Comments: | 10+14 pages, 7+12 figures |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL); Machine Learning (stat.ML) |
| Cite as: | arXiv:2610.07168 [cs.LG] |
| (or arXiv:2610.07168v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.07168 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Darshil Doshi [view email]
[v1]
Mon, 5 Oct 2026 18:00:09 UTC (2,029 KB)
来源：arXiv：cs.LG · arxiv.org