arXiv：cs.AI· Alexey Zemtsov, Albina Klepach, Vadim Lomshakov, Andrei Polubarov, Nikita Lyubaykin, Alexander Derevyagin, Vladislav Kurenkov, Alexander Nikulin·· 16 小时前
Looped Transformer 的逐块保角注入：Rotate the Loop 方法提升状态跟踪与语言建模
Rotate the Loop: Blockwise Conformality for Looped Transformers
AI 导读
Looped Transformer 将残差在递归步之间的线性映射视为几何问题，提出逐块保角注入（blockwise conformal injections）：残差拆分为坐标对齐的分组并在各组内旋转，使每组拥有自己的时间尺度且几何保持精确。
正文
Abstract:Looped transformers reuse one shared block across many recurrent steps, so depth becomes a quantity that can be spent at runtime rather than a stack of distinct layers. Those extra steps help only if a write made on one recurrent step is still readable on later ones, and that depends on the linear map applied to the residual between recurrent steps. We show that whether the loop learns to track state is a geometric question about the form of this map, and we propose blockwise conformal injections: the residual splits into coordinate-aligned groups and rotates inside each one, so every group has its own timescale while the geometry stays exact within it. Our experiments show a performance increase on a demanding form of state tracking - last-token accuracy when the rest of the prefix is already correct - and on language modelling, where a signal written earlier has to be retrieved at the end of the sequence.
| Comments: | Accepted at NeurIPS 2026 Workshop "Transitioning from Pre-Training to Post-Training" |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML) |
| Cite as: | arXiv:2610.14030 [cs.LG] |
| (or arXiv:2610.14030v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14030 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Alexey Zemtsov [view email]
[v1]
Wed, 30 Sep 2026 23:32:34 UTC (141 KB)
来源：arXiv：cs.AI · arxiv.org