arXiv：cs.LG· Amartya Roy, Sayar Karmakar·· 4 小时前AI 评分41
Transformer 如何用线性注意力执行因果结构学习
Executing Causal Structure Learning with Linear-Attention Transformers
AI 导读
研究者构造了一个固定权重 Transformer，其前向传播可精确复现连续因果发现方法的一次图更新，用堆叠模块重现整个优化轨迹，并需在更新间保留算法乘子才能精确执行。实验显示该模块与参考更新在浮点精度内一致，在合成数据及七个公开基准网络拓扑上复现了参考求解器的成功与失败，而普通注意力模型在相同训练预算下无法可靠执行该更新或迁移到更大图。
正文
Abstract:Transformers can execute algorithms on data given in their input. We ask whether they can do the same for causal discovery. We study a standard continuous method that repeatedly updates a candidate causal graph while enforcing acyclicity. We explicitly construct a fixed-weight transformer whose forward pass exactly reproduces one update of this method, so repeated blocks reproduce its optimization trajectory. The transformer carries the current graph and the algorithm's multiplier between updates. We show that retaining the multiplier is essential for exact execution, since different multiplier values can lead to different next updates. We also give conditions under which, within a fixed stage, the number of updates needed to reach a target accuracy can be computed in advance and rounding errors stay bounded as depth grows. Experiments show that the constructed block agrees with a reference update to floating-point precision, while arithmetic replay on synthetic data and seven published benchmark network topologies inherits the reference solver's successes and failures. This separates accurate algorithm execution from accurate causal recovery. In contrast, the ordinary attention models tested under our training budgets do not reliably execute the update or transfer to larger graphs. Whether gradient training can learn an executor in the architecture class of the construction remains open.
| Comments: | 32 pages, 8 Figures |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.10395 [cs.LG] |
| (or arXiv:2610.10395v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.10395 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Amartya Roy [view email]
[v1]
Wed, 7 Oct 2026 16:48:51 UTC (390 KB)
来源：arXiv：cs.LG · arxiv.org