Anthropic：Transformer Circuits（可解释性研究）·· 12 小时前AI 评分50
Anthropic 研究探究 Transformer 残差流中的特权基现象，初步指向 Adam 优化器
Privileged Bases in the Transformer Residual Stream Our mathematical theories of the Transformer architecture suggest that individual coordinates in the residual stream should have no special significance, but recent work has shown that this observation is false in practice. We investigate this phenomenon and provisionally conclude that the per-dimension normalizers in the Adam optimizer are to blame for the effect.
AI 导读
Anthropic Transformer Circuits 团队研究 Transformer 残差流中为什么会出现与标准基对齐的极端离群值（单个坐标可达其他坐标 20 倍以上），理论上残差流各坐标本不应有特殊意义。
来源：Anthropic：Transformer Circuits（可解释性研究） · transformer-circuits.pub