arXiv：cs.CL· William Timkey, Tal Linzen·· 5 小时前
句法注意力头是否影响模型行为？arXiv 研究质疑其因果作用
Do Syntax Heads Influence Model Behavior?
AI 导读
Transformer 语言模型会发展出专门处理句法关系的注意力头，但 arXiv 新研究显示这些头对模型句法预测的因果作用证据"不一甚至偏负面"：即使强制这些头关注非主语的名词干扰项，模型的句法预测仍更符合主语。对专门句法头的干预在两个测量指标中仅一个显著强于随机头干预。
正文
Abstract:Transformer language models develop attention heads that specialize in syntactic relations, for example, attending from verbs to their subjects. Do these heads play a causal role in the model's syntactic predictions? Evidence for such a role is mixed to negative: even when we force these heads to attend to a distractor noun other than the subject, the model's syntactic predictions are still more consistent with the subject than with the distractor. Interventions on specialized syntax heads had a significantly stronger effect on syntactic predictions than interventions on random heads, but only in one of the two measures we propose.
| Comments: | 4 pages, 2 figures |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.19231 [cs.CL] |
| (or arXiv:2610.19231v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19231 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: William Timkey [view email]
[v1]
Tue, 13 Oct 2026 21:47:26 UTC (116 KB)
来源：arXiv：cs.CL · arxiv.org