arXiv：cs.CL· David Yiyang Liu, Aditya Joshi, Paul Dawson, Andrew Piper·· 11 小时前
语言模型具有可解释的叙事机制：GPT-2-Small、Gemma-2-2B 与 Llama-3.1-8B 中的叙事组件
Language Models Have Interpretable Narrative Mechanisms
AI 导读
研究在 GPT-2-Small、Gemma-2-2B 和 Llama-3.1-8B 中识别出激活与叙事性相关的注意力头、MLP 神经元和稀疏自编码器特征。消融这些神经元和特征集后，高叙事性文本的下一 token 预测损失增幅大于低叙事性文本，开放式生成的叙事性也随之下降。下游影响最大的是故事补全和叙事问答，但在虚构与幻觉基准上的结果喜忧参半。
正文
Abstract:We investigate whether language models contain internal components that contribute to representing narrativity, the degree to which a text exhibits narrative properties. Across GPT-2-Small, Gemma-2-2B, and Llama-3.1-8B, we identify attention heads, MLP neurons, and sparse autoencoder features whose activations correlate with narrativity. Ablating selected neuron and feature sets increases next-token prediction loss more for high-narrativity than for low-narrativity texts, and reduces the narrativity of open-ended generations. On downstream benchmarks, the largest effects occur in story completion and narrative question answering, although they vary across models and languages; results on confabulation and hallucination benchmarks are mixed. These findings provide causal evidence that selected sets of narrative-associated components contribute to model behaviour, while their internal organisation, relation to narrative theory, and broader functions remain open questions.
| Comments: | 24 pages, 5 figures, 16 tables |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.12989 [cs.CL] |
| (or arXiv:2610.12989v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12989 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: David Liu [view email]
[v1]
Fri, 9 Oct 2026 04:45:52 UTC (524 KB)
来源：arXiv：cs.CL · arxiv.org