arXiv：cs.CL· Kohei Kajikawa, Lin Ai, Tatsuki Kuribayashi, Ethan Gotlieb Wilcox·· 4 小时前
用 GPT-2 建模句子理解中的上下文与共指效应
Using LMs to Model the Effects of Context and Coreference during Sentence Comprehension
AI 导读
研究系统性调整 GPT-2 上下文窗口，在四个大规模英语阅读时间数据集上发现 U 型关系：限制上下文（< 20 tokens）能捕捉局部记忆限制，而扩展上下文（500–1,000 tokens）心理语言学拟合度最高。通过代词化重复篇章实体破坏跨句实体链的反事实实验，大上下文窗口的预测力下降 20%–40%，表明长程共指关系是 LM 惊奇度与人类阅读行为对齐的重要因素。
正文
Abstract:Language models (LMs) are often used as a tool to model human language processing. Recent studies suggest that severely restricting LMs' context window improves their fit to human psycholinguistic data by simulating human working memory constraints. However, it is possible that this strict memory-decay approach overlooks humans' reliance on long-range structural representations, such as discourse structre. In this work, we systematically vary the context window size of GPT-2 across four large-scale naturalistic English reading-time datasets and observe a U-shaped relationship: Although restricted contexts (< 20 tokens) successfully capture local memory limitations, expanded contexts (500--1,000 tokens) ultimately yield the highest overall psycholinguistic fit. To investigate the mechanism driving this benefit, we conduct a counterfactual inference-time experiment that disrupts cross-sentential entity chains by pronominalizing repeated discourse entities. Obscuring these structural linkages significantly degrades the predictive power of larger context windows by 20% to 40%. Our experiments demonstrate that tracking long-range coreference relations is one important factor for the alignment between LM surprisal and human reading behavior, and approximate the extent to which human comprehenders use global discourse relations during language processing.
| Comments: | EMNLP 2026 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2609.32119 [cs.CL] |
| (or arXiv:2609.32119v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2609.32119 arXiv-issued DOI via DataCite |
Submission history
From: Kohei Kajikawa [view email]
[v1]
Sat, 26 Sep 2026 00:43:56 UTC (580 KB)
[v2]
Thu, 8 Oct 2026 01:48:04 UTC (580 KB)
来源：arXiv：cs.CL · arxiv.org