arXiv：cs.AI· Pengcheng Wang, Zhexuan Zhang, Chao Lou, Kewei Tu·· 20 小时前
句法语言模型大规模实证研究：标准因果注意力即可媲美结构掩码
A Scaled-Up Empirical Study of Syntactic Language Models
AI 导读
一项大规模实证研究对 12 种配置的句法语言模型（SLM）进行了测试，规模最高达 1B 参数和 100B 训练 token，覆盖 11 个下游任务和两个句法基准。结果显示，采用标准因果注意力的 SLM 在下游与句法基准上匹配或超越结构掩码变体，且无需架构修改并兼容 FlashAttention。但扩展到 1B 参数规模后，表现最佳的 SLM 变体相对算力匹配的非结构基线不再有明显优势。
正文
Abstract:Syntactic language models (SLMs) that generate sentences along with their syntactic tree structures have shown promise in both syntactic generalization and downstream tasks. Prior studies, however, have mostly evaluated smaller models trained on less pretraining data and tested on narrower task suites. In addition, the design space of tree linearization and attention masks remains underexplored. We conduct a large-scale empirical study of 12 controlled configurations---one terminal baseline and 11 SLM variants spanning different linearizations and masks---at scales up to 1B parameters and 100B training tokens, evaluated across 11 downstream tasks and two syntactic benchmarks. The experimental results show that SLMs with standard causal attention match or exceed structural-mask variants across downstream and syntactic benchmarks. They require no architectural modifications and remain compatible with FlashAttention. We also observe that when scaling up to the 1B-parameter scale, the top-performing SLM variant no longer shows a clear advantage over a compute-matched non-structural baseline, suggesting that SLMs may become less competitive with scale. Our code is available at this https URL.
| Comments: | Accepted to EMNLP 2026 |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.16837 [cs.CL] |
| (or arXiv:2610.16837v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.16837 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Pengcheng Wang [view email]
[v1]
Mon, 12 Oct 2026 14:16:58 UTC (44 KB)
来源：arXiv：cs.AI · arxiv.org