arXiv：cs.LG· Li Lixing·· 5 小时前AI 评分35
用交叉注意力层锚定系统提示词：CAL 插入位置研究
System-Prompt Anchoring with Cross-Attention Layers
AI 导读
研究将 Cross-Attention Layer（CAL）块插入系统提示词与文本之间、冻结因果解码器主干，在 1.5B 模型上做十种配置扫描，发现性能依赖任务且受插入位置强烈影响，靠后的位置通常更有效、更省参数。在 8B 规模研究中仅训练整体最佳配置并与参数量匹配的适配基线对比，交叉注意力改变了指令遵循与安全行为，同时大体保持通用任务性能。
正文
Abstract:Cross-attention provides a dedicated route from a selected information source into a model's computation, but the effect of where that route is inserted remains underexplored. We study this question when the source is a privileged system-prompt span. We insert Cross-Attention Layer (CAL) blocks between the system prompt and text while keeping the causal-decoder backbone frozen. A ten-configuration sweep on a 1.5B backbone shows that performance is task-dependent and strongly affected by placement: later placements are generally more effective and parameter-efficient. In an 8B scaling study, we train only the overall best configuration and compare it with parameter-matched adaptation baselines. Across the evaluated benchmarks, the effects remain task-dependent: cross-attention changes instruction-following and security behavior while largely preserving general-task performance. Together, these experiments characterize placement as an important design variable when injecting system-prompt information through cross-attention.
| Comments: | Preprint. 14 pages, 4 figures, 5 tables |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.09737 [cs.LG] |
| (or arXiv:2605.09737v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.09737 arXiv-issued DOI via DataCite |
Submission history
From: Lixing Li [view email]
[v1]
Sun, 10 May 2026 20:12:39 UTC (890 KB)
[v2]
Fri, 21 Aug 2026 22:14:49 UTC (1,169 KB)
[v3]
Fri, 2 Oct 2026 02:16:56 UTC (1,170 KB)
来源：arXiv：cs.LG · arxiv.org