arXiv：cs.LG· Elad Hoffer, Yochai Blau, Edan Kinderman, Ron Banner, Daniel Soudry, Boris Ginsburg·· 2 天前AI 评分42
INTRA：基于注意力的模型自带检索能力，无需外挂检索器
Retrieval from Within: An Intrinsic Capability of Attention-Based Models
AI 导读
研究者提出 INTRA（INTrinsic Retrieval via Attention）框架，让注意力编码器-解码器直接从自身内部表示中检索，用解码器注意力对预编码证据块打分并复用为生成上下文，从而统一检索与生成、消除 RAG 中检索器与生成器的错配，并通过复用预计算编码状态摊销上下文编码开销。
正文
Abstract:Retrieval-augmented generation (RAG) typically treats retrieval and generation as separate systems. We ask whether an attention-based encoder-decoder can instead retrieve directly from its own internal representations. We introduce INTRA (INTrinsic Retrieval via Attention), a framework where decoder attention queries score pre-encoded evidence chunks that are then directly reused as context for generation. By construction, INTRA unifies retrieval and generation, eliminating the retriever-generator mismatch typical of RAG pipelines. This design also amortizes context encoding by reusing precomputed encoder states across queries. On question-answering benchmarks, INTRA outperforms strong engineered retrieval pipelines on both evidence recall and end-to-end answer quality. Our results demonstrate that attention-based models already possess a retrieval mechanism that can be elicited, rather than added as an external module.
| Comments: | Accepted to NeurIPS 2026 |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.05806 [cs.LG] |
| (or arXiv:2605.05806v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.05806 arXiv-issued DOI via DataCite |
Submission history
From: Elad Hoffer [view email]
[v1]
Thu, 7 May 2026 07:42:28 UTC (1,727 KB)
[v2]
Fri, 8 May 2026 05:21:54 UTC (1,727 KB)
[v3]
Wed, 30 Sep 2026 23:13:47 UTC (865 KB)
来源：arXiv：cs.LG · arxiv.org