arXiv：cs.AI· Yukun Huang, Sanxing Chen, Jian Pei, Manzil Zaheer, Bhuwan Dhingra·· 5 小时前
Cite Pretrain：让大语言模型无需检索即可完成知识引用
Cite Pretrain: Retrieval-Free Knowledge Attribution for Large Language Models
AI 导读
研究提出 CitePretrainBench，评估大语言模型能否不依赖推理时检索、直接从（持续）预训练中引用文档，涵盖 Wikipedia、Common Crawl、arXiv 及未见文档。
正文
Abstract:Large language models can answer questions correctly and still hallucinate their citations. Most citation systems address this by retrieving documents at inference time. Retrieval adds latency and depends on external infrastructure. Retrieval noise can also affect the answer. We study whether LLMs can cite documents from their (continual) pretraining without test-time retrieval. We introduce CitePretrainBench to evaluate this setting. It includes Wikipedia, Common Crawl, and arXiv as real-world corpora and also includes novel, unseen documents. The benchmark includes short-form citation tasks for a single fact and long-form tasks for multiple facts. Continual-pretraining first binds factual knowledge to a persistent identifier for each document. Instruction tuning then teaches the model to put these identifiers in its answers as citations. Passive Indexing appends an identifier at the end of each document. Synthetic data used in Active Indexing restate facts in diverse, compositional forms. We train two mappings together. These mappings are source$\to$fact and fact$\to$source. The first direction involves the model generating content given a cited source. In the second, it learns to attribute its own answers when facts are paraphrased or combined. On Qwen-2.5-7B and 3B, Active Indexing improves citation precision over Passive Indexing on every evaluated citation task, with gains of up to 30.2 percentage points. In the scaling ablation, citation performance continues to improve as we add augmented data, including at 16 times the original token count. Combining internal and external citations also improves robustness to retrieval noise.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2506.17585 [cs.AI] |
| (or arXiv:2506.17585v4 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2506.17585 arXiv-issued DOI via DataCite |
Submission history
From: Yukun Huang [view email]
[v1]
Sat, 21 Jun 2025 04:48:05 UTC (614 KB)
[v2]
Tue, 28 Oct 2025 18:06:24 UTC (683 KB)
[v3]
Fri, 3 Apr 2026 18:53:25 UTC (900 KB)
[v4]
Sun, 11 Oct 2026 21:46:50 UTC (893 KB)
来源：arXiv：cs.AI · arxiv.org