arXiv：cs.CL· Yunhao Wang, Feng Zhang, Tinghao Yu, Xing Wu, Jiang Zhou·· 16 小时前
AFCS：面向事实知识预训练的原子事实条件合成流水线
AFCS: Atomic-Fact-Conditioned Synthesis for Factual Knowledge Pretraining
AI 导读
AFCS 是一种先抽取源文档中受支持的原子事实、再以每条事实驱动生成的知识语料合成流水线，采用结构感知抽取、源依据验证、局部上下文扩展和多格式生成。
正文
Abstract:Synthetic data is increasingly used in language-model pretraining. Existing knowledge-corpus synthesis pipelines, including WRAP and Nemotron-CC, usually take a source passage or document segment as input and generate rewritten or QA-style samples from that text. This can improve surface form and prompt diversity, but it does not explicitly require fine-grained source facts -- especially secondary entities, list/table entries, and precise temporal attributes -- to appear in the synthesized corpus. We propose AFCS (Atomic-Fact-Conditioned Synthesis), a pipeline that first extracts source-supported atomic facts and then uses each retained fact to drive generation. AFCS uses structure-aware extraction over the document tree, source-grounded verification, local context expansion, and multi-format generation. In continued-pretraining experiments with OLMo-3-7B and a 32B base on the same FineWiki source corpus, AFCS outperforms WRAP and Nemotron-CC on SimpleQA, SimpleQA Verified, PopQA, and HotpotQA. Granularity ablations show that fact-level synthesis improves over chunk-level, paragraph-level, and entity-level synthesis; and a 100B-token mid-training integration improves factual benchmarks and raises the 12-task general benchmark average by +1.09.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.20086 [cs.CL] |
| (or arXiv:2610.20086v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20086 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yunhao Wang [view email]
[v1]
Wed, 14 Oct 2026 12:24:40 UTC (780 KB)
来源：arXiv：cs.CL · arxiv.org