OctoLong：用跨仓库代码上下文做中期训练提升长上下文建模

View PDF HTML (experimental)

Abstract:Context lengths of language models (LMs) have dramatically increased, driven by the demands for in-context learning, self-improvement, and long-horizon agentic workflows. However, existing long-context corpora are dominated by books, academic articles, and code repositories, which are finite and often lack long-distance dependencies. In this work, we introduce OctoLong, a context-engineering pipeline that instruments an AST parser, a language server backend, and a package manager to recursively retrieve code references, enabling the curation of dependency-rich code contexts up to millions of tokens long. We then train OctoLong-Instruct, a suite of capable long-context open LMs, derived from base models ranging in size from 600M to 14B parameters, via context-extension mid-training on a approx 50B-token mixture containing approx 6.2B tokens of OctoLong code contexts, followed by approx 10B tokens of instruction tuning. Our ablations and experimental evaluations against 18 strong long-context open-weight LMs show that supplanting just 12% of conventional context-extension corpora with OctoLong data yields substantial gains in long-range retrieval, long-term state tracking, repository-level code understanding, and downstream agentic tasks, while also enhancing API usage in regular coding scenarios.

Subjects:	Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Software Engineering (cs.SE)
Cite as:	arXiv:2608.05141 [cs.AI]
	(or arXiv:2608.05141v2 [cs.AI] for this version)
	https://doi.org/10.48550/arXiv.2608.05141 arXiv-issued DOI via DataCite

Submission history

From: Indraneil Paul Mr. [view email]
[v1] Wed, 5 Aug 2026 17:58:15 UTC (640 KB)
[v2] Tue, 13 Oct 2026 12:47:27 UTC (678 KB)