arXiv：cs.LG· Bryce Sandlund·· 3 小时前AI 评分43
训练智能体式上下文管理：Qwen3.6-35B-A3B 用 8K 上下文追平 GPT-5.4 的 1M
Trained Agentic Context Management
AI 导读
研究团队不原生训练长上下文，而是让模型通过"自我调用"和"读取输入上下文 token 区间"两个工具来管理上下文，并据此微调 Qwen3.6-35B-A3B。在 OOLONG-synth 基准上，当文档长度超过 40K tokens 时，仅用 8,000 tokens 上下文的该小模型表现与拥有 1M tokens 上下文的 GPT-5.4 相当。
正文
Abstract:We study long context language models. Instead of training long context natively, or designing a long context harness, we train a model over the simplest possible harness: a tool to call itself with any specified prompt and a tool to read tokens in a range from the input context. We finetune Qwen3.6-35B-A3B on a diverse synthetic dataset using this harness. With only 8,000 tokens of context, our small model is as strong as GPT-5.4 with 1M tokens of context on the OOLONG-synth benchmark when document length exceeds 40K tokens.
| Comments: | 17 pages, 6 figures, 4 tables. Code: this https URL. Under review |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.02404 [cs.LG] |
| (or arXiv:2610.02404v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.02404 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Bryce Sandlund [view email]
[v1]
Thu, 1 Oct 2026 19:32:12 UTC (126 KB)
来源：arXiv：cs.LG · arxiv.org