arXiv：cs.LG· Jintian Zhang, Feng He, Xinnian Liang, Ziwen Xu, Haoming Xu, Haojing Huang, Shuangzhi Wu, Ningyu Zhang·· 3 小时前
Cortex：将上下文整合进参数化记忆
Cortex: Consolidating Context into Parametric Memory
AI 导读
Cortex 通过生成前瞻性查询并蒸馏上下文条件输出，将上下文带来的性能增益内化为可合并的低秩或结构化前馈网络更新，覆盖 4B–120B 稠密与 MoE 骨干模型。在 14B 模型上连续 50 次事实编辑后，Cortex-Structural 恢复 88.9% 的上下文增益，同时保留 99.8% 的原始非目标性能。
正文
Abstract:Context can improve large language model performance, but these gains may be lost after context removal. Inspired by biological systems consolidation, we formulate Memory Consolidation (MC) as internalizing these gains while preserving non-target capabilities, connecting context-specified editing, skill learning, and behavioral steering under a common objective. We introduce a five-task-family testbed measuring consolidation and retention under single, batch, and sequential updates. We propose Cortex, which generates prospective queries to anticipate uses of context and distills context-conditioned outputs into mergeable low-rank or structural feed-forward network updates. Across 4B--120B dense and mixture-of-experts backbones, including both text-only and multimodal models, Cortex lies on the empirical consolidation--retention Pareto frontier in most evaluated settings. After 50 sequential factual edits on a 14B model, Cortex-Structural recovers 88.9% of the context-induced gain while retaining 99.8% of original non-target performance.
| Comments: | Technical report |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG); Multiagent Systems (cs.MA) |
| Cite as: | arXiv:2610.20399 [cs.CL] |
| (or arXiv:2610.20399v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20399 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ningyu Zhang [view email]
[v1]
Wed, 14 Oct 2026 16:03:01 UTC (5,250 KB)
来源：arXiv：cs.LG · arxiv.org