CV-ICRL：用上下文价值改进上下文强化学习

View PDF HTML (experimental)

Abstract:In-context reinforcement learning (ICRL) has emerged as an effective paradigm for test time adaptation to unseen tasks without parameter updates. However, existing ICRL methods can exhibit brittle and unstable adaptions, and the mechanisms underlying such adaptions remain poorly understood. We provide a policy-gradient view of ICRL and argue that relying on trajectory-level return feedback can lead to high-variance updates when test time rollout is limited. Motivated by the actor--critic principle, we propose Context Value Informed ICRL (CV-ICRL), which equips the in-context policy with an explicit context value: the expected discounted return conditioned on the current state and accumulated interaction context. CV-ICRL trains a value head and writes its prediction back into the context as a value token, enabling TD-style return targets for lower-variance policy updates. Experiments on the Dark Room, Minigrid, and Procgen testbeds show that CV-ICRL substantially improves the stability of test time adaptation and achieves higher returns across tasks and environments. Code and data are available at this https URL .

Comments:	35 pages, 10 figures. Camera-ready version for NeurIPS 2026. Updated title, substantially revised presentation, and expanded experiments
Subjects:	Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2509.23209 [cs.LG]
	(or arXiv:2509.23209v2 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2509.23209 arXiv-issued DOI via DataCite

Submission history

From: Wenhao Zhang [view email]
[v1] Sat, 27 Sep 2025 09:42:19 UTC (2,343 KB)
[v2] Wed, 14 Oct 2026 08:35:06 UTC (3,567 KB)