从策略评估到离策略控制：Transformer 上下文 Q-Learning 的可证明涌现

View PDF HTML (experimental)

Abstract:Transformers can implement reinforcement-learning algorithms in context, yet how off-policy control emerges within Transformers remains poorly understood. We study Q-learning, a foundational model-free control algorithm that combines off-policy bootstrapping with greedy policy improvement. We prove that a two-layer attention-only Transformer can implement a finite-temperature form of Q-learning, where the temperature $\tau>0$ controls the approximation to greedy action selection, recovering window Q-learning as $\tau\to0$. We train the structured Transformer on randomly generated MDP tasks using output-only supervision. Experiments show that the theoretically predicted parameter structure emerges during training, including the internal greedy action-selection mechanism, and that the trained Transformer achieves performance comparable to Q-learning on unseen tasks. Our theoretical analysis further shows that the equivalent realizations form a solution manifold and that, under suitable task-diversity conditions, the training iterates converge locally to this manifold at an exponential rate. For the random-task model analyzed here, the local worst-case number of gradient steps needed to reduce the distance to the manifold by a factor $\epsilon$ is $\Theta\left(\tau^{-5}\log(1/\varepsilon)\right)$, quantifying the increasing optimization cost as the learned update approaches window Q-learning.

Subjects:	Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2610.18057 [cs.LG]
	(or arXiv:2610.18057v1 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2610.18057 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Yingjun Mo [view email]
[v1] Tue, 13 Oct 2026 09:44:36 UTC (823 KB)