arXiv：cs.LG· Jeffrey B. Lai, John Quinn, Anthony Bao, William Gilpin·· 4 小时前
Test-Time Forecasting：循环深度让 Transformer 在上下文内预测动力学系统
Test-Time Forecasting: Recurrent Depth Enables In-Context Dynamics Prediction
AI 导读
研究者提出 Test-Time Forecasting（TTF）框架，让单步预测器把梯度下降的迭代当作循环深度使用，从而在推理时模拟动力学系统的传播算子。该框架建立在线性注意力与（扩展）动态模式分解之间的联系之上，仅用 ODE 和 PDE 单步预测训练的轻量 TTF 模型，在自回归展开中可泛化到分布外的系统动力学。
正文
Abstract:Foundation models for dynamical systems can forecast unseen partial differential equations (PDEs), ordinary differential equations (ODEs), and time series during inference. However, it remains an open question whether these foundation models truly learn generalizable in-context forecasting strategies. We show that looped transformers are capable of emulating propagators for dynamical systems in-context by first establishing a link between linear attention and (extended) dynamic mode decomposition. We then generalize this insight into the $\textit{Test-Time Forecasting}$ (TTF) framework in which a one-step forecaster leverages iterations of gradient descent as recurrent depth. We demonstrate that lightweight TTF models trained on ODEs and PDEs generalize to out-of-distribution system dynamics during autoregressive rollout after being trained for one-step prediction. Our results reinforce TTF as a theoretically motivated framework for designing recurrent depth models capable of forecasting unseen dynamical systems at test-time.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.19552 [cs.LG] |
| (or arXiv:2610.19552v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19552 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jeffrey B. Lai [view email]
[v1]
Wed, 14 Oct 2026 04:08:44 UTC (4,009 KB)
来源：arXiv：cs.LG · arxiv.org