arXiv：cs.LG· Hao-Run Cai, Si-Yang Liu, Zi-Jian Cheng, Kun-Yang Yu, Jin-Hao Sheng, Guo Yu, Chonghan Liu, Zhi Zhou, Jun-Peng Jiang, Lan-Zhe Guo, Han-Jia Ye·· 3 小时前AI 评分37
Retro：面向表格基础模型的回溯推理
Thinking in Depth: Retrospective Inference for Tabular Foundation Models
AI 导读
研究者提出表格基础模型 Retro，通过回溯推理让网络后段显式重访并重组早期中间表示。Retro 采用 Attention Residuals 自适应重加权不同深度的贡献，并用 query-conditioned Gated Attention 按查询逐元素调制注意力输出，使预测细化更早、更广泛地分布在各层。
正文
Authors:Hao-Run Cai, Si-Yang Liu, Zi-Jian Cheng, Kun-Yang Yu, Jin-Hao Sheng, Guo Yu, Chonghan Liu, Zhi Zhou, Jun-Peng Jiang, Lan-Zhe Guo, Han-Jia Ye
Abstract:Tabular foundation models (TFMs) are pretrained across diverse tabular tasks and make predictions on a new table at inference time using its labeled examples as context. Most recent TFMs perform such in-context prediction with stacked Transformer layers, repeatedly transforming how examples are represented and compared. By tracing individual queries through several strong TFMs, we find that predictive refinement is highly uneven across depth and is often concentrated in later layers. This uneven refinement motivates us to reconsider how intermediate representations are constructed and reused throughout the network. We introduce Retro, a tabular foundation model based on retrospective inference, where later stages can explicitly revisit and recombine intermediate information produced earlier in the network. Retro organizes this process around two complementary operations: which intermediate information to revisit, and how the resulting contextual update should be shaped for each query. Attention Residuals address the former by adaptively reweighting contributions from different depths, while query-conditioned Gated Attention addresses the latter by modulating the attention output element-wise across representation dimensions. Our analysis shows that Retro shifts predictive refinement earlier and more broadly across depth, with different stages revising different subsets of queries in a pattern suggestive of multi-view refinement. Across TabArena, TALENT, and RelArena, Retro ranks among the top three and lies on the Pareto frontier. These results indicate that directly reusing intermediate representations provides a practical way to better exploit depth in TFMs.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.10317 [cs.LG] |
| (or arXiv:2610.10317v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.10317 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Haorun Cai [view email]
[v1]
Wed, 7 Oct 2026 16:11:04 UTC (9,142 KB)
来源：arXiv：cs.LG · arxiv.org