arXiv：cs.LG· Armand de Villeroch\'e, Sibo Cheng, Vincent Le Guen, Marc Bocquet, Rem-Sophia Mouradi, Patrick Armand, Alban Farchi, Patrick Massin·· 2 天前AI 评分36
Transformer 神经算子如何实现更大域上的零样本泛化
Zero-shot generalization of transformer neural operators to larger domains
AI 导读
研究者提出一种在注意力 logits 中引入可分解偏置的方法，结合旋转位置嵌入，让 Transformer 神经算子在训练域显著更大的空间域上实现零样本推理，且无需改动 Transformer 架构。该方法在两个 PDE 基准和 3D 工业大气流应用上大幅提升了向更大域的零样本泛化能力，代码与数据集已公开。
正文
Abstract:Transformer-based neural operators have shown remarkable performance for approximating solution operators of partial differential equations on complex geometries. However, existing approaches implicitly assume a fixed domain size, which limits their ability to generalize at inference. In this work, we investigate domain extension, namely zero-shot inference on spatial domains that are significantly larger than those encountered during training. We argue that this setting fundamentally requires spatial locality and translation equivariance. We propose to implement this locality via a decomposable bias in the attention logits computation, enabling finely controllable locality while remaining fully decomposable into query-key inner products and directly compatible with optimized attention kernels. Combined with rotary positional embeddings, it enables expressive embeddings with controllable spatial support without altering the transformer architecture. We empirically show that our approach substantially improves zero-shot generalization to larger domains across two PDE benchmarks and a 3D industrial atmospheric flow application. Our code and datasets are available at this https URL.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2606.14597 [cs.LG] |
| (or arXiv:2606.14597v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2606.14597 arXiv-issued DOI via DataCite |
Submission history
From: Armand De Villeroché [view email]
[v1]
Fri, 12 Jun 2026 16:17:31 UTC (10,750 KB)
[v2]
Thu, 1 Oct 2026 09:13:45 UTC (10,813 KB)
来源：arXiv：cs.LG · arxiv.org