arXiv：cs.LG· Soichiro Kumano·· 4 小时前AI 评分33
对抗训练过的线性 Transformer 是高斯混合模型的最优鲁棒上下文学习器
Adversarially Trained Linear Transformers Are Optimal Robust In-Context Learners for Gaussian Mixtures
AI 导读
一项理论研究证明，在跨任务对抗预训练后，足够深的线性 Transformer 无需额外任务特定对抗训练，即可通过干净示例的上下文学习，在未见过的任务上渐近达到鲁棒 Bayes 误差，而标准训练的模型无法做到。研究针对一族高斯混合分类任务，并进一步分析了梯度流下的收敛性、精度与鲁棒性的权衡以及示例复杂度。
正文
Abstract:Adversarial training is one of the most reliable defenses against adversarial attacks, but its high computational cost must generally be paid anew for each task. Robust foundation models offer a promising alternative: adversarially pretrain a model once and then transfer its robustness to downstream tasks through lightweight adaptation. However, a fundamental question remains open: can robustness acquired during pretraining transfer to unseen tasks without further adversarial training? In this study, we answer this question affirmatively. A single model adversarially pretrained at scale can achieve optimal robustness on new tasks without additional task-specific training. Specifically, we show that, for a family of Gaussian-mixture classification tasks, a sufficiently deep linear transformer adversarially trained across tasks can asymptotically attain the robust Bayes error on previously unseen tasks through in-context learning from clean demonstrations. By contrast, a standardly trained model cannot. We further analyze convergence under gradient flow, an accuracy--robustness trade-off, and demonstration complexity.
| Subjects: | Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (stat.ML) |
| Cite as: | arXiv:2610.07754 [cs.LG] |
| (or arXiv:2610.07754v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.07754 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Soichiro Kumano [view email]
[v1]
Tue, 6 Oct 2026 04:51:08 UTC (139 KB)
来源：arXiv：cs.LG · arxiv.org