arXiv：cs.CL· Haoyuan Wu, Aoqi Wu, Jiani Hu, Bei Yu·· 21 天前
视觉内容是多模态预训练的可扩展锚点：超越图像理解的研究
More Than Seeing: Vision as Scalable Anchors in Native Multimodal Pre-Training
AI 导读
一项受控消融实验发现，原生多模态预训练中将图像嵌入替换为无内容填充嵌入后，保留视觉内容能在图文样本上改善文本预测，但不实质性改变同等计算预算下的最低纯文本损失。多模态损失的相对收益随计算量近似呈幂律增长，且视觉增益在定位、导航和符号规划任务上最为持续，空间关系与物理常识方面则不太一致。
正文 · 原文
Abstract:Native multimodal pre-training trains a single model from scratch on text and multimodal data, enabling it to understand both language and visual inputs. However, how visual signals shape scaling behavior and support capabilities beyond visual understanding remains unclear. We investigate this question through controlled visual-content ablations across model sizes and training budgets. Specifically, we replace image embeddings with content-free padding token embeddings. Retaining visual content improves text prediction on image-text examples without materially changing the lowest text-only loss at a given compute budget, the optimal balance of model size and training data, or aggregate language scores. The relative multimodal loss benefit grows approximately as a power law of compute. To test benefits beyond understanding images, we evaluate spatial reasoning from text descriptions alone. Both conditions improve with model and data scaling. During training of the largest model, visual gains are most sustained in localization, navigation, and symbolic planning, which involve maintaining object configurations or composing spatial updates; gains in spatial relations, geometry, and physical commonsense are less consistent. These findings identify visual content as a scalable anchor for multimodal learning, with benefits that extend beyond visual inputs while preserving aggregate language performance.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.19443 [cs.CL] |
| (or arXiv:2610.19443v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19443 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Haoyuan Wu [view email]
[v1]
Wed, 14 Oct 2026 02:33:34 UTC (450 KB)
来源：arXiv：cs.CL · arxiv.org