arXiv：cs.AI· Hanpeng Liu, Yaqian Li, Zidan Wang, Shuoxi Zhang, Zonglin Zhao, Zihao Bo, Rinyoichi Takezoe, Kaiwen Long, Kun He·· 13 小时前
ITO：面向图文预训练的多视图对齐与训练时融合框架
ITO: Multi-View Alignment and Training-Time Fusion for Image-Text Pretraining
AI 导读
ITO 通过多模态多重对齐和训练时融合模块改进图文对比预训练，让表征按语义而非模态组织，融合模块在推理时丢弃以保持双编码器效率。在 100M–1B 图文对规模下，ITO 在相同数据、骨干和算力下相较 CLIP 在分类、检索和多模态基准上取得一致提升；对照实验表明增益并非来自额外训练算力或图像增强。
正文
Abstract:Image--text contrastive pretraining has become a dominant paradigm for visual representation learning, yet existing methods often yield representations that remain partially organized by modality rather than by semantics. We propose ITO, a framework addressing this limitation through two complementary mechanisms with distinct roles. Multimodal multiple alignment enriches supervision by constructing diverse cross-modal correspondences from multi-view image augmentations, providing the primary source of discriminative gain. A lightweight training-time multimodal fusion module then acts as a training-time regularizer, encouraging the encoders to produce features that are compatible under fusion. Crucially, the fusion module is discarded at inference, preserving the efficiency of standard dual-encoder architectures while incurring training-time overhead only. Extensive experiments across pretraining scales from millions to billions of image--text pairs show that ITO outperforms strong contrastive baselines at moderate scale and yields consistent gains over CLIP under identical data, backbone, and compute at the 100M--1B scale, on classification, retrieval, and multimodal benchmarks. Controlled comparisons show that these gains are not explained by additional training compute or image augmentation alone. Our analysis further reveals that the contribution of fusion grows with data scale and that it stabilizes optimization, mitigating the late-stage overfitting commonly observed in aggressive contrastive learning. Code is available at this https URL.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2603.02767 [cs.CV] |
| (or arXiv:2603.02767v5 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2603.02767 arXiv-issued DOI via DataCite |
Submission history
From: Hanpeng Liu [view email]
[v1]
Tue, 3 Mar 2026 09:08:53 UTC (6,521 KB)
[v2]
Wed, 4 Mar 2026 01:56:54 UTC (6,521 KB)
[v3]
Mon, 9 Mar 2026 09:27:09 UTC (6,521 KB)
[v4]
Wed, 30 Sep 2026 08:27:16 UTC (6,522 KB)
[v5]
Fri, 9 Oct 2026 01:14:50 UTC (6,523 KB)
来源：arXiv：cs.AI · arxiv.org