ToMA：面向半监督视觉语言学习的拓扑感知表示对齐

View PDF HTML (experimental)

Abstract:Vision-language models have shown strong performance, but they often generalize poorly to specialized domains. While semi-supervised vision-language learning mitigates this limitation by leveraging a small set of labeled image-text pairs together with abundant unlabeled images, existing methods remain fundamentally pairwise and fail to model the global structure of multimodal representation manifolds. Most existing topology-based alignment methods rely on persistence diagram matching, which neither guarantees geometric alignment nor uses the image-text pairing information central to vision-language learning. We propose Topology-Aware Representation Alignment for Semi-Supervised Vision-Language Learning (ToMA), a framework that uses persistent homology to decompose pairwise relations into $H_0$-death and $H_1$-birth relations and aligns them across modalities through available cross-modal correspondences. This lightweight decomposition captures both connectivity and cycle-closing structure without constructing 2-simplices. Experiments show clear improvements on remote sensing, while fashion results reveal strong gains in classification and particularly strong retrieval improvements for difficult queries. Additional analysis shows that ToMA provides more consistent gains than alternative topology-based objectives and that lightweight $H_1$-birth edges provide useful higher-order structural signals.

Comments:	38 pages, 13 figures, 32 tables
Subjects:	Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG); Algebraic Topology (math.AT)
Cite as:	arXiv:2604.26370 [cs.CV]
	(or arXiv:2604.26370v2 [cs.CV] for this version)
	https://doi.org/10.48550/arXiv.2604.26370 arXiv-issued DOI via DataCite

Submission history

From: Junwon You [view email]
[v1] Wed, 29 Apr 2026 07:30:33 UTC (17,753 KB)
[v2] Wed, 14 Oct 2026 14:28:53 UTC (17,786 KB)