arXiv：cs.AI· Harsha Patnala, Debopriyo Banerjee, Ayush Sunil Munot, Somak Aditya·· 13 小时前
TDDN：面向拼图理解的对齐文本扩散 DINO 网络
TDDN: Text-aligned Diffused DINO Network for Puzzle Understanding
AI 导读
TDDN 将 DINOv3 与 CleanDIFT 表征融合为感知编码器 DiffusedDINO，并与 RoBERTa-L 对齐，在冻结骨干、仅约 590K 对齐样本下超越 CLIP ViT-L/14。
正文
Abstract:Structured visual reasoning, such as image puzzles, demands fine-grained visual perception, which current Vision Language Models (VLMs) often lack. VLMs built on CLIP-based ViT backbones tend to trade fine-grained visual details for high-level semantics. We show that this deficit propagates downstream, affecting various vision and language tasks. To address this, we fuse DINOv3 and CleanDIFT representations into a perception encoder (DiffusedDINO), and align it with RoBERTa-L, yielding a text-aligned model (TDDN) that preserves this perceptual advantage: with frozen backbones and only ~590K alignment pairs, TDDN surpasses CLIP ViT-L/14 in image-text retrieval tasks: text-to-image on Flickr30K and both directions on MS-COCO-2014. TDDN more than triples CLIP's dense-prediction accuracy (ADE20K 5.20 -> 18.11 mIoU, COCO-Stuff 7.35 -> 24.44), despite CLIP's massive training corpus. TDDN leads on segmentation benchmarks among general-purpose contrastive encoders, including SigLIP 2. We further introduce PuzzlePerception, a segmentation and visual question answering dataset that probes fine-grained spatial understanding, on which TDDN doubles CLIP's segmentation accuracy (11.04 -> 22.51 mIoU). Our code and data are available at this https URL.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.07937 [cs.CV] |
| (or arXiv:2609.07937v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2609.07937 arXiv-issued DOI via DataCite |
Submission history
From: Debopriyo Banerjee [view email]
[v1]
Mon, 7 Sep 2026 19:52:26 UTC (2,256 KB)
[v2]
Fri, 9 Oct 2026 01:06:20 UTC (2,883 KB)
来源：arXiv：cs.AI · arxiv.org