arXiv：cs.LG· Alan Baade, Kyle Sargent, Vincent Sitzmann, Eric Ryan Chan, Justin Johnson, Li Fei-Fei, Gordon Wetzstein·· 5 小时前
生成式表征学习实现可扩展潜在 tokenization，比 DINOv2 少 7 倍 FLOPs
Generative Representation Learning for Scalable Latent Tokenization
AI 导读
研究者提出一种自监督表征学习方法，通过分解数据的逐点互信息来监督表征损失，可用作生成建模的 tokenizer，并避免了 DINO 和 LeJEPA 等方法出现的维度坍缩与特征抑制。
正文
Abstract:We introduce an approach for self-supervised representation learning designed to be used as a tokenizer for generative modeling. Our objective supervises a representation loss by factorizing the pointwise mutual information of the data, which we estimate by taking information differences between conditional predictions from a learned generative model. We justify our algorithm theoretically and empirically, demonstrating that our generative approach to representation learning avoids dimensional collapse and feature suppression in cases where commonly used methods like DINO and LeJEPA fail. Moreover, we show our representation learning technique is scalable; expending more compute leads to predictably better latent tokenization. For the first time, we perform a careful analysis of the combined flop efficiency of tokenizer and generative model and demonstrate that our approach is a compute-optimal improvement over direct diffusion for pixel-space generation, obtaining an FID-50K of 2.18 with JiT-L. Applied to ImageNet and the text-to-image dataset GPIC, we show tokenizer encoder scaling laws for diffusability, where extrapolating our approach achieves representations in 7x fewer FLOPs than DINOv2 on ImageNet and 8x fewer FLOPs than WebSSL on GPIC.
| Comments: | 22 pages, 8 figures |
| Subjects: | Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2610.20533 [cs.LG] |
| (or arXiv:2610.20533v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20533 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Alan Baade [view email]
[v1]
Wed, 14 Oct 2026 17:29:09 UTC (3,389 KB)
来源：arXiv：cs.LG · arxiv.org