Vision Foundation Models Can Be Good Tokenizers for Latent Diffusion Models

Abstract: The performance of Latent Diffusion Models (LDMs) is critically dependent on the quality of their visual tokenizer. While recent works have explored incorporating Vision Foundation Models (VFMs) via distillation, we identify a fundamental flaw in this approach: it inevitably weakens the robustness of alignment with the original VFM, causing the aligned latents to deviate semantically under distribution shifts. In this paper, we bypass distillation by proposing a more direct approach: Vision Foundation Model Variational Autoencoder (VFM-VAE). To resolve the inherent tension between the VFM's semantic focus and the need for pixel-level fidelity, we redesign the VFM-VAE decoder with Multi-Scale Latent Fusion and Progressive Resolution Reconstruction blocks, enabling high-quality reconstruction from spatially coarse VFM features. Furthermore, we provide a comprehensive analysis of representation dynamics during diffusion training, introducing the proposed SE-CKNNA metric as a more precise tool for this diagnosis. This analysis allows us to develop a joint tokenizer-diffusion alignment strategy that dramatically accelerates convergence. Our innovations in tokenizer design and training strategy lead to superior performance and efficiency: our system reaches a gFID (w/o CFG) of 2.20 in merely 80 epochs (a 10x speedup over prior tokenizers). With continued training to 640 epochs, it further attains a gFID (w/o CFG) of 1.62, establishing direct VFM integration as a superior paradigm for LDMs.

• The paper demonstrates that integrating frozen vision foundation models into a VAE encoder improves semantic preservation and accelerates convergence compared to traditional distillation methods.
• It introduces the VFM-VAE framework that utilizes multi-scale feature extraction and modular loss functions to balance pixel fidelity with semantic richness.
• Empirical results, including competitive gFID scores on ImageNet, indicate that VFM-VAE achieves robust alignment and superior generative performance.

Vision Foundation Models as Tokenizers for Latent Diffusion Models: An Authoritative Analysis

Introduction and Motivation

The paper "Vision Foundation Models Can Be Good Tokenizers for Latent Diffusion Models" (2510.18457) critically examines the role of visual tokenizers within Latent Diffusion Models (LDMs). Standard practice uses VAEs as tokenizers, with recent efforts seeking to distill representations from Vision Foundation Models (VFMs) to enhance token quality. The authors identify core limitations in alignment-based distillation: empirical brittleness and degradation of semantic information under distributional shifts. Their central thesis is the proposal and systematic validation of VFM-VAE, a framework that directly integrates frozen VFMs into the VAE encoder, bypassing distillation and optimizing both semantic richness and pixel-level fidelity.

Architectural Innovations: VFM-VAE

Encoder Design

Unlike prior approaches that mimic VFMs via alignment loss, VFM-VAE leverages a frozen, pretrained VFM as the encoder. Multi-scale features (shallow, intermediate, final VFM layers) are extracted and concatenated, then projected to a compact latent via a lightweight network. This design preserves distinct semantic hierarchies and ensures efficient latent compression for diffusion training.

Decoder Design

The decoder departs from conventional pixel-oriented architectures:

• Multi-Scale Latent Fusion: Decomposes latent into global and multiple spatial components, utilizing pixel shuffle/unshuffle and hierarchical convolutional operations.
• Progressive Resolution Reconstruction: Employs a sequence of ConvNeXt-based blocks, each synthesizing features at progressively higher resolutions. Global style (from pooled latent) modulates each block, ensuring consistent semantics, while spatial components inform coarse-to-fine structure.
• Direct Supervision: Each block connects to a ToRGB output head, enforcing learning signals at all scales and facilitating stable, detail-rich reconstruction.

Training Objectives

VFM-VAE's loss is modular:

• Representation Regularization: KL divergence combined with VF loss (cosine/matrix similarity to VFM features).
• Multi-Resolution L1 Reconstruction: Enforces pixel fidelity at each synthesis stage.
• Adversarial and Perceptual Losses: DINOv2-based discriminator and LPIPS further enhance perceptual realism.

This comprehensive loss portfolio is crucial for balancing semantic preservation and continuous latent quality.

Representation Analysis and Metrics: CKNNA and SE-CKNNA

Recognizing the inadequacy of standard CKNNA in capturing semantic equivalence, the paper introduces SE-CKNNA, applying semantic-preserving transformations and measuring feature robustness. VFM-VAE exhibits substantially higher SE-CKNNA than conventional VAEs and alignment-based approaches, indicating superior retention of VFM properties under distributional perturbations.

Layer-wise CKNNA analysis in diffusion models shows that VFM-VAE achieves higher alignment, especially when paired with shallow-layer supervision strategies (e.g., REG), unlike baselines that exhibit diminishing alignment in lower layers. These findings corroborate the hypothesis that direct VFM encoding avoids representation collapse and supports stronger internal feature consistency.

Empirical Results: Convergence, Fidelity, and Efficiency

The authors report highly competitive quantitative metrics:

• ImageNet 256×256 Results: VFM-VAE achieves a gFID of 2.20 after only 80 epochs and 1.62 after 640 epochs (without classifier-free guidance), representing a $10\times$ speedup in convergence compared to prior methods.
• Reconstruction Quality: Standalone VFM-VAE attains rFID and IS scores on par with VA-VAE baselines, despite using fewer training images, confirming its architectural advantage.
• Joint Alignment: Integrating VFM-VAE with REG alignment in diffusion models yields uniformly high CKNNA across all layers and improved generative metrics, substantiating the effectiveness of combined tokenizer/model alignment.

Ablation studies demonstrate that each module (multi-scale fusion, modern block substitution, encoder modifications) is indispensable, contributing cumulatively to significant fidelity and semantic gains.

Practical and Theoretical Implications

Practically, the VFM-VAE paradigm enables rapid, stable, and semantically-rich diffusion model training, requiring fewer data and epochs. Its compatibility with multiple VFMs (DINOv2, SigLIP2, EVA-CLIP) and integration with state-of-the-art generative backbones (LightningDiT, REG, BLIP3-o for text-to-image tasks) suggests broad applicability in visual synthesis, multimodal modeling, and efficient large-scale generative pipelines.

Theoretically, the approach reinforces the viability of frozen foundational models as robust semantic encoders in generative settings, challenging the necessity of distillation and continual alignment. The introduction and empirical validation of SE-CKNNA set a new standard for assessing latent quality under semantic transformations, potentially informing the design of future representation metrics and diagnostic protocols.

Limitations and Future Directions

VFM-VAE currently targets continuous latent spaces and moderate resolutions, leaving open questions regarding discrete tokenization, ultra-high-resolution synthesis, and adaptation to novel generative domains (e.g., 3D synthesis, video modeling). More exhaustive exploration of the trade-offs across different VFMs and further scaling experiments are warranted. Future work may examine its integration into unified vision-LLMs and the limits of latent space compression without sacrificing semantic fidelity.

Conclusion

The paper presents VFM-VAE, an innovative architectural and training methodology for leveraging Vision Foundation Models as tokenizers in Latent Diffusion Models. By structurally and loss-wise fusing semantic richness with reconstruction fidelity, the approach achieves superior convergence rates, alignment robustness, and generative performance, validated by both novel metrics and strong empirical results. VFM-VAE provides a scalable foundation for efficient, semantically-aware generative modeling, with broad implications for future advances in AI-driven visual synthesis.