MM-Tokenizer: Unified Multimodal Encoding

• MM-Tokenizer is a unified system that transforms diverse modalities using modality-specific encoders and quantizers into discrete token sequences.
• End-to-end training with joint reconstruction and caption losses enables semantic alignment and gradient flow, improving multimodal understanding by 2–7%.
• Integrated with autoregressive sequence models, MM-Tokenizers facilitate efficient token compression, scalable reconstruction, and high-performance generation across domains.

A Multimodal Tokenizer (MM-Tokenizer) is a module that converts diverse input modalities—including images, audio, videos, and segmentation masks—into discrete token sequences amenable to sequence modeling by LLMs or multimodal transformers. MM-Tokenizers bridge the gap between non-text modalities and autoregressive modeling by enabling all modalities to be processed, predicted, and generated as token sequences within a joint or shared token space. They are foundational to unified multimodal LLMs, supporting both understanding and generation tasks across modalities, and facilitating gradient flow, semantic alignment, and scalable integration with next-token prediction paradigms.

1. Architectural Paradigms of Multimodal Tokenizers

MM-Tokenizers encompass a broad design space; their core is a modality-specific encoder that compresses the input into high-level features, followed by a quantizer producing discrete token indices. For images, state-of-the-art designs predominantly use vector quantization (VQ) mechanisms with learnable codebooks. Early paradigms decoupled tokenizer training—optimizing for low-level reconstruction—while assuming these tokens would generalize across downstream tasks (e.g., image captioning, visual question answering). This separation introduces critical misalignment, as task-specific information is not captured in the discretization process. The End-to-End Tokenizer Tuning (ETT) method (Wang et al., 15 May 2025) demonstrates that replacing the conventional frozen-tokenizer paradigm with a fully differentiable architecture—where gradients from downstream losses flow into both encoder and discrete codebook embeddings—eliminates the representational bottleneck introduced by task-agnostic quantization.

For complex tasks such as multimodal recommendation, MMQ (Xu et al., 21 Aug 2025) introduces a mixture-of-experts framework, in which modality-shared and modality-specific subnetworks extract comprehensive and unique features respectively, followed by a mixture-of-quantization step leveraging shared codebooks and a “soft index” mechanism for gradient flow. In temporal domains, such as video, MM-Tokenizers like Divot (Ge et al., 2024) integrate temporal transformers, Perceiver modules, and VAE or diffusion-based latent spaces to produce continuous or quantized token streams representing both spatial content and temporal dynamics.

The general pipeline consists of:

• Modality-specific encoder: e.g., convolutional nets, ViTs, temporal Transformers, audio encoders.
• Quantizer: either vector-quantization (nearest neighbor or product-quantized) or residual quantization schemes.
• Codebook(s): trained by VQ, Gumbel, or product quantization, often with tens to hundreds of thousands of centroids.
• Embedding projector: linear/MLP transformation mapping codebook vectors to LLM embedding size.
• Optional: multi-expert (shared/specific) paths, dynamic token-length prediction, hierarchical codebook schemes, or clustering-based aggregation for compression.

2. Training Objectives, Losses, and End-to-End Optimization

Early MM-Tokenizer training relied solely on pixel/feature reconstruction objectives (e.g., $\ell_2$ loss, perceptual loss, adversarial loss, codebook commitment penalties). This approach neglects modality semantics that emerge in downstream sequence modeling, resulting in substantial information loss for tasks requiring high-level reasoning. End-to-end frameworks, as exemplified by ETT (Wang et al., 15 May 2025), jointly optimize reconstruction and caption (autoregressive) losses: $L = L_\text{cap} + \alpha L_\text{vq}$ where $L_\text{cap}$ is the cross-entropy loss over next-token prediction (using both visual and text tokens), and $L_\text{vq}$ pools reconstruction, codebook, perceptual (LPIPS) and adversarial (GAN) losses. The weighting parameter $\alpha$ determines the trade-off: a value of $\alpha=0.25$ is observed to yield nearly the original reconstruction fidelity and strong multimodal semantic alignment. Gradients flow through the token embeddings back to the codebook and encoder, enabled by continuous embeddings and light MLP projectors.

Other schemes exploit multi-stage or curriculum training, e.g., MedITok (Ma et al., 25 May 2025) first cold-starts on image reconstruction and weak semantic alignment with vision encoders, then aligns the quantized latent space with text encoders on paired image–caption data using a contrastive InfoNCE loss. MMQ (Xu et al., 21 Aug 2025) additionally adds auxiliary reconstruction per modality and orthogonal regularization to prevent mode collapse among MoE experts.

For segmentation and adaptive-length tasks, hierarchical mask losses as in HiMTok (Wang et al., 17 Mar 2025) or length-penalized losses as in ALTo (Wang et al., 22 May 2025) ensure that both coarse- and fine-grained information is recoverable from variable-length or prefix tokens, with differentiable chunking to support end-to-end gradient flow.

3. Token Structure: Discrete/Continuous, Hierarchical, and Adaptive Schemes

Modern MM-Tokenizers depart from fixed patch-based discretization in favor of more expressive—and sometimes adaptive—tokenization strategies:

• Hierarchical Codebooks: SemHiTok (Chen et al., 9 Mar 2025), TokenFlow (Qu et al., 2024), and related methods decouple semantic and pixel-level representations. They assign each patch first to a semantic codebook (e.g., CLIP-based, $K=16\text{K}$), then quantize local texture against a small, context-dependent sub-codebook. At inference, a single patch yields a tuple of indices, mapped bijectively to a unified token id.
• Slot- or Object-centric Tokenization: Slot-MLLM (Chi et al., 23 May 2025) adopts slot attention to carve an image into $N$ sets of continuous object-centric tokens, further quantized via residual VQ and allowing object-level alignment of tokens with language primitives.
• Adaptive Token Lengths: ALTo (Wang et al., 22 May 2025) introduces a token length predictor, allowing segmentation masks (and potentially other signals) to be decoded with exactly as many tokens as their intrinsic complexity requires, regularized by explicit penalty terms, and optimized with straight-through soft chunking.
• Clustering and Semantic Preservation: SeTok (Wu et al., 2024) and token compression schemes (Omri et al., 24 Apr 2025) dynamically cluster visual features (e.g., density-peak or k-means), yielding a token budget proportional to semantic complexity, and capturing object boundaries for better alignment with text.
• Continuous/Hybrid Representations: Diffusion-based video tokenizers (e.g., Divot (Ge et al., 2024)), employ Perceiver resamplers and model token distributions as Gaussian mixtures, permitting both comprehension and generation without fully discrete code indices. Hybrid schemes may allow LLMs to attend to both discrete tokens and continuous global features.

4. Integration with Autoregressive Sequence Models and Downstream Tasks

A central requirement for MM-Tokenizers is seamless integration with LLMs under the next-token prediction paradigm. The canonical pipeline is:

1. Token emission: The tokenizer produces a sequence of visual/audio tokens, typically flattened and concatenated (or interleaved) with text tokens. The mapping is facilitated by a learned joint embedding matrix or a lightweight projector.
2. Autoregressive modeling: The LLM processes the composite sequence as a standard text stream. Modified vocabularies incorporate new token indices for each discrete symbol type (semantic, pixel, segmentation, etc.). No architectural modification to attention or head layers is generally required, except for extending the embedding table and, optionally, separate modality indicator embeddings.
3. End-to-end backpropagation: When using continuous codebook embeddings and lightweight projectors as in ETT (Wang et al., 15 May 2025), or soft-index mechanisms as in MMQ (Xu et al., 21 Aug 2025), gradients from downstream losses flow into the tokenizer, enabling end-to-end adaptation.
4. De-tokenization: Downstream, the LLM’s predicted (or decoded) visual token sequence is mapped back to latent representations via the codebook, then decoded to pixels by a generator (VQ decoder, U-Net, or GAN). Segmentation and mask tokens can similarly be reconstructed without the original image.
5. Multi-modal extensions: The same modeling framework generalizes to audio (TEAL (Yang et al., 2023), DM-Codec (Ahasan et al., 2024)), video (Divot (Ge et al., 2024)), and joint item representations for recommendation (MMQ (Xu et al., 21 Aug 2025)).

Empirical benchmarks show that end-to-end trained MM-Tokenizers yield 2–6% absolute gains in multimodal understanding tasks (GQA, TextVQA, MME) and state-of-the-art image/text generation performance (e.g., rFID ≈1.65, GenEval score 0.43–0.66) (Wang et al., 15 May 2025, Qu et al., 2024, Chen et al., 9 Mar 2025).

5. Modality Extensions and Domain-Specific Customization

While most MM-Tokenizer work has centered on images, the methods generalize to other modalities:

• Audio: Replace the encoder/decoder with waveform or spectrogram VQ-VAEs; optimize with caption, ASR, or self-supervised losses. Distillation of contextual/semantic signals is essential for speech tokenizers, as in DM-Codec (Ahasan et al., 2024).
• Video: Employ spatiotemporal encoders, diffusion decoders, and dense-to-sparse temporal sampling. Perceiver modules and temporal transformers capture motion cues, with de-tokenization via diffusion steps (Ge et al., 2024).
• Text+Item representations: Recommendation frameworks (MMQ (Xu et al., 21 Aug 2025)) implement mixture-of-experts architectures, jointly quantizing text, vision, and user-behavioral features to produce semantic IDs for cross-modal retrieval.
• Medical imaging: Domain-specific encoders (e.g., ViTamin-L) and text aligners (e.g., BiomedCLIP) enable MM-Tokenizers to compress, reconstruct, and interpret specialized image modalities (CT, X-ray, ultrasound) while aligning token space with clinical semantics (Ma et al., 25 May 2025).

6. Compression, Efficiency, and Practical Considerations

Token compression and adaptive token selection are critical for scaling multimodal LLMs. Token aggregators leveraging clustering (e.g., k-means++) reduce token counts by over 8× with minimal impact (<1%) on understanding accuracy (Omri et al., 24 Apr 2025). Saliency-based or clustering-based selection methods are favored over cross-modal attention-based saliency alone, which may not correlate with human interpretability or task relevance. Dynamic clustering and hierarchical codebooks enable further improvements by varying token budgets per sample.

Implementation best practices include:

• Keeping projectors shallow to prevent overfitting.
• Choosing small $\alpha$ for loss balancing (0.1–0.5).
• Scheduling freeze/unfreeze of encoder, codebook, and downstream modules to stabilize gradients.
• Mitigating memory blowup with gradient checkpointing and mixed precision.
• Structuring vocabularies and embedding tables to flexibly accommodate new token classes or modalities.

Key pitfalls are over-tuning for captioning at the expense of reconstruction, excessive codebook sizes resulting in prohibitive memory cost, and lack of semantic alignment between tokens and high-level concepts or tasks.

7. Empirical Outcomes and Significance

The adoption of MM-Tokenizers has led to marked advances in:

• Multimodal Understanding: Consistent 2–7% gain in end-task metrics on VQA and captioning benchmarks over frozen-tokenizer or patch-based baselines (Wang et al., 15 May 2025, Qu et al., 2024, Chen et al., 9 Mar 2025).
• Generation Quality: State-of-the-art autoregressive image and video generation results, with rFID and FID metrics competitive with continuous-token and diffusion model frameworks (e.g., GenEval scores up to 0.66, rFID ≈1.24–1.65) (Wang et al., 15 May 2025, Chen et al., 9 Mar 2025).
• Compression and Scalability: >80% compute savings at <1% accuracy loss through clustering and token budget adaptation (Omri et al., 24 Apr 2025, Wu et al., 2024).
• Specialized Domains: In medical imaging and recommendation, MM-Tokenizers outperform domain-specific and general baselines on reconstruction, classification, and retrieval metrics, and scale to tens of millions of items/modalities (Ma et al., 25 May 2025, Xu et al., 21 Aug 2025).
• Segmentation and Adaptive Tasks: High mask accuracy and token efficiency in segmentation tasks with hierarchical and adaptive-length tokenization (Wang et al., 17 Mar 2025, Wang et al., 22 May 2025).

A plausible implication is that the tokenizer design and tight integration with downstream objectives have become as important as model scale or architecture for achieving robust, scalable, and generalizable multimodal reasoning and generation. MM-Tokenizers have become the linchpin enabling contemporary multimodal LLMs to move beyond naive patchification, toward semantically structured, efficient, and extensible token interfaces for all modalities.