BitDance: Scaling Autoregressive Generative Models with Binary Tokens

Abstract: We present BitDance, a scalable autoregressive (AR) image generator that predicts binary visual tokens instead of codebook indices. With high-entropy binary latents, BitDance lets each token represent up to $2^{256}$ states, yielding a compact yet highly expressive discrete representation. Sampling from such a huge token space is difficult with standard classification. To resolve this, BitDance uses a binary diffusion head: instead of predicting an index with softmax, it employs continuous-space diffusion to generate the binary tokens. Furthermore, we propose next-patch diffusion, a new decoding method that predicts multiple tokens in parallel with high accuracy, greatly speeding up inference. On ImageNet 256x256, BitDance achieves an FID of 1.24, the best among AR models. With next-patch diffusion, BitDance beats state-of-the-art parallel AR models that use 1.4B parameters, while using 5.4x fewer parameters (260M) and achieving 8.7x speedup. For text-to-image generation, BitDance trains on large-scale multimodal tokens and generates high-resolution, photorealistic images efficiently, showing strong performance and favorable scaling. When generating 1024x1024 images, BitDance achieves a speedup of over 30x compared to prior AR models. We release code and models to facilitate further research on AR foundation models. Code and models are available at: https://github.com/shallowdream204/BitDance.

• The paper introduces BitDance, which scales autoregressive generative models by using binary tokenization to achieve high reconstruction fidelity and enhanced latent expressivity.
• It employs a binary diffusion head that jointly models token distributions, effectively mitigating exponential parameter growth and error accumulation typical in softmax-based approaches.
• Additionally, the paper presents a next-patch diffusion method for fast, parallel decoding, resulting in significant speedups and robust performance in high-resolution image synthesis.

BitDance: Scaling Autoregressive Generative Models with Binary Tokens

Overview

The paper "BitDance: Scaling Autoregressive Generative Models with Binary Tokens" (2602.14041) introduces BitDance, a novel autoregressive (AR) model for high-fidelity visual generation, operating on binary visual tokens with unprecedented token entropy. By extending the vocabulary size up to $2^{256}$ through binary quantization, BitDance achieves reconstruction fidelity comparable to or exceeding continuous VAEs. The work incorporates a binary diffusion head that enables efficient joint prediction and sampling in extremely large discrete token spaces, alongside a next-patch diffusion paradigm for fast, parallel decoding. Extensive experiments validate BitDance's superior generative capacity, efficiency, and data scalability across both class-conditional and text-to-image generation tasks.

Figure 1: Performance vs. efficiency compared with SOTA diffusion models and autoregressive models.

Binary Visual Tokenization: Entropy and Scalability

BitDance leverages Lookup-Free Quantization (LFQ) to encode visual information into binary tokens, sidestepping codebook collapse issues inherent to VQ-based approaches. Through group-wise entropy calculation, the model pushes representation cardinality to $2^{256}$, resulting in significantly enhanced expressivity and regularization for AR long-sequence generation. This discrete scheme closes the performance gap with continuous tokenizers:

• On ImageNet 256$\times$256, BitDance with a 32$\times$ downsampling ratio and $2^{256}$ codebook size attains higher PSNR and SSIM than leading continuous VAEs and previous discrete architectures.
• The scaling study shows optimal generative quality when transformer size co-expands with token vocabulary, affirming that larger AR models efficiently exploit the increased latent expressivity.

  Figure 2: High-resolution samples generated by the 14B BitDance model showcasing diverse prompt fidelity, spatial reasoning, and text rendering.

Binary Diffusion Head: Sampling in Large Discrete Spaces

The core innovation is the binary diffusion head, which replaces the traditional softmax-based classification with a diffusion objective in continuous space. Binary tokens are embedded as hypercube vertices, and the model learns their joint distribution via a rectified flow-based velocity-matching loss. This approach addresses the exponential scaling of parameters seen in direct index modeling and recovers intra-token dependencies lost in bitwise classification paradigms. Critically:

• The binary diffusion head achieves efficient sampling without sacrificing precision, circumventing error accumulation plaguing unconstrained continuous AR models.
• Empirical ablation shows binary diffusion head outperforms both token classification (OOM issues) and bitwise independent heads (low fidelity).

  Figure 3: Comparison of sampling paradigms—joint modeling offers high fidelity without exponential parameter growth.

  

  Figure 4: Output distribution of binary diffusion head at different noise levels: predictions sharpen toward discrete binary values as noise decreases.

Next-Patch Diffusion: Accelerated Parallel Prediction

BitDance introduces next-patch diffusion, partitioning token sequences into local patches and enabling parallel token prediction using a block-wise causal attention mask. Unlike prior parallel AR frameworks that assume token independence per patch, BitDance's joint patch modeling aligns the training and inference objectives, preserving structure and coherence. The binary diffusion head naturally extends to multi-token prediction, leveraging DiT-style networks for scalable patch-wise decoding:

• BitDance achieves parameter-efficient, high-fidelity parallel prediction, outperforming 1.4B-parameter SOTA baselines (RandAR-XXL) with only 260M parameters.
• Throughput benchmarks demonstrate significant speedups (up to 8.7$\times$ at 256$\times$256 and 30$\times$ at 1024$\times$1024) over both AR and diffusion models.

  Figure 5: BitDance architecture—images are encoded to binary latents and AR-modeled through next-patch diffusion.

  

  Figure 6: Parallel prediction—binary diffusion head models the joint distribution for coherent multi-token sampling.

Experimental Results

Class-Conditional Generation

BitDance attains an FID of 1.24 on ImageNet 256$\times$256, surpassing previous AR models and diffusion transformers at equivalent model scales. The scaling of token entropy and transformer size yields consistent performance improvements. Ablation studies validate the effectiveness of binary tokenization, joint diffusion modeling, and patch-wise attention configuration.

Figure 7: Generative performance across vocabulary and Transformer sizes—scaling both offers optimal convergence.

Text-to-Image Synthesis

BitDance scales to 14B parameters, leveraging large multimodal datasets (e.g., LAION, Seedream, Z-Image-Turbo) and multi-stage training with mixed resolutions for robust high-resolution generalization.

• Achieves top-tier scores on GenEval (0.86), DPG-Bench (88.28), OneIG-EN (0.532), OneIG-ZH (0.512), and TIIF (79.64).
• The distilled model, transitioning from 16-token to 64-token parallel prediction, maintains generative quality while further accelerating inference.

  Figure 8: Images generated by SFT and distilled models—distillation enables faster inference with comparable quality.

  

  Figure 9: Binary diffusion head sampling steps—high-quality generation is achieved even with few steps (10–20).

Theoretical and Practical Implications

The BitDance framework demonstrates that increasing token entropy in discrete AR models is both feasible and beneficial for generative fidelity, provided scalable sampling and parallel decoding are available. The binary diffusion head fundamentally expands the tractability of sampling in massive discrete spaces, opening new directions for AR generative modeling beyond traditional softmax architectures.

Practically, BitDance's efficient parallel prediction makes key advances toward real-time high-resolution generation, closing the gap with proprietary and diffusion-based systems in both open-source and commercial settings. Its data efficiency and multimodal scalability also suggest new paradigms for unified vision-language modeling and autoregressive foundation model training.

Future Developments

Potential advancements include further scaling of data and model size, extension to agentic multimodal tasks, integration with multi-turn reasoning, and broader exploration of structural priors in discrete latent spaces for AR generative modeling. The approach is theoretically extensible to video and 3D generation, given its robust tokenization and efficient sampling capabilities.

Conclusion

BitDance establishes a modular, scalable AR generative framework centered on binary token modeling with extreme entropy. The binary diffusion head and next-patch diffusion paradigm enable efficient, high-fidelity multi-token prediction, overcoming sampling bottlenecks and accelerating inference. The method’s empirical results and architectural innovations contribute significant insight to the design space of visual AR modeling and provide practical solutions for multimodal generative systems (2602.14041).