iFSQ: Improving FSQ for Image Generation with 1 Line of Code

Abstract: The field of image generation is currently bifurcated into autoregressive (AR) models operating on discrete tokens and diffusion models utilizing continuous latents. This divide, rooted in the distinction between VQ-VAEs and VAEs, hinders unified modeling and fair benchmarking. Finite Scalar Quantization (FSQ) offers a theoretical bridge, yet vanilla FSQ suffers from a critical flaw: its equal-interval quantization can cause activation collapse. This mismatch forces a trade-off between reconstruction fidelity and information efficiency. In this work, we resolve this dilemma by simply replacing the activation function in original FSQ with a distribution-matching mapping to enforce a uniform prior. Termed iFSQ, this simple strategy requires just one line of code yet mathematically guarantees both optimal bin utilization and reconstruction precision. Leveraging iFSQ as a controlled benchmark, we uncover two key insights: (1) The optimal equilibrium between discrete and continuous representations lies at approximately 4 bits per dimension. (2) Under identical reconstruction constraints, AR models exhibit rapid initial convergence, whereas diffusion models achieve a superior performance ceiling, suggesting that strict sequential ordering may limit the upper bounds of generation quality. Finally, we extend our analysis by adapting Representation Alignment (REPA) to AR models, yielding LlamaGen-REPA. Codes is available at https://github.com/Tencent-Hunyuan/iFSQ

• The paper introduces iFSQ, which replaces tanh with a sigmoid-based mapping to achieve a near-uniform activation distribution and optimal bin utilization.
• It rigorously benchmarks AR and diffusion models using a unified tokenizer, identifying a latent sweet spot at approximately 4 bits per dimension for optimal performance.
• The study adapts the REPA method to accelerate semantic feature emergence in AR models, offering practical insights for scalable, high-fidelity image generation.

iFSQ: Finite Scalar Quantization Optimization for Unified Image Generation

Introduction

The landscape of image generation has long been divided between autoregressive (AR) models using discrete tokenization and diffusion models operating in a continuous latent space. This bifurcation, primarily derived from the underlying tokenizers—Vector Quantized VAEs (VQ-VAE) for AR and standard VAEs for diffusion—has stymied fair benchmarking and unified modeling frameworks. Finite Scalar Quantization (FSQ) was proposed as a codebook-free route for bridging this gap. However, vanilla FSQ's equal-interval quantization, mismatched to the typically Gaussian-distributed neural activations, produces suboptimal bin utilization ("activation collapse"), forcing a trade-off between reconstruction fidelity and information efficiency.

This paper introduces iFSQ, an optimized FSQ variant implemented via a one-line replacement of the standard activation function. By substituting $tanh$ with a distribution-matching mapping ($2.0 \cdot \sigma(1.6 x) - 1$), iFSQ enforces an approximately uniform activation distribution, ensuring optimal bin utilization and superior reconstruction precision. Through exhaustive empirical analysis, iFSQ is leveraged for controlled comparative studies of AR and diffusion models, elucidating scaling laws and the inherent trade-offs in discrete versus continuous latent representations.

Figure 1: Empirical analysis showing the impact of equal-probability vs. equal-interval quantization on bin utilization and reconstruction error for Gaussian-like activations.

Methodological Advances: Activation Distribution Matching in FSQ

FSQ compresses the latent vector $z \in \mathbb{R}^{N \times d}$ by bounding it via $f(z)$ (traditionally $tanh$), followed by uniform quantization. This retains continuous latents for diffusion and enables discrete scalar indices for AR models, free of explicit or memory-intensive codebooks. However, $tanh$-bounded Gaussian activations concentrate values centrally, leaving edge bins underutilized (bin utilization $\approx83\%$), thus degrading information entropy and diversity.

iFSQ replaces $tanh(z)$ with $2.0 \cdot \sigma(1.6 x) - 1.0$, where $\sigma$ is the sigmoid function and $\alpha=1.6$ yields activation distributions maximally aligned with the uniform target (minimized RMSE and KS statistics).

Figure 2: Numerical study of the effect of varying $\alpha$ in $2.0 \cdot \mathrm{sigmoid}(\alpha x) - 1.0$ on transforming Gaussian to uniform—optimizing for uniformity at $\alpha=1.6$.

This simple change guarantees, both theoretically and empirically, high entropy bin utilization and reconstruction accuracy simultaneously. The modification is computationally negligible and fully compatible with extant tokenization pipelines.

Figure 3: Performance metrics (PSNR, SSIM, LPIPS) and distributional alignment (RMSE, KS) as a function of $\alpha$; iFSQ ($\alpha=1.6$) optimally balances fidelity and efficiency.

Unified Benchmarking: AR vs. Diffusion via iFSQ

Integrating iFSQ as a universal tokenizer, the study establishes a rigorously controlled benchmark to dissect the comparative strengths of AR and diffusion models. With both using the same pretrained tokenizer, direct attribution of observed differences to model architecture rather than quantization artifacts becomes feasible.

Key results:

• AR models exhibit rapid convergence in early training under identical reconstruction constraints.
• Diffusion models attain a higher asymptotic performance ceiling, especially as computational budget increases.

  Figure 4: FID performance across quantization levels and latent dimensionality for iFSQ and AE; optimal quality is achieved at $\approx4$ bits per dimension.

  

  Figure 5: Scaling law depicting the linear relationship between generation quality and compression ratio on log scale for iFSQ and baselines.

A critical empirical observation is the emergence of a latent "sweet spot" at $\approx4$ bits per dimension, where the trade-off between discrete and continuous regime fidelity is optimized. Increasing beyond this point yields diminishing returns in both AR and diffusion models.

Figure 6: Layer-wise analysis of AR model features (STS, NTS, CKNNA), indicating the transition from self-encoding to next-token prediction and alignment with semantic representation.

Representation Alignment in AR: Extension of REPA

Representation Alignment (REPA), originally developed to expedite semantic feature formation in diffusion transformers, is adapted here for autoregressive LlamaGen, producing LlamaGen-REPA. Three metrics are tracked per layer: Self-Token Similarity (STS), Next-Token Similarity (NTS), and semantic alignment (CKNNA) with DINOv2 features.

Empirical analysis elucidates:

• STS diminishes across depth, while NTS and CKNNA spike, coincidentally indicating the transition point to semantic, generation-ready feature representation.
• Applying REPA to intermediate layers accelerates semantic emergence; optimal alignment is achieved when aligning at roughly one-third depth for both AR and diffusion architectures.
• AR models require significantly higher REPA regularization ($\lambda \approx2.0$) compared to diffusion ($\lambda \approx0.5$), likely due to autoregressive inductive biases.

  Figure 7: CKNNA scores illustrating semantic alignment peaks corresponding to REPA-applied layers for varying depths and regularization coefficients.

  

  Figure 8: Ablation study showing optimal REPA depth scales proportionally with model depth for AR and diffusion architectures (best at $\approx$ one-third of total layers).

Theoretical and Practical Implications

The work demonstrates that activation function selection for quantization can decisively affect tokenization efficiency and generative fidelity, opening doors for unified tokenizers in multimodal and cross-paradigm settings. iFSQ offers a scalable, codebook-free approach that affords flexible trade-offs and strong performance for both discrete and continuous regimes.

Practically, iFSQ is well-suited for compact representation learning, efficient large-scale generation, and high-throughput tasks. The revealed scaling laws and sweet spots inform future tokenizer and architecture design, particularly for resource-constrained deployment. The REPA adaptation for AR models provides a direct route to enhancing semantic capacity and efficiency.

Theoretically, the results highlight the critical role of latent distribution shaping in compressive representation learning and the limitations imposed by sequential inductive biases, suggesting avenues for hybrid or non-autoregressive generative schemes.

Conclusion

This paper presents iFSQ, a mathematically grounded, implementation-simple enhancement to Finite Scalar Quantization, aligning activation distributions for optimal bin efficiency and reconstruction fidelity. Via unified benchmarking, the study identifies distinct scaling properties and trade-offs between autoregressive and diffusion models, locating the discrete-continuous equilibrium at approximately 4 bits per dimension. The adaptation of REPA further accelerates semantic feature emergence in AR models. These findings advance the design of unified tokenizers and inform practical choices in high-quality image generative modeling.