Scaling autoregressive model capacity with increasing iFSQ codebook size

Establish whether increasing the implicit codebook size L^d of the iFSQ tokenizer—by raising the bits per dimension K in L = 2^K + 1—requires proportionally scaling the capacity of the autoregressive transformer (such as LlamaGen) to maintain or improve image generation quality, and characterize how model capacity should scale as the iFSQ codebook grows.

Background

The paper investigates iFSQ as a unified tokenizer for both diffusion and autoregressive models. In the autoregressive setting (e.g., LlamaGen), iFSQ’s implicit codebook size is determined by the number of quantization levels per dimension (L = 2^K + 1) and the latent dimensionality d, yielding an effective codebook size of L^d.

Empirically, the authors report that generation performance peaks around 4 bits per dimension and does not monotonically improve as the number of bits increases. Based on this observation, they explicitly conjecture that as the implicit codebook grows, the autoregressive model may need greater capacity to effectively predict tokens drawn from a larger discrete space.