Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens

Abstract: Scaling up autoregressive models in vision has not proven as beneficial as in LLMs. In this work, we investigate this scaling problem in the context of text-to-image generation, focusing on two critical factors: whether models use discrete or continuous tokens, and whether tokens are generated in a random or fixed raster order using BERT- or GPT-like transformer architectures. Our empirical results show that, while all models scale effectively in terms of validation loss, their evaluation performance -- measured by FID, GenEval score, and visual quality -- follows different trends. Models based on continuous tokens achieve significantly better visual quality than those using discrete tokens. Furthermore, the generation order and attention mechanisms significantly affect the GenEval score: random-order models achieve notably better GenEval scores compared to raster-order models. Inspired by these findings, we train Fluid, a random-order autoregressive model on continuous tokens. Fluid 10.5B model achieves a new state-of-the-art zero-shot FID of 6.16 on MS-COCO 30K, and 0.69 overall score on the GenEval benchmark. We hope our findings and results will encourage future efforts to further bridge the scaling gap between vision and LLMs.

• The paper demonstrates that continuous tokens significantly enhance image fidelity by preserving information lost in discrete representations.
• It reveals that random-order token generation, combined with continuous tokens, outperforms raster-order methods with notable FID and GenEval improvements.
• The study introduces the Fluid model, a 10.5B parameter system that scales effectively and sets new benchmarks on the MS-COCO dataset.

Scaling Autoregressive Text-to-Image Generative Models with Continuous Tokens: An Overview

The research presented in "Fluid: Scaling Autoregressive Text-to-Image Generative Models with Continuous Tokens" addresses the scaling challenges of autoregressive models in the vision domain. The study specifically focuses on text-to-image generation, aiming to understand the effects of using continuous versus discrete tokens and different token generation orders.

Core Contributions and Findings

Autoregressive models have shown promise in NLP tasks, but their application in computer vision, specifically text-to-image generation, has lagged. The paper identifies two critical design factors affecting performance: the nature of tokens (continuous vs. discrete) and the order of token generation (random vs. raster). The research demonstrates that:

1. Continuous Tokens: Models using continuous tokens significantly outperform those using discrete tokens in terms of visual quality. The continuous approach avoids the information loss inherent in vector quantization, leading to better image fidelity.
2. Random Order Generation: Random-order models, particularly with continuous tokens, achieve superior performance metrics, such as FID and GenEval scores, compared to raster-order models. This approach benefits from a bidirectional attention mechanism, allowing greater flexibility in adjusting the global structure of generated images.
3. Scaling Behavior: Validation loss scales predictably with model size following a power law, but evaluation metrics do not strictly adhere to this trend. However, substantial improvements in evaluation performance are observed with increased model size and training compute for the most effective configuration—random-order models with continuous tokens.
4. Fluid Model: The introduction of the Fluid model, a random-order autoregressive configuration using continuous tokens, achieves state-of-the-art performance. The 10.5B parameter model records an FID score of 6.16 on the MS-COCO dataset and shows a compelling GenEval overall score of 0.69.

Implications

The findings highlight the potential for continuous tokenization to bridge the performance gap between vision and LLMs, suggesting that improvements observed in LLMs through scaling can be extended to vision tasks under the right configurations. The Fluid model's success in achieving better text-to-image alignment emphasizes the transformative impact of re-thinking token representation and generation order.

Future Directions

The study opens several avenues for future research:

• Advanced Techniques: Exploring other architectural innovations beyond token and order configurations could yield further improvements. Techniques from other domains, such as information retrieval, could enhance generative quality and efficiency.
• Broader Applications: Extending the frameworks and insights from this research to other forms of generative tasks or even multi-modal domains could offer new opportunities for enhanced model capabilities.
• Long-Term Scaling: Understanding the limitations of current models as they are scaled further, including the continuous assessment of trade-offs between compute resources and performance, remains critical.

Conclusion

This research provides a detailed empirical investigation into the scaling behaviors of autoregressive text-to-image models. The introduction of continuous token representations and random-order generation not only marks a significant step in understanding vision model scaling but also establishes a new benchmark in generative model performance. The work signifies a promising direction in aligning the strengths of autoregressive methodologies with the demands of high-quality image generation.