CogView3: Finer and Faster Text-to-Image Generation via Relay Diffusion

Abstract: Recent advancements in text-to-image generative systems have been largely driven by diffusion models. However, single-stage text-to-image diffusion models still face challenges, in terms of computational efficiency and the refinement of image details. To tackle the issue, we propose CogView3, an innovative cascaded framework that enhances the performance of text-to-image diffusion. CogView3 is the first model implementing relay diffusion in the realm of text-to-image generation, executing the task by first creating low-resolution images and subsequently applying relay-based super-resolution. This methodology not only results in competitive text-to-image outputs but also greatly reduces both training and inference costs. Our experimental results demonstrate that CogView3 outperforms SDXL, the current state-of-the-art open-source text-to-image diffusion model, by 77.0\% in human evaluations, all while requiring only about 1/2 of the inference time. The distilled variant of CogView3 achieves comparable performance while only utilizing 1/10 of the inference time by SDXL.

• The paper introduces a relay diffusion framework that refines low-resolution images into high-quality outputs, enhancing both efficiency and fidelity.
• It achieves a 77% improvement in human evaluations while significantly reducing inference time compared to leading models like SDXL.
• Its distilled version maintains competitive performance with just one-tenth of SDXL's inference time, demonstrating strong scalability for future applications.

Innovations in Text-to-Image Generation with CogView3: Leveraging Relay Diffusion

Introduction to CogView3

The field of generative models, particularly text-to-image models, has witnessed a substantial evolution in recent years, largely propelled by the advent and refinement of diffusion models. These models, which generate images from textual descriptions, have become a focal point in the intersection of natural language processing and computer vision. Amidst this burgeoning field, the paper introduces CogView3, a model that stands out by adopting a novel cascaded framework known as relay diffusion to enhance the efficiency and fidelity of text-to-image generation.

Relay Diffusion: A Novel Approach

CogView3's introduction of relay diffusion in text-to-image synthesis presents a significant shift from conventional single-stage diffusion processes. Instead of generating high-resolution images directly, CogView3 first generates images at a lower resolution, which then serve as the foundation for a relay-based super-resolution process. This methodology has multiple benefits:

• Computational Efficiency: By breaking down the diffusion process into stages, CogView3 significantly reduces both training and inference costs. This is achieved by initially dealing with images at lower resolutions and minimizing the computational load in the early stages.
• Enhanced Image Quality: The relay diffusion process allows for the rectification of any artifacts or inconsistencies in the base generation through its super-resolution phase. This iterative refinement leads to outputs with higher fidelity and detail.

Benchmarks and Comparisons

The empirical evaluation of CogView3 offers compelling evidence of its capabilities. When benchmarked against SDXL, a leading open-source text-to-image diffusion model, CogView3 achieved a 77.0\% improvement in human evaluations, all while requiring approximately half the inference time. Moreover, the model's distilled version maintains comparable performance to SDXL while significantly reducing inference time to about 1/10th of SDXL's requirement. These results not only underscore CogView3's computational advantages but also its ability to produce text-to-image outputs that are competitive in quality.

Implications and Future Directions

CogView3's success points toward several implications and potential avenues for future research:

• Relay Diffusion in Other Domains: The relay diffusion approach, proven effective for text-to-image generation, could be explored within other generative tasks. Its adaptability to different stages of resolution might offer similar benefits in domains like video generation or three-dimensional model synthesis.
• Model Distillation in Generative Models: The effective distillation of CogView3 hints at the untapped potential of model distillation techniques in enhancing the efficiency of generative models without compromising quality. Further research could explore more sophisticated distillation strategies that optimize performance and computational cost.
• Refinement of Generative Processes: The two-stage process of CogView3, combining base generation with subsequent refinement, offers a blueprint for future generative models. This approach could stimulate the development of models that are not only more efficient but also capable of self-correction and iterative improvement.

Conclusion

CogView3 represents a significant advancement in the field of text-to-image generation, primarily through its innovative use of relay diffusion. By addressing the twin challenges of computational efficiency and image detail refinement, CogView3 sets a new benchmark for future developments in generative models. Its remarkable performance, verified through rigorous benchmarks and human evaluations, attests to the potential of relay diffusion as a cornerstone technique in the ongoing evolution of generative AI.