ENAT: Rethinking Spatial-temporal Interactions in Token-based Image Synthesis

Abstract: Recently, token-based generation have demonstrated their effectiveness in image synthesis. As a representative example, non-autoregressive Transformers (NATs) can generate decent-quality images in a few steps. NATs perform generation in a progressive manner, where the latent tokens of a resulting image are incrementally revealed. At each step, the unrevealed image regions are padded with mask tokens and inferred by NAT. In this paper, we delve into the mechanisms behind the effectiveness of NATs and uncover two important patterns that naturally emerge from NATs: Spatially (within a step), although mask and visible tokens are processed uniformly by NATs, the interactions between them are highly asymmetric. In specific, mask tokens mainly gather information for decoding, while visible tokens tend to primarily provide information, and their deep representations can be built only upon themselves. Temporally (across steps), the interactions between adjacent generation steps mostly concentrate on updating the representations of a few critical tokens, while the computation for the majority of tokens is generally repetitive. Driven by these findings, we propose EfficientNAT (ENAT), a NAT model that explicitly encourages these critical interactions inherent in NATs. At the spatial level, we disentangle the computations of visible and mask tokens by encoding visible tokens independently, while decoding mask tokens conditioned on the fully encoded visible tokens. At the temporal level, we prioritize the computation of the critical tokens at each step, while maximally reusing previously computed token representations to supplement necessary information. ENAT improves the performance of NATs notably with significantly reduced computational cost. Experiments on ImageNet-256, ImageNet-512 and MS-COCO validate the effectiveness of ENAT. Code is available at https://github.com/LeapLabTHU/ENAT.

• The paper introduces ENAT, which rethinks spatial-temporal interactions to achieve superior image quality and 24% performance gains.
• It deploys a disentangled architecture that independently processes visible tokens and reuses computations for masked tokens, cutting redundant effort.
• Empirical results on ImageNet and MS-COCO demonstrate ENAT’s capability to deliver efficient synthesis with remarkable FID score improvements.

Analysis of ENAT: Rethinking Spatial-Temporal Interactions in Token-Based Image Synthesis

The paper "ENAT: Rethinking Spatial-temporal Interactions in Token-based Image Synthesis" explores innovative advancements in the domain of token-based visual content generation, particularly through the framework of Non-Autoregressive Transformers (NATs). This study dissects the spatial and temporal dynamics underpinning NATs and establishes the Efficient Non-Autoregressive Transformer (ENAT) as a superior model for image synthesis in terms of computational efficiency and image quality.

Spatial-Temporal Interactions in NATs

The paper examines spatial interactions within the NATs, highlighting a unique asymmetry: Visible tokens focus on conveying reliable information for the masked tokens, which progressively decode unknown aspects of the image. Temporally, the study emphasizes that only critical tokens require updates in each generation step, while the majority of computations on other tokens are redundant across iterations.

ENAT: Architectural Innovations

Inspired by these findings, ENAT proposes a disentangled architecture where visible tokens are encoded independently, and masked tokens are inferred based on fully contextualized visible tokens. This methodological refinement emphasizes prioritizing the computation for visible tokens, yielding significant performance improvements.

ENAT further introduces a temporal computation reuse strategy to update only critical tokens across steps and reuses previous computations for unnecessary repetitive tasks. These modifications reduce computational costs substantially, evidencing a 24% improved performance with 1.8 times lower computational cost compared to traditional NATs.

Empirical Validation

Through extensive empirical evaluation on datasets such as ImageNet and MS-COCO, ENAT showcases substantial improvements over existing token-based models. The ENAT models achieve remarkable FID scores, indicating superior image synthesis quality with a significantly reduced computational footprint. These findings assert the model’s effectiveness in scenarios demanding high efficiency and quick generation turnaround.

Implications and Future Prospects

The implications of this research are manifold. The disentangled spatial processing within ENAT enhances our understanding of token dynamics in image synthesis, offering a framework that could be adapted to multimodal or generalized vision models. By reducing computational overhead, this approach also broadens the accessibility and applicability of token-based image synthesis technologies across various sectors, including novel applications in real-time visual generation and lightweight deployment on edge devices.

Future progress may involve scaling ENAT models using larger datasets or integrating additional adaptive inference techniques to further enhance efficiency. Additionally, applying ENAT's principles to a broader spectrum of modalities and tasks could uncover new frontiers in efficient AI-driven synthesis and representation learning.

Conclusion

The paper presents a well-founded advancement in non-autoregressive image synthesis, with ENAT standing out as a model that efficiently capitalizes on the inherent spatial-temporal interactions within token-based systems. This contributes meaningfully to both the theoretical understanding and practical deployment of efficient generative models, underscoring a promising direction for future AI research and application.