Genie: Generative Interactive Environments

Abstract: We introduce Genie, the first generative interactive environment trained in an unsupervised manner from unlabelled Internet videos. The model can be prompted to generate an endless variety of action-controllable virtual worlds described through text, synthetic images, photographs, and even sketches. At 11B parameters, Genie can be considered a foundation world model. It is comprised of a spatiotemporal video tokenizer, an autoregressive dynamics model, and a simple and scalable latent action model. Genie enables users to act in the generated environments on a frame-by-frame basis despite training without any ground-truth action labels or other domain-specific requirements typically found in the world model literature. Further the resulting learned latent action space facilitates training agents to imitate behaviors from unseen videos, opening the path for training generalist agents of the future.

• The paper presents a novel framework combining spatiotemporal video tokenization, latent action inference, and autoregressive dynamics to achieve controllable video generation.
• The methodology utilizes ST-transformers to efficiently handle spatial and temporal self-attention, addressing the quadratic scaling limitations of traditional models.
• Experimental results demonstrate that the 11-billion-parameter Genie model outperforms conventional approaches in video fidelity and controllability across diverse datasets.

Genie: Generative Interactive Environments

The paper "Genie: Generative Interactive Environments" introduces a novel generative model framework for creating interactive and action-controllable virtual environments through unsupervised learning from unlabelled internet videos. This approach uses a substantial model architecture and explores the potential of autonomous creation of diverse and dynamic environments.

Methodology

The Genie model employs a multifaceted architecture composed of three primary components: a spatiotemporal video tokenizer, a latent action model, and an autoregressive dynamics model. The framework adapts ST-transformers for efficient video generation. The video tokenizer encodes video frames into discrete tokens, which serve as inputs for the dynamics model. A unique latent action model infers actions in an unsupervised manner, allowing the system to generate controllable video sequences without explicit action labels.

Figure 1: Genie model training: Genie takes in $T$ frames of video as input, tokenizes them into discrete tokens $z$ via the video tokenizer, and infers the latent actions $\tilde{a}$ between each frame.

The spatiotemporal transformer network is a key innovation, optimizing computational efficiency by leveraging both spatial and temporal self-attention mechanisms. This method addresses the quadratic scaling problem inherent to traditional transformers and significantly improves the model's capacity to handle complex video dynamics.

Figure 2: ST-transformer architecture. The architecture is composed of $L$ spatiotemporal blocks, each containing a spatial layer, temporal layer and feed-forward layer.

Experimental Results

The Genie model demonstrates robust performance across various datasets, notably outperforming traditional models in both video fidelity and controllability metrics such as FVD and a proprietary signal-to-noise ratio-based controllability measure. Training occurs on vast and diverse datasets, such as curated 2D platformer videos and robotics videos, enabling the model to generalize across different domains.

A significant finding is the model's ability to scale gracefully with increased parameters and batch sizes, with detailed experiments showing linear scalability. The final model, with an extensive parameter count of 11 billion, exhibits exceptional video generation quality, reaffirming the scalability and robustness of the architecture.

Applications and Implications

The Genie model extends beyond conventional video generation, paving the way for applications in simulation, gaming, and agent training. The unsupervised latent action space offers potential for developing generalist agents capable of learning from diverse video data. Additionally, its applicability to robotics and RL environments highlights its versatility in learning dynamic physical interactions without explicit action labels.

Figure 3: Controllable, consistent latent actions in Robotics: trajectories beginning from three different starting frames from our Robotics dataset.

Conclusion

Genie represents a significant contribution to generative AI, offering scalable and controllable video generation with minimal supervision. Future work may focus on improving efficiency and extending the model's capacity for creating complex, interactive environments at real-time speeds. Given the current trajectory of advancements in AI, Genie illustrates a promising direction for autonomous content creation, simulation, and interactive experience design in both virtual and real-world applications.