Simultaneously learning dense spatio-temporal and global semantic video representations

Determine training objectives and architectures for video representation learning that simultaneously preserve dense spatio-temporal structure necessary for localization, geometry, and tracking while also capturing motion dynamics that support global scene understanding and high-level recognition.

Background

The paper focuses on self-supervised learning from videos and highlights a trade-off observed in prior work: video JEPA models excel at global understanding and dynamics but struggle with fine-grained local structure, whereas image-focused SSL methods yield strong dense features but do not directly learn temporal dynamics.

V-JEPA 2.1 is proposed to address this gap by applying a dense predictive loss to both masked and visible tokens, combined with deep self-supervision, multi-modal tokenizers, and data/model scaling. The authors frame the broader challenge of unifying dense and global capabilities as an open problem in representation learning.