Track4World: Feedforward World-centric Dense 3D Tracking of All Pixels

Abstract: Estimating the 3D trajectory of every pixel from a monocular video is crucial and promising for a comprehensive understanding of the 3D dynamics of videos. Recent monocular 3D tracking works demonstrate impressive performance, but are limited to either tracking sparse points on the first frame or a slow optimization-based framework for dense tracking. In this paper, we propose a feedforward model, called Track4World, enabling an efficient holistic 3D tracking of every pixel in the world-centric coordinate system. Built on the global 3D scene representation encoded by a VGGT-style ViT, Track4World applies a novel 3D correlation scheme to simultaneously estimate the pixel-wise 2D and 3D dense flow between arbitrary frame pairs. The estimated scene flow, along with the reconstructed 3D geometry, enables subsequent efficient 3D tracking of every pixel of this video. Extensive experiments on multiple benchmarks demonstrate that our approach consistently outperforms existing methods in 2D/3D flow estimation and 3D tracking, highlighting its robustness and scalability for real-world 4D reconstruction tasks.

• The paper introduces a novel feedforward approach that decomposes dense 3D tracking into efficient 2D-to-3D scene flow estimation, bypassing heavy 3D correlations.
• It employs GRU-based iterative refinement and joint 2D/3D supervision to overcome monocular 3D reconstruction challenges with robust precision.
• Empirical results show competitive gains in scene flow, trajectory tracking, pose estimation, and computational efficiency across multiple benchmarks.

Track4World: Feedforward World-centric Dense 3D Tracking of All Pixels

Problem Formulation and Motivation

Track4World addresses the challenge of holistic, dense 3D tracking of every pixel from monocular video in a world-centric coordinate system. Existing monocular 3D tracking approaches either rely on tracking sparse points, suffer from optimization bottlenecks for dense tracking, or are restricted to computationally expensive correlation mechanisms in 3D space. The immanent problem is twofold: monocular 3D reconstruction is ill-posed given single-view observations, and associating 3D points densely across arbitrary frames compounds complexity.

Track4World capitalizes on current advancements in feedforward 3D scene representation (e.g., VGGT-style ViTs) and decomposes dense 3D tracking into efficient pairwise 2D/3D scene flow estimation between arbitrary frames. The method proposes novel architectural components for scalability, precision, and flexibility in world-centric pixel trajectory tracking.

Architectural Overview

Track4World's pipeline comprises:

• Global scene representation extraction, yielding geometric embeddings, camera-centric point clouds, and camera poses via a backbone initialized from state-of-the-art monocular geometry models.
• Sparse-to-dense scene flow decoder utilizing a 2D-to-3D correlation scheme to predict 2D/3D flow between arbitrary frame pairs, with a joint supervision mechanism harnessing abundant 2D training data to compensate for limited 3D ground-truth.
• Global trajectory fusion, aggregating pairwise flows to construct holistic world-centric 3D tracking.

  Figure 1: Track4World pipeline, illustrating input, scene representation extraction, sparse-to-dense 2D-3D flow estimation, and fusion for world-centric holistic tracking.

The approach circumvents explicit 3D spatial correlation and $k$-nearest neighbor search complexity, instead leveraging image-plane 2D correlation and geometric lifting for flow refinement.

2D-to-3D Correlation Mechanism

The core innovation is a 2D-to-3D correlation module that iteratively updates pixel-wise flows. The design operates as follows:

• At each iteration, 2D flows are refined via geometric and semantic correlation volumes, with updates managed through GRU-based operators.
• The resulting 2D flow guides the 3D flow update by interpolating global point maps at predicted target positions.
• A dedicated 3D flow head further fuses geometric features and spatial similarity correlations, leveraging trajectory priors to ensure smoothness.
• Dense flow is recovered through learned upsampling and hybrid unprojection, merging high-fidelity 2D flow with Z-axis displacement for refined 3D outputs.

This mechanism is less computationally demanding and supports robust joint supervision.

Figure 2: Comparison of correlation mechanisms, highlighting the efficiency of Track4World's direct anchoring of 3D updates to image-plane correlations.

Numerical Evaluation and Empirical Claims

Track4World achieves strong quantitative improvements in 2D/3D flow estimation, point map prediction, camera pose estimation, and long-term 3D tracking:

• Scene/Optical flow: On Kubric-3D, KITTI, and BlinkVision datasets, Track4World outperforms prior state-of-the-art in Abs Rel, $\delta<1.25$, EPE3D, and strict/relaxed accuracy metrics for both 2D and 3D flow inference.
• 3D tracking: The model surpasses SpatialTracker, DELTA, STV2, POMATO, ZeroMSF, Any4D, and V-DPM on the PointOdyssey, ADT, PStudio, and DriveTrack datasets across temporal windows up to 50 frames, in both camera and world coordinates.
• 2D tracking: Achieves best performance (AJ, OA, $\delta_{\text{vis}}^{\text{avg}$) on Kinetics, RoboTAP, and RGB-Stacking, exceeding CoTracker3 and LocoTrack.
• Geometry/pose: Empirical results demonstrate state-of-the-art point map accuracy and competitive pose estimation, both feedforward and after test-time optimization.
• Efficiency: The method attains superior runtime, memory, and parameter efficiency; alternative dense correlation approaches cause out-of-memory bottlenecks under high-resolution dense tracking scenarios.

Strong empirical claims are substantiated through ablations: the exclusion of 2D supervision collapses 3D flow performance (EPE3D jumps to 0.6511), iterative refinement is essential for precision, and hybrid 2D+Z lifting surpasses pure 2D or 3D regression (EPE3D drops from 0.2815 to 0.2056).

Qualitative Visualization

Track4World demonstrates temporally stable, geometrically consistent correspondences and flow fields across dynamic in-the-wild scenes, shown via:

Figure 3: Qualitative results illustrating flow, camera-centric tracking, and long-term world-centric trajectories.

Figure 4: Visualization of dense trajectories across frames in the unified global world coordinate system, showing spatially coherent background and dynamic foreground motion.

Theoretical and Practical Implications

Track4World's feedforward architecture and efficient pairwise flow decomposition set a foundation for scalable, real-time 4D perception in monocular video. The method enables robust object and pixel tracking indispensable for robotics, animation, and dynamic scene understanding. The 2D-3D joint supervision paradigm overcomes annotation bottlenecks and points to broader multimodal training strategies. The decoupling of camera ego-motion from object dynamics in world coordinates is critical for stable, physically plausible modeling.

Theoretically, the sparse-to-dense correlation and hierarchical lifting advance dense video trajectory inference beyond computational limitations of explicit 3D spatial affinity. The design is backbone-agnostic and can generalize across state-of-the-art geometry models.

Limitations and Future Directions

The approach's dependence on annotated 4D motion datasets limits generalization to scenes with extreme poses or topological changes. Future research should explore generative synthetic data pipelines, semi-supervised and unsupervised paradigms, and integration with video diffusion priors for further robustness. Extension to online streaming and adaptation to topological discontinuities remains an open direction.

Conclusion

Track4World delivers robust, efficient feedforward world-centric dense 3D tracking for all pixels in monocular videos. Its 2D-to-3D correlation scheme achieves scalable precision, empirically validated as state-of-the-art across flow, tracking, geometry, and pose benchmarks. Theoretical innovations diminish computational barriers and open avenues for large-scale 4D modeling, with implications for downstream robotics, vision, and dynamic scene applications (2603.02573).