VGG-T$^3$: Offline Feed-Forward 3D Reconstruction at Scale

Abstract: We present a scalable 3D reconstruction model that addresses a critical limitation in offline feed-forward methods: their computational and memory requirements grow quadratically w.r.t. the number of input images. Our approach is built on the key insight that this bottleneck stems from the varying-length Key-Value (KV) space representation of scene geometry, which we distill into a fixed-size Multi-Layer Perceptron (MLP) via test-time training. VGG-T$^3$ (Visual Geometry Grounded Test Time Training) scales linearly w.r.t. the number of input views, similar to online models, and reconstructs a $1k$ image collection in just $54$ seconds, achieving a $11.6\times$ speed-up over baselines that rely on softmax attention. Since our method retains global scene aggregation capability, our point map reconstruction error outperforming other linear-time methods by large margins. Finally, we demonstrate visual localization capabilities of our model by querying the scene representation with unseen images.

• The paper introduces a method that replaces quadratic global attention with a test-time optimized MLP for a fixed-size scene representation.
• The approach achieves linear scaling by compressing global KV spaces, delivering competitive geometric accuracy with significant throughput gains.
• Empirical results indicate up to 33× speedup and robust pointmap, depth, and visual localization performance, with a noted trade-off in camera pose precision.

Scalable Feed-Forward 3D Reconstruction via Test-Time Linearization

Introduction and Motivation

"VGG-T$^3$: Offline Feed-Forward 3D Reconstruction at Scale" (2602.23361) addresses the quadratic computational bottleneck of state-of-the-art feed-forward multi-view 3D reconstruction models such as VGGT, which rely on scene-level memory stored in variable-length Key-Value (KV) spaces queried via global softmax attention. While these architectures achieve robust 3D reconstruction from unposed, in-the-wild image collections, their complexity precludes practical deployment on large-scale scenes. The VGG-T$^3$ method transforms this paradigm by compressing the global KV representation into a fixed-size MLP via test-time training (TTT), yielding linear scaling with respect to the number of input views and enabling massive throughput improvements without sacrificing global feature aggregation.

Figure 1: VGGT architecture overview illustrating the alternating image-wise and global self-attention blocks for multi-view feature aggregation.

Architecture and Methodology

The approach builds upon a pre-trained multi-view feed-forward transformer (VGGT). The model tokenizes all input images and processes them through alternating local and global attention blocks. Critically, in VGG-T$^3$, every global attention layer is replaced by a test-time trained MLP that compresses the KV mapping into a fixed-dimensional implicit scene representation. This is achieved by minimizing a token-space reconstruction loss, optimizing MLP weights $\theta$ such that $\text{T}_{\theta}(k_i) \approx v_i$ for all tokens. Once optimized, the global scene can be efficiently queried by applying the MLP to the query tokens. This operation requires only $O(n)$ compute.

The model incorporates sequence mixing via ShortConv2D layers on the Value space to break the trivial linear relationship between Keys and Values, thereby increasing expressivity and improving convergence. The linearized architecture facilitates mini-batch computation and distributed inference, allowing large image collections to be sharded across multiple GPUs with gradient synchronization restricted to the small MLP weights. This is in contrast to quadratic attention models that require all QKV vectors to reside in device memory.

Figure 2: Sequence-length generalization analysis demonstrates effective error scaling and optimizer step selection for large input collections.

Empirical Evaluation and Results

Standard Benchmarks

VGG-T$^3$ is evaluated across standard geometric tasks: pointmap estimation, video depth prediction, and camera pose estimation. In pointmap and video depth estimation, VGG-T$^3$ achieves error rates closely matching or outperforming all linear-time baselines (TTT3R), while remaining competitive against quadratic-time baselines (VGGT, SparseVGGT, FastVGGT). The method substantially outperforms TTT3R on DTU, ETH3D, and NRGBD-D benchmarks, reducing error by 2–2.5× compared to prior linear models. On video depth benchmarks (Bonn, KITTI, Sintel), VGG-T$^3$ delivers on-par or improved depth accuracy, especially on KITTI.

Camera pose estimation reveals a notable limitation: while VGG-T$^3$ supports unordered input sets and maintains competitive performance, the accuracy lags behind ordered-sequence models and quadratic baselines. This discrepancy is attributed to heterogeneous modality structure in VGGT’s pose heads, which impedes memorization in the fixed-size MLP.

Figure 3: Qualitative comparison of reconstruction output across VGGT, TTT3R, and VGG-T$^3$, highlighting improved geometric consistency in VGG-T$^3$.

Large-Scale Scalability

On large-scale validation (7 Scenes), VGG-T$^3$ reconstructs 1,000 images in 58 seconds (11.6× speedup over VGGT), outperforming all previous linear-time models in accuracy and runtime. Distributed inference experiments confirm linear scaling with the number of GPUs and efficient processing of image collections exceeding device memory capacity.

Figure 4: Runtime versus Chamfer distance plot on 7scenes demonstrates both superior runtime and competitive reconstruction quality as image collection size increases.

Figure 5: Pointmap error analysis as a function of optimizer steps and input size confirms robust scaling and low error for moderate computational increase.

Feed-forward Visual Localization

The TTT-compressed MLP enables querying with novel images for feed-forward visual localization. On 7scenes and Wayspots, VGG-T$^3$ yields localization accuracy improvements over TTT3R by substantial margins, particularly for challenging scenes (e.g., Wayspots). Despite not relying on explicit mapping or database queries, the model achieves rotational and translational errors substantially lower than sequential alternatives.

Figure 6: Visual localization examples in 7scenes and Wayspots, contrasting ground truth and predicted camera poses/geometries.

Figure 7: In-the-wild localization: localizing a tourist image against a scene reconstructed from KITTI sequence taken seven years earlier.

Ablation Studies

Detailed ablations validate the superiority of test-time linearization over direct scratch training, T2R-style adaptation, and LoLCats-style low-rank approaches. Incorporating ShortConv2D consistently closes the residual performance gap to softmax attention-based models, confirming its utility in breaking non-expressive mappings in token space.

Figure 8: Qualitative comparison across methods for 1,000-image collections, demonstrating the geometric robustness of VGG-T$^3$.

Discussion and Implications

VGG-T$^3$ demonstrates that offline feed-forward 3D reconstruction can achieve linear scaling through post-training linearization and token-space test-time optimization. The unified architecture produces competitive pointmaps and depth estimates and unlocks efficient distributed inference and joint visual localization within a single end-to-end framework. The compression of global scene memory into an MLP challenges the prevailing quadratic complexity of transformer-based visual geometry models and bridges the gap between global aggregation and sequence scalability.

Remaining limitations include degraded camera pose estimation on wide-baseline datasets and a slight expressivity gap compared to full softmax attention models, especially in complex or heterogeneous scenes. These results prompt further research in adaptive computation allocation, expressive linear attention mechanisms, and specialized heads for pose regression in fixed-size representations.

Figure 9: Comparison of VGG-T$^3$ and VGGT reconstructions in complex extended scenes, highlighting both successful cases and failure modes.

Conclusion

VGG-T$^3$ advances scalable offline 3D reconstruction by linearizing global attention through test-time compression of KV spaces into fixed-dimensional MLPs. The proposed method achieves substantial throughput gains (up to 33× for large collections), competitive accuracy in geometric tasks, and distinctive query-ability for feed-forward localization. Empirical evidence supports the feasibility of compressing global scene memory in high-fidelity 3D geometry, opening new avenues for scaling visual transformers and bridging mapping/localization tasks. Future research should explore hybrid models capable of adaptive attention computation, enhancing robustness across the full spectrum of scene complexities and geometric modalities.