tttLRM: Test-Time Training for Long Context and Autoregressive 3D Reconstruction

Abstract: We propose tttLRM, a novel large 3D reconstruction model that leverages a Test-Time Training (TTT) layer to enable long-context, autoregressive 3D reconstruction with linear computational complexity, further scaling the model's capability. Our framework efficiently compresses multiple image observations into the fast weights of the TTT layer, forming an implicit 3D representation in the latent space that can be decoded into various explicit formats, such as Gaussian Splats (GS) for downstream applications. The online learning variant of our model supports progressive 3D reconstruction and refinement from streaming observations. We demonstrate that pretraining on novel view synthesis tasks effectively transfers to explicit 3D modeling, resulting in improved reconstruction quality and faster convergence. Extensive experiments show that our method achieves superior performance in feedforward 3D Gaussian reconstruction compared to state-of-the-art approaches on both objects and scenes.

• The paper introduces a novel test-time training framework using LaCT blocks, achieving linear complexity and high-quality 3D reconstructions.
• It employs a fast-weight neural memory mechanism to progressively update latent representations, enabling real-time explicit 3D outputs like Gaussian Splatting and triplanes.
• Experimental results show superior PSNR, SSIM, and LPIPS metrics compared to state-of-the-art methods on diverse object-level and scene-level benchmarks.

Test-Time Training for Long Context and Autoregressive 3D Reconstruction: An In-Depth Essay

Overview and Motivation

"tttLRM: Test-Time Training for Long Context and Autoregressive 3D Reconstruction" (2602.20160) introduces a novel framework for efficient, scalable, and high-fidelity 3D reconstruction from multi-view image streams. The core innovation leverages Test-Time Training (TTT) with Large Chunk Test-Time Training (LaCT) blocks, enabling linear complexity with respect to token count and supporting both feedforward and autoregressive scenarios—even with extremely long input sequences. The model compresses input observations into fast weights, forming latent representations amenable to decoding into explicit 3D formats, such as 3D Gaussian Splatting (3DGS) and triplanes. This approach unifies high-quality neural rendering with real-time explicit 3D outputs, addressing critical deficits in scalability, sequence length handling, and practical rendering speed seen in prior explicit and implicit reconstruction methods.

Model Architecture and Fast-Weight Mechanism

The tttLRM architecture operates by patchifying posed input images, encoding them into tokens, and passing them through multiple LaCT blocks. Fast weights are iteratively updated during inference, functioning as a neural memory that condenses the sequence of observations into a compact latent space. Virtual tokens are then used to query the updated fast weights, and the resulting outputs are decoded into explicit 3D representations. This design is illustrated in (Figure 1).

Figure 1: tttLRM pipeline for encoding input images into fast weights and decoding queried virtual tokens to high-fidelity 3D representations.

TTT enables online adaptation and progressive updating of internal representations, facilitating autoregressive streaming reconstruction (Algorithm 1). The causal mechanism ensures incremental updates as new views arrive, akin to RNN-like hidden state evolution. Notably, the fast-weight update and apply operations scale linearly with sequence length, avoiding the quadratic bottleneck inherent to attention-based architectures.

Distributed Feedforward Training and Scalability

To address the immense token count from large numbers of high-resolution input views, tttLRM introduces sequence parallelism for distributed training and inference. Tokenized inputs are sharded across devices; fast-weight synchronization enables independent prediction of 3D representations for each shard, followed by global aggregation and loss computation. This distributed design further boosts scalability and throughput, as visualized in (Figure 2).

Figure 2: Distributed feedforward reconstruction training with token sharding, independent prediction, aggregation, and synchronized optimization across multiple GPUs.

By leveraging PyTorch DDP and fast-weight linearity, gradients are efficiently reduced and backpropagated, supporting reconstruction of extensive scenes at resolutions up to $1024 \times 1024$ pixels with sequences of up to 64 input views, and potentially scaling beyond with sufficient compute.

Flexibility: Decoding Latent Memory to Multiple 3D Formats

The architecture’s flexibility is demonstrated by decoding fast-weight latent representations into various explicit formats beyond 3DGS. By replacing virtual tokens with triplane queries and fine-tuning with appropriate rendering losses, tttLRM can reconstruct triplane-based NeRFs, enabling broader applicability of the model to downstream tasks and formats, as shown in (Figure 3).

Figure 3: Decoding tttLRM latent memory into triplane-based representations for NeRF reconstruction and corresponding renderings and depth maps.

Experimental Results and Strong Numerical Outcomes

Experiments are conducted on object-level (Objaverse, GSO) and scene-level (DL3DV-10K, Tanks & Temples) datasets, benchmarked against state-of-the-art feedforward and optimization-based methods (GS-LRM, Long-LRM, 3DGS, Mip-Splatting, Scaffold-GS). Across all settings—for varying resolutions and numbers of input views—tttLRM consistently outperforms baselines in PSNR, SSIM, and LPIPS metrics.

• Object-level: At $512 \times 512$ resolution, tttLRM is twice as fast as attention-based GS-LRM and achieves a >1 dB PSNR gain. It generalizes robustly to longer sequences even when trained only on eight views.
• Scene-level: tttLRM outperforms Long-LRM feedforward models by approximately 1 dB PSNR—even when Long-LRM employs per-input-view post-optimization. Adaptability is evidenced by competitive or superior performance relative to optimization-based methods, but at orders-of-magnitude faster runtime. One single tttLRM model is used across varied input sequence lengths, with linear acceleration via distributed inference.

Qualitative results (Figure 4) and high-resolution scene reconstructions (Figure 5) exhibit sharper, artifact-free outputs and fine-grained photorealism, with tttLRM reconstructing intricate details (hair, fur, text) from sparse or single images.

Figure 4: Qualitative comparison—tttLRM delivers higher fidelity 3DGS reconstructions than optimization-based and feedforward baselines.

Figure 5: tttLRM reconstructs fine-grained, photorealistic details from input images via high-resolution image-to-3D generation.

Pretraining and Transfer from Novel View Synthesis Models

tttLRM leverages pretrained weights from large novel view synthesis models (LVSM, TTT-LVSM) as initialization, yielding substantially accelerated convergence and higher final quality relative to training from scratch. The transfer of implicit knowledge into explicit 3D reconstruction demonstrates the inductive bias and practical benefits of pretraining, as in (Figure 6).

Figure 6: LVSM pretraining accelerates tttLRM convergence and enhances reconstruction quality compared to training from scratch.

Autoregressive Streaming and Optimization Strategies

Autoregressive streaming enables tttLRM to reconstruct scenes progressively as views arrive, with observable gains in coverage and rendering quality. Alternatives such as “Predict & Merge” offer computational efficiency, but accumulate errors and degrade performance relative to full autoregressive reconstructions. Various optimization strategies (e.g., employing Muon optimizer, depth and opacity regularization) further reduce opaque Gaussians and refine quality.

Scalability and Attention Complexity Trade-Off

A critical architectural choice is the avoidance of attention layers due to their quadratic complexity. Empirical comparisons (Figure 7) underscore that even three attention layers lag behind 24 LaCT blocks beyond 2M tokens. The linearity of tttLRM supports efficient scaling to millions of tokens (hundreds of input views) as hardware permits.

Figure 7: Runtime comparison—LaCT blocks scale linearly with sequence length, outperforming attention layers for large token counts.

Implications and Future Directions

tttLRM addresses foundational limits in scalable 3D reconstruction: feedforward models restricted by sequence length and attention complexity, and optimization-based models hindered by rendering speed and adaptability. By distilling latent neural representations into explicit 3D outputs, tttLRM closes the gap between high-quality neural rendering and real-time deployment, facilitating practical applications in computer vision, graphics, robotics, and AR/VR.

Theoretical implications include the demonstration of linear sequence modeling in high-dimensional visual domains, the efficacy of fast-weight memory for compressing long context, and the generalization from implicit neural memory to explicit 3D outputs via flexible decoding.

Practically, tttLRM enables rapid reconstruction for streaming inputs, seamless scaling with distributed compute, and generalized applicability to multiple 3D representations. Future developments may integrate more adaptive memory mechanisms, explore selective fast-weight updates and elastic regularization, and further optimize kernel implementations to reach real-time, high-resolution 3D reconstruction with arbitrary input streams.

Conclusion

tttLRM establishes a unified, scalable, and efficient framework for 3D reconstruction from multi-view images, synthesizing advanced sequence modeling, test-time adaptation, and explicit 3D decoding. Empirical results demonstrate superiority in both quality and speed compared to prior state-of-the-art baselines. The architecture's flexibility, scalability, and performance open new possibilities for practical downstream applications in AI-driven 3D modeling, rendering, and perception (2602.20160).