6Img-to-3D: Few-Image Large-Scale Outdoor Driving Scene Reconstruction

Abstract: Current 3D reconstruction techniques struggle to infer unbounded scenes from a few images faithfully. Specifically, existing methods have high computational demands, require detailed pose information, and cannot reconstruct occluded regions reliably. We introduce 6Img-to-3D, an efficient, scalable transformer-based encoder-renderer method for single-shot image to 3D reconstruction. Our method outputs a 3D-consistent parameterized triplane from only six outward-facing input images for large-scale, unbounded outdoor driving scenarios. We take a step towards resolving existing shortcomings by combining contracted custom cross- and self-attention mechanisms for triplane parameterization, differentiable volume rendering, scene contraction, and image feature projection. We showcase that six surround-view vehicle images from a single timestamp without global pose information are enough to reconstruct 360$^{\circ}$ scenes during inference time, taking 395 ms. Our method allows, for example, rendering third-person images and birds-eye views. Our code is available at https://github.com/continental/6Img-to-3D, and more examples can be found at our website here https://6Img-to-3D.GitHub.io/.

• The paper demonstrates that transformer-based triplane parameterization enables efficient 3D scene inference from sparse images.
• It employs a differentiable renderer with tailored cross- and self-attention mechanisms to achieve high-fidelity reconstructions in real time.
• Results show improved metrics such as PSNR and LPIPS, highlighting robust performance in large-scale outdoor driving environments.

Exploring the Efficacy of 6Img-to-3D for Large-Scale 3D Scene Reconstruction in Autonomous Driving

Introduction

The recently studied 6Img-to-3D presents a noteworthy contribution to the domain of 3D scene reconstruction from a limited number of images. This approach employs a transformer-based encoder-renderer architecture to achieve single-shot 3D scene inference from sparse multi-view image settings, specifically tailored for large-scale unbounded driving scenarios. The model circumvents the reliance on high computational resources and intricate pre-processing by integrating tailored cross- and self-attention mechanisms with an efficient rendering technique.

Methodology

The approach detailed in the paper introduces several innovative components critical for its success:

1. Triplane Parameterization:
   • Utilizing a transformer architecture, the method leverages contracted cross- and self-attention mechanisms to parameterize a triplane. This triplane aids in the condensed representation of 3D scenes by discretizing the continuous spatial domain into three orthogonal planes, enhancing both memory efficiency and rendering speed.
2. Differentiable Renderer with Scene Understanding:
   • The renderer uses a differentiable volume rendering technique that factors in scene contraction and projections of image features. This setup allows the network to infer dense 3D representations from the parameterized triplanes effectively.
3. Efficiency and Self-Supervised Learning:
   • The model stands out in its operational efficiency, able to run on a single 42GB GPU, which includes training using synthetic datasets. Self-supervised learning paradigms allow the system to adapt to unseen scenarios effectively without additional pose information, validating the model's generalization capability over new datasets.

Results

Experimental results reveal that 6Img-to-3D significantly outperforms traditional approaches that require denser viewpoint sampling or more computational resources. The reported inference time of 395ms to parameterize the triplane and the flexibility in rendering novel viewpoints are particularly compelling for real-time applications. The model achieves a competitive balance across several standard metrics, including PSNR and LPIPS, indicating a robust capability for reconstructive fidelity and perceptual quality.

Discussion

The major innovation in 6Img-to-3D lies in its ability to effectively synthesize high-fidelity 3D views from minimal input, a challenge notably pronounced in dynamic driving environments. The paper discusses several important ablations and extensions demonstrating the model's responsiveness to various operational settings and configurations. The efficient GPU training and inference underscore its potential utility in on-board vehicle systems, potentially reducing reliance on heavier computation and bringing closer the reality of fully autonomous navigation systems capable of complex environment understanding.

Future Work and Final Thoughts

Looking ahead, the possibility of extending this model's applicability to real-world scenarios presents an exciting avenue for research. Future work could also explore the integration of sensory data (like LiDAR), further enhancing depth accuracy and model robustness under varying operational conditions.

In summary, 6Img-to-3D offers a compelling new approach to 3D scene reconstruction with its efficient computational profile and promising experimental results, contributing significantly to the fields of autonomous driving and robotic vision.