Leveraging 3D Representation Alignment and RGB Pretrained Priors for LiDAR Scene Generation

Abstract: LiDAR scene synthesis is an emerging solution to scarcity in 3D data for robotic tasks such as autonomous driving. Recent approaches employ diffusion or flow matching models to generate realistic scenes, but 3D data remains limited compared to RGB datasets with millions of samples. We introduce R3DPA, the first LiDAR scene generation method to unlock image-pretrained priors for LiDAR point clouds, and leverage self-supervised 3D representations for state-of-the-art results. Specifically, we (i) align intermediate features of our generative model with self-supervised 3D features, which substantially improves generation quality; (ii) transfer knowledge from large-scale image-pretrained generative models to LiDAR generation, mitigating limited LiDAR datasets; and (iii) enable point cloud control at inference for object inpainting and scene mixing with solely an unconditional model. On the KITTI-360 benchmark R3DPA achieves state of the art performance. Code and pretrained models are available at https://github.com/valeoai/R3DPA.

• The paper leverages RGB pretrained models and 3D representation alignment to boost LiDAR scene generation fidelity.
• It introduces a joint training pipeline using VAE alignment and a transformer-based SiT, achieving a 17% improvement on key benchmarks.
• Empirical results validate enhanced controllability and realism in generated LiDAR scenes, addressing data scarcity in autonomous applications.

Leveraging 3D Representation Alignment and RGB Pretrained Priors for LiDAR Scene Generation

Introduction

The paper "Leveraging 3D Representation Alignment and RGB Pretrained Priors for LiDAR Scene Generation" (2601.07692) presents a methodology for high-fidelity LiDAR point cloud scene generation by integrating priors from large-scale RGB image generative models and leveraging self-supervised 3D representations. This approach addresses the scarcity of comprehensive LiDAR datasets, which hinders the progress of downstream applications such as autonomous driving. The method, called R3DPA, introduces a new training pipeline incorporating VAE alignment and end-to-end representation alignment, enabling both unconditional scene generation and controllable editing capabilities.

Pipeline Overview and Motivations

Traditional LiDAR scene generation pipelines use a two-stage approach: first, train a VAE on range images and freeze its weights; then, learn the generative model on the VAE's latent space. However, this method fails to capitalize on the substantial knowledge available in 2D vision foundation models. R3DPA proposes to bridge this gap by initializing the generative model with weights from a large-scale, RGB image-trained model and aligning the VAE so that its latent representations are compatible with this pretrained backbone.

This alignment is accomplished in two steps. First, the VAE is adapted to match the latent distribution expected by the pretrained generative model, using a 3D representation alignment loss between the internal model features and 3D self-supervised features. Second, both the VAE and the generative model are jointly trained end-to-end with 3D alignment supervision, further enhancing the expressiveness of the latent space.

Figure 1: Pipeline overview showing the integration of RGB-pretrained priors and 3D representation alignment within the LiDAR scene generation framework.

Methodology

Equirectangular Projection and Range Image Representation

R3DPA adopts range images as the intermediate modality for LiDAR point cloud generation. Equirectangular projection maps each point in the 3D cloud to a 2D grid based on yaw and pitch, allowing efficient processing using architectures from the 2D domain.

VAE and Generative Model Structure

A convolutional VAE with adversarial loss, following architectures such as LiDM [ran_towards_2024], is used for encoding/decoding range images. The generative component is a flow-matching (FM) model—specifically, Scalable Interpolant Transformer (SiT)—initialized with weights pretrained on ImageNet. During VAE alignment, the transformer’s patch embedding layer and the projection head are re-initialized to resolve channel mismatches, while the remaining network is frozen to preserve pretrained knowledge.

3D Representation Alignment Loss

To align internal generative representations with semantic structure in the LiDAR data, features are extracted from a 3D self-supervised backbone (e.g., ScaLR [puy2024three]) and projected onto the same 2D grid. The alignment loss, based on feature similarity across the grid, is used to supervise both the FM model and the VAE encoder in the end-to-end phase.

Figure 2: End-to-end training—jointly optimizing VAE and FM model while supervising with 3D representation alignment loss.

Empirical Effects of Alignment

The effectiveness of joint training with 3D feature guidance is visualized through PCA projections of latent codes. Latents from end-to-end alignment exhibit more separable and semantically meaningful structures compared to those trained with conventional two-stage decoupled pipelines.

Figure 3: Effect of end-to-end training on latent structure—aligned latents distinguish objects and backgrounds more distinctly.

Experimental Evaluation

Experiments on the KITTI-360 dataset demonstrate that R3DPA achieves strong improvements over previous methods across multiple metrics, including FRID, FSVD, FPVD, JSD, MMD, and FLD. Particularly, R3DPA shows at least a 17% gain in generation quality on key benchmarks. The use of point-based, range-image, and bird's-eye view metrics ensures reliability across the full spectrum of scene realism evaluation.

Qualitative results validate that R3DPA generates diverse, realistic point cloud scenes, faithfully replicating fine-grained LiDAR sampling patterns and object distributions.

Figure 4: Qualitative comparisons of generated point clouds highlight fidelity and diversity across scenes.

Figure 5: Diverse scene synthesis with detailed object and context preservation.

Ablation Studies

• RGB Priors and VAE Alignment: Simply initializing with RGB-pretrained weights offers negligible improvement; however, performing VAE alignment using the proposed method leads to consistent numerical gains in all perceptual and distributional metrics.
• 3D Alignment and End-to-End Training: Incorporating 3D feature alignment during joint optimization significantly improves alignment, as measured by the discriminative structure of latent spaces and by quantitative metrics.
• Model Architecture: Switching from UNet to the transformer-based SiT backbone enables competitive or superior results with less than half the parameter count.

Controllable Scene Editing via Representation Alignment

Thanks to 3D representation alignment, R3DPA enables test-time conditional generation without any additional retraining. By conditioning the generative process on target feature grids, the model accomplishes tasks like object inpainting—replacing occluded or erased regions with plausible objects of a desired class—and scene mixing, where features from multiple scenes are spatially stitched to compose new configurations. These capabilities demonstrate the generality and applicability of the representation-aligned pipeline.

Figure 6: Object inpainting—test-time conditioning fills masked regions using target object class features guided by 3D representation alignment.

Figure 7: Scene mixing—composed scenes inherit structural elements from multiple real inputs through conditional 3D features.

Implications and Future Directions

R3DPA introduces a scalable and modular framework linking rich 2D pretrained knowledge with 3D scene generation. The use of representation-aligned priors overcomes sample limitations in LiDAR data, enhancing generative modeling for real-world autonomous systems. Importantly, the method enables plug-and-play controllable editing for downstream robotic and simulation tasks, reducing the annotation burden for rare or hazardous scenarios.

From a theoretical perspective, this work establishes a strong precedent for cross-modality adaptation in generative modeling, highlighting the utility of self-supervised and foundation model representations in bridging 2D and 3D domains. Practically, adopting this architecture may stimulate further research into robust, semantically controllable, and data-efficient 3D generative models.

Conclusion

R3DPA advances the state of the art in LiDAR scene generation by explicitly aligning 3D representations and incorporating RGB image pretrained priors into the generative process. The proposed two-phase VAE alignment and end-to-end joint training pipeline enables both superior unconditional scene synthesis and powerful test-time controllability. These innovations pave the way for future methods that more deeply integrate foundation models across visual modalities and offer practical tools for editing complex 3D environments (2601.07692).