TOLiD: Bridging the Architecture Gap in Vision Foundation Model to LiDAR Pretraining via Token Lifting for Distillation

Abstract: Cross-modal distillation from Vision Foundation Models (VFMs) to LiDAR backbones has recently emerged as a self-supervised pretraining strategy that reduces reliance on dense point-wise annotation for 3D scene understanding. However, existing distillation pipelines typically treat the VFM as a frozen feature source and train a heterogeneous 3D backbone to match fixed image embeddings, forcing the student to bridge both the modality gap and the cross-architecture gap between dense ViT token representations and sparse 3D encoders. We propose TOLiD, a self-supervised pretraining method for LiDAR representation learning that addresses this gap by coupling a LiDAR backbone with a student Vision Transformer (ViT) initialized from a frozen VFM teacher and applying supervision over compatible patch-token representations. TOLiD converts the set of point features within each image patch frustum into a token using Frustum Pooling followed by Frustum Attention, and performs token-level distillation with visibility masking. For LiDAR-only deployment, we lift token features back to per-point representations using masked bilinear sampling to avoid patches that have limited LiDAR points. We extensively evaluate TOLiD on five heterogeneous LiDAR datasets and four cross-sensor adaptation pairs, demonstrating improved transfer with frozen backbones and lightweight heads.

• The paper introduces TOLiD, which trains a LiDAR backbone and a student ViT in a unified token space to overcome architectural mismatches in cross-modal distillation.
• It employs innovative mechanisms like Frustum Pooling and Frustum Attention to aggregate LiDAR points into patch tokens, ensuring geometry-aware feature alignment.
• Experimental results demonstrate robust improvements in mIoU and domain adaptation, advancing self-supervised 3D scene pretraining for autonomous systems.

TOLiD: Bridging Vision Foundation Model to LiDAR Pretraining via Token Lifting for Distillation

Introduction and Motivation

Cross-modal distillation from large Vision Foundation Models (VFMs) to LiDAR backbones has become a compelling paradigm for self-supervised 3D scene pretraining, significantly reducing annotation requirements for downstream tasks in robotics and autonomous driving. However, current pipelines impose a strong architectural mismatch: they distill from frozen ViT token representations into structurally heterogeneous 3D networks, thereby conflating the challenges of both modality and architectural transfer. This paper proposes TOLiD—a self-supervised framework that couples the LiDAR and ViT backbones as a composite student, implementing a unified ViT-centric representation for distillation with patch-level alignment, geometry-aware pooling, and robust visibility masking, thus directly bridging the cross-architecture gap.

Figure 1: Overview of the TOLiD method, showing LiDAR-to-patch token conversion via Frustum Pooling/Attention (pretraining) and masked bilinear sampling for feature lifting (finetuning).

Methodological Contributions

Token Lifting for Cross-Modal Distillation

TOLiD jointly trains a LiDAR backbone and a student ViT, the latter initialized from a frozen VFM, ensuring supervision and alignment in a shared patch-token feature space. The distillation no longer forces the 3D backbone to regress to incompatible ViT embeddings, but instead aggregates point features to ViT tokens through calibrated projection.

Frustum Pooling and Frustum Attention

Patch-level tokens for each image are constructed by aggregating the LiDAR point features that project into a patch’s frustum. Two mechanisms are deployed:

• Frustum Pooling (FP): Linear projection followed by mean pooling, assigning each observed patch a token embedding, while patches without supporting points get a learned empty token.
• Frustum Attention (FA): Content-aware, per-token query-key-value attention over points inside a frustum, enabling the model to emphasize semantically critical regions and suppress geometric clutter.

  Figure 2: Frustum Pooling initializes patch tokens from projected LiDAR features; at deeper ViT blocks, Frustum Attention dynamically weights frustum point contributions.

Visibility-Masked Distillation

Supervision is only applied where LiDAR geometry supports image evidence, preventing spurious teacher-student alignment in empty or occluded regions. The loss function incorporates both [CLS]—for global feature structure—and patch token alignment, using a cosine plus smooth-$\ell_1$ metric for both directionality and magnitude consistency.

Masked Bilinear Sampling and Feature Lifting

For LiDAR-only deployment, patch tokens are lifted back to the point domain via masked bilinear interpolation. Only tokens corresponding to non-empty patches contribute, with weights renormalized—this avoids semantic contamination from unsupported image regions.

Figure 3: Masked bilinear sampling prevents empty or non-informative patch tokens from influencing lifted features.

Experimental Validation

Data and Protocols

TOLiD is pretrained on up to five heterogeneous automotive datasets (nuScenes, SemanticKITTI, PandaSet-64, PandaSet-GT, Waymo Open), enabling strong sensor and environment generalization. Evaluations include linear probing, few-shot finetuning, full finetuning, and unsupervised domain adaptation under severe cross-sensor and cross-domain shifts.

Scaling and Data Diversity

TOLiD exhibits robust performance scaling across increasing pretraining data and sensor diversity. With 342k camera-LiDAR pairs, linear probing mIoU averages (across nuScenes, SemanticKITTI, PandaSet-64, PandaSet-GT) reach up to 70.9%, 60.1%, 42.0%, and 42.6% respectively, outperforming previous SOTA (ScaLR) by up to +6.7 mIoU points, especially in settings with large cross-sensor gap.

Figure 4: TOLiD’s gains over ScaLR expand with more data diversity, reflecting improved utilization of geometric variety.

Cross-Sensor Domain Adaptation and Robustness

Without ANY target self-training, TOLiD—when pretrained on all five datasets—outperforms the MuDDoS framework by 2.2 mIoU on average across major domain adaptation targets, with gains up to 5.1 mIoU (N→K). Importantly, even without pretraining on the target sensor, TOLiD generalizes strongly, signifying enhanced robustness to unseen domains.

On the Robo3D benchmark—designed to probe robustness to severe sensor corruptions and weather—TOLiD’s frozen representations yield significant improvements across all corruption types versus all prior methods (+7.9 mIoU in LP mode), with especially large gains under internal sensor disruptions like crosstalk.

Qualitative and Ablation Analysis

Qualitative comparison demonstrates more accurate and consistent per-point semantics from linear probes, relative to ScaLR.

Figure 5: Qualitative LP results highlight TOLiD’s improved semantic consistency over previous approaches on SemanticKITTI.

Ablation studies confirm that Frustum Attention yields additional benefit over simple Pooling, and that masked (vs naive) bilinear sampling is critical for contamination avoidance and robust feature lifting.

Implications and Future Directions

TOLiD’s architecture-centric approach explicitly closes the architectural gap in cross-modal knowledge distillation. By relocating the interface from direct 3D regression to a joint ViT-patch-token space, it enhances the quality, scalability, and portability of self-supervised LiDAR representations. The improved frozen backbone robustness dramatically lowers the annotation and computational barrier for domain transfer and rapid deployment in safety-critical autonomous systems.

Practically, this framework enables effective capitalisation on continually growing fleets of unlabeled multi-sensor data, and is especially suited for scalable robotics and autonomous vehicle perception stacks. Theoretical implications include the observed benefit of architecture-aware distillation strategies for heterogeneous modalities, with potential applicability to other sensor fusion problems.

Anticipated future developments include more complex multi-modal lifting (e.g. radar, event camera inputs), advanced visibility and occlusion modeling, and tightly coupled downstream tasks such as open-vocabulary 3D semantic reasoning or vision-language scene grounding.

Conclusion

TOLiD introduces a principled, unified framework for VFM-to-LiDAR pretraining, coupling the LiDAR backbone to a student ViT, and deploying geometry-aware token lifting and masked distillation. Experimental results across scaling, adaptation, and robustness tasks conclusively show that closing the architecture gap yields superior self-supervised representations, higher downstream accuracy, and dramatically improved deployment readiness in the presence of real-world sensor shifts and corruptions. This architecture-aware distillation marks an important methodological advance in scalable 3D perception for robotics and autonomous driving (2607.10762).