Sonata: Self-Supervised Learning of Reliable Point Representations

Abstract: In this paper, we question whether we have a reliable self-supervised point cloud model that can be used for diverse 3D tasks via simple linear probing, even with limited data and minimal computation. We find that existing 3D self-supervised learning approaches fall short when evaluated on representation quality through linear probing. We hypothesize that this is due to what we term the "geometric shortcut", which causes representations to collapse to low-level spatial features. This challenge is unique to 3D and arises from the sparse nature of point cloud data. We address it through two key strategies: obscuring spatial information and enhancing the reliance on input features, ultimately composing a Sonata of 140k point clouds through self-distillation. Sonata is simple and intuitive, yet its learned representations are strong and reliable: zero-shot visualizations demonstrate semantic grouping, alongside strong spatial reasoning through nearest-neighbor relationships. Sonata demonstrates exceptional parameter and data efficiency, tripling linear probing accuracy (from 21.8% to 72.5%) on ScanNet and nearly doubling performance with only 1% of the data compared to previous approaches. Full fine-tuning further advances SOTA across both 3D indoor and outdoor perception tasks.

Overview of Sonata: Self-Supervised Learning of Reliable Point Representations

Sonata presents a self-supervised learning framework for point cloud data, addressing the particular challenges faced in 3D representation learning where traditional 2D methodologies fall short. Unlike images, where well-developed self-supervised learning (SSL) models can achieve near-supervised performance with linear probing, existing point cloud SSL approaches suffer from collapse into low-level geometric shortcuts, limiting representation quality.

The authors identify the geometric shortcut as the key obstacle, whereby models tend to latch onto trivial spatial cues inherent in 3D data, such as surface normals or point heights. This undermines the efficacy of learned representations, particularly when assessed through linear probing—a fundamental criterion in SSL which measures the semantic depth encoded by representations through minimal trainable layers.

Sonata seeks to overcome this limitation using a point self-distillation methodology that emphasizes obscuring spatial information and scaling-up task complexity. The framework generates multiple views of input point clouds, leveraging random spatial cropping, photometric augmentations, and masking strategies to challenge the model with increasingly complex tasks. An exponential moving average mechanism stabilizes training, allowing the student model to progress deeply into semantic learning without collapsing into shortcuts.

The removal of hierarchical decoders traditionally used in U-Net structures, and focusing solely on the encoder, forms a cornerstone of Sonata’s design principle. This decision alleviates the undue influence of spatial details at finer resolutions, paving the way for richer multi-scale context aggregation. Additionally, the implementation of up-casting—akin to hypercolumn mappings used in image segmentation—supplements feature depth, while progressive parameter scheduling mitigates shortcut reliance, adapting challenges as the model maturates.

This robust approach allows Sonata to scale significantly across diverse datasets, achieving a size 86 times that of PointContrast. The data scale leverages both real and simulated environments, fostering a heterogeneous learning backdrop conducive to generalization.

In empirical results, Sonata significantly enhances linear probing benchmarks, achieving an accuracy leap from 21.8% to 72.5% mIoU on ScanNet semantic segmentation tasks. Furthermore, it outperforms DINOv2, a prominent image-based self-supervised model, in raw 3D probing by capturing spatial semantics either overlooked or underrepresented in 2D paradigms. Qualitative analyses, such as PCA and dense matching visualizations, reveal Sonata’s depth in zero-shot semantic discrimination across complex indoor environments, showcasing an intuitive grasp of scene-level layouts.

Besides addressing the deficiencies in current methodologies, Sonata sets new records across varied 3D perception tasks—both indoors and outdoors—even when processed under limited data conditions. This efficiency arises from both its point self-distillation schema and feature aggregation capabilities, demonstrating Sonata's pivotal role in harnessing reliable and rich 3D point representations.

Ultimately, Sonata challenges researchers to consider novel pathways for 3D SSL, guiding future endeavors in deploying cross-modal and cross-domain strategies. Future research can build upon these insights, unifying indoor-outdoor mapping scenarios, leveraging video data elevation techniques, and exploring cross-modal distillation techniques to enrich Sonata's framework further. The potential for enhancing geometric priors is apparent in surface reconstruction validations, wherein Sonata features effectively capture scene intricacies from sparse point clouds.

In conclusion, Sonata delivers reliable self-supervised representations by circumventing the geometric shortcuts arising from 3D data's sparse nature. It exemplifies a scalable, multi-scale approach, offering profound implications for semantic and instance-level tasks across diverse applications in AI and beyond.