Recurrent Slice Networks for 3D Segmentation of Point Clouds

Abstract: Point clouds are an efficient data format for 3D data. However, existing 3D segmentation methods for point clouds either do not model local dependencies \cite{pointnet} or require added computations \cite{kd-net,pointnet2}. This work presents a novel 3D segmentation framework, RSNet\footnote{Codes are released here https://github.com/qianguih/RSNet}, to efficiently model local structures in point clouds. The key component of the RSNet is a lightweight local dependency module. It is a combination of a novel slice pooling layer, Recurrent Neural Network (RNN) layers, and a slice unpooling layer. The slice pooling layer is designed to project features of unordered points onto an ordered sequence of feature vectors so that traditional end-to-end learning algorithms (RNNs) can be applied. The performance of RSNet is validated by comprehensive experiments on the S3DIS\cite{stanford}, ScanNet\cite{scannet}, and ShapeNet \cite{shapenet} datasets. In its simplest form, RSNets surpass all previous state-of-the-art methods on these benchmarks. And comparisons against previous state-of-the-art methods \cite{pointnet, pointnet2} demonstrate the efficiency of RSNets.

• The paper introduces RSNet, which combines slice pooling, RNN layers, and slice unpooling to efficiently capture local dependencies in unordered point clouds.
• It employs an O(n) segmentation process that outperforms methods like PointNet and 3D-CNNs on datasets such as S3DIS and ScanNet.
• RSNet's lightweight architecture is ideal for real-time applications in autonomous driving, augmented reality, and robotics.

Recurrent Slice Networks for 3D Segmentation of Point Clouds

The paper introduces a novel approach named Recurrent Slice Network (RSNet) for handling the problem of 3D segmentation of point clouds. This research addresses significant challenges in modeling local dependencies within point clouds—a data format generated by many 3D capturing devices such as LiDARs and depth sensors. Traditional methods either fail to account for local dependencies effectively or introduce computationally intensive procedures. The proposed RSNet innovatively combines a slice pooling layer, recurrent neural network (RNN) layers, and a slice unpooling layer to create a lightweight and efficient local dependency module.

Methodology

The RSNet framework deals with the unordered and unstructured nature of point clouds by projecting them onto ordered sequences of feature vectors using a slice pooling layer. This crucial process arranges points into ordered slices, thereby enabling the application of RNN architectures that traditionally require ordered sequences. The RNN layers then model dependencies within these sequences, effectively capturing local contextual information. The slice unpooling layer completes the cycle by mapping the enhanced features back onto the original points.

An important aspect of RSNet is its efficiency, demonstrating an $O(n)$ time complexity with respect to the number of input points and $O(1)$ with respect to local context resolution. This efficiency addresses the computational challenges faced by other point cloud segmentation methods, which often scale linearly with increased resolution requirements.

Experimental Validation

The RSNet model was evaluated using three challenging datasets: S3DIS, ScanNet, and ShapeNet. On the S3DIS dataset, RSNet achieved state-of-the-art results, outperforming existing methods such as PointNet and 3D-CNN based approaches. This is particularly noteworthy given the rich and diverse architectural features in the dataset. The efficiency of RSNet has been further demonstrated on the ScanNet dataset, where it achieved superior performance respecting both mean IOU and mean accuracy metrics, showcasing its robustness across different data settings. For the ShapeNet dataset, RSNet's performance was competitive, reinforcing its adaptability and effectiveness in addressing synthetic data challenges typical of part segmentation tasks.

Numerical Results and Key Contributions

RSNet outperformed traditional methods such as PointNet and its derivative PointNet++, demonstrating improved mean IOUs and class accuracies across various datasets. These improvements highlight the effectiveness of the proposed lightweight local dependency module in capturing local geometric dependencies without incurring excessive computational costs. The introduction of slice pooling and unpooling strategies stands as a key contribution, offering a practical approach to apply sequence-based learning techniques like RNNs to unordered point cloud data, thus leveraging established sequence-learning advancements in 3D space segmentation.

Implications and Future Directions

The advances presented by RSNet have substantial implications for real-time 3D processing applications, particularly in areas where computational resources are limited or where on-the-fly data processing is critical, such as autonomous driving, augmented reality, and robotics. The improvement in inference speed and reduction in memory usage, as demonstrated, make RSNet particularly appealing for these practical applications. Further research could explore adaptive slicing techniques, potentially enhancing the resolution control dynamically based on context-specific needs.

Future developments could integrate advanced recurrent architectures or explore attention mechanisms that selectively enhance significant areas of point cloud data, thus making further strides towards improving the accuracy and efficiency of segmenting complex 3D environments. Additionally, incorporating multi-domain inputs or leveraging multi-view data representations in concurrence with RSNet may push the boundaries of current 3D segmentation technologies.

RSNet represents a significant stride in efficient 3D segmentation, positioning it as a valuable tool for both academic exploration and industry application.