PillarMamba: Learning Local-Global Context for Roadside Point Cloud via Hybrid State Space Model

Abstract: Serving the Intelligent Transport System (ITS) and Vehicle-to-Everything (V2X) tasks, roadside perception has received increasing attention in recent years, as it can extend the perception range of connected vehicles and improve traffic safety. However, roadside point cloud oriented 3D object detection has not been effectively explored. To some extent, the key to the performance of a point cloud detector lies in the receptive field of the network and the ability to effectively utilize the scene context. The recent emergence of Mamba, based on State Space Model (SSM), has shaken up the traditional convolution and transformers that have long been the foundational building blocks, due to its efficient global receptive field. In this work, we introduce Mamba to pillar-based roadside point cloud perception and propose a framework based on Cross-stage State-space Group (CSG), called PillarMamba. It enhances the expressiveness of the network and achieves efficient computation through cross-stage feature fusion. However, due to the limitations of scan directions, state space model faces local connection disrupted and historical relationship forgotten. To address this, we propose the Hybrid State-space Block (HSB) to obtain the local-global context of roadside point cloud. Specifically, it enhances neighborhood connections through local convolution and preserves historical memory through residual attention. The proposed method outperforms the state-of-the-art methods on the popular large scale roadside benchmark: DAIR-V2X-I. The code will be released soon.

An Analysis of PillarMamba: Hybrid State Space Model for Roadside Point Cloud Perception

Roadside perception plays a critical role in the Intelligent Transport System (ITS) and Vehicle-to-Everything (V2X) applications by extending the perception capabilities of connected vehicles beyond their immediate surroundings. Traditional techniques for 3D object detection in point clouds have primarily focused on vehicle-mounted sensors, and there has been limited exploration of algorithms tailored for roadside point cloud data. This paper presents "PillarMamba," a novel framework that leverages the hybrid state space model introduced by Mamba to improve the efficiency and accuracy of 3D object detection in roadside point clouds.

Overview of PillarMamba Framework

The PillarMamba framework integrates the state space model into pillar-based roadside point cloud perception, addressing two main challenges: insufficient utilization of scene context and computational inefficiency in dense point clouds. The method enhances the traditional Mamba approach, focusing on overcoming limitations such as disrupted local connections and forgotten historical relationships in the state space equations.

Key Contributions

1. Cross-stage State-space Group (CSG): This module efficiently extracts global context from dense roadside scenes. It achieves computational efficiency by reducing channel dimensions, splitting channels, and allowing cross-stage connections between network layers. This not only accelerates inference but also enhances expressive network capabilities.

2. Hybrid State-space Block (HSB): Designed to address the limitations of standard Mamba, the HSB combines the benefits of local convolutions and residual attention. Local convolutions maintain neighborhood connections within the point cloud, while residual attention preserves historical memory, thus boosting the network’s ability to differentiate between closely situated objects and in distinguishing foreground from background amidst sparse point cloud data.

3. Selective Scan Methodology: Utilizing a four-way selective scan technique, PillarMamba captures the global context by processing sequences in a recursive manner. This technique filters out irrelevant information, ensuring that local spatial dependencies are preserved, enhancing the model’s ability to identify small objects or those positioned distantly from the scanning reference point.

Experimental Validation and Results

The research demonstrates that PillarMamba outperforms state-of-the-art methods on the DAIR-V2X-I benchmark, a dataset specifically designed for roadside point cloud perception. Notably, PillarMamba achieves higher average precision scores across vehicle, pedestrian, and cyclist categories compared to existing pillar-based methods, evidencing the effectiveness of its novel architectural components.

Implications and Future Directions

The integration of state space models into deep learning frameworks is a relatively new approach in computer vision, particularly in 3D object detection tasks involving point clouds. PillarMamba showcases promising advancements in modeling long-range dependencies within a dense roadside context, with potential applications extending into other domains requiring efficient 3D perception under similar conditions.

Future research might explore further enhancements to hybrid state space models, possibly by incorporating additional multi-modal data such as image or radar data, or by refining selective scan methodologies to further optimize computational efficiency. As roadside perception technology continues to evolve, frameworks like PillarMamba will likely play an increasingly critical role in advancing autonomous driving technologies and improving traffic safety systems through enhanced environmental estimation.