NTU4DRadLM: 4D Radar-centric Multi-Modal Dataset for Localization and Mapping

Abstract: Simultaneous Localization and Mapping (SLAM) is moving towards a robust perception age. However, LiDAR- and visual- SLAM may easily fail in adverse conditions (rain, snow, smoke and fog, etc.). In comparison, SLAM based on 4D Radar, thermal camera and IMU can work robustly. But only a few literature can be found. A major reason is the lack of related datasets, which seriously hinders the research. Even though some datasets are proposed based on 4D radar in past four years, they are mainly designed for object detection, rather than SLAM. Furthermore, they normally do not include thermal camera. Therefore, in this paper, NTU4DRadLM is presented to meet this requirement. The main characteristics are: 1) It is the only dataset that simultaneously includes all 6 sensors: 4D radar, thermal camera, IMU, 3D LiDAR, visual camera and RTK GPS. 2) Specifically designed for SLAM tasks, which provides fine-tuned ground truth odometry and intentionally formulated loop closures. 3) Considered both low-speed robot platform and fast-speed unmanned vehicle platform. 4) Covered structured, unstructured and semi-structured environments. 5) Considered both middle- and large- scale outdoor environments, i.e., the 6 trajectories range from 246m to 6.95km. 6) Comprehensively evaluated three types of SLAM algorithms. Totally, the dataset is around 17.6km, 85mins, 50GB and it will be accessible from this link: https://github.com/junzhang2016/NTU4DRadLM

• The paper introduces NTU4DRadLM, a novel dataset integrating six sensor modalities to enhance SLAM accuracy in diverse and adverse conditions.
• It offers meticulously calibrated ground truth odometry across both low-speed robotic and high-speed vehicular scenarios to support effective loop closures.
• Methodological evaluations using pure 4D radar, radar-IMU, and radar-thermal SLAM approaches show significant improvements in handling environmental challenges.

Overview of the NTU4DRadLM Dataset Paper

The paper presents a novel dataset named NTU4DRadLM, specifically designed for Simultaneous Localization and Mapping (SLAM) research using a comprehensive set of sensors. The dataset integrates 4D radar, a thermal camera, an inertial measurement unit (IMU), 3D LiDAR, a visual camera, and RTK GPS, offering a rich data fusion environment for developing robust SLAM systems under adverse environmental conditions. The authors emphasize that existing datasets primarily focus on either object detection or lack the incorporation of diverse sensory data suited for SLAM challenges in varying conditions.

Dataset Characteristics

The NTU4DRadLM dataset distinguishes itself through several critical attributes:

1. Sensor Suite Integration: The dataset is unique in its simultaneous inclusion of six sensory modalities—4D radar, thermal imaging, inertial measurements, 3D LiDAR scanning, visual imaging, and precise GPS. This combination offers a robust testbed for SLAM algorithms designed to operate reliably in the presence of environmental adversities such as rain, fog, and snow.
2. Task Design and Ground Truth: Primarily tailored for SLAM, the dataset provides meticulously calibrated ground truth odometry. This feature is geared towards facilitating accurate loop closures, critical to graph-based SLAM approaches that require robust optimization against odometry drift.
3. Platform Versatility: Data collection spanned both low-speed robotic environments and high-speed vehicular scenarios. This choice ensures the dataset's applicability to a wide range of SLAM applications, from indoor robotics to autonomous vehicles.
4. Environmental Diversity: The dataset encompasses structured, semi-structured, and unstructured settings, offering varying levels of complexity and challenges to SLAM systems. It covers a broad spectrum of operational scenarios, enhancing the dataset's utility in developing adaptable SLAM solutions.
5. Scale and Scope: Trajectories within the dataset vary significantly, from short 246-meter paths to extensive routes nearing 7 kilometers, facilitating research on scalability and performance over diverse distances.

Methodological Evaluation

The authors conducted evaluations using three categories of SLAM algorithms to validate the dataset's applicability:

• Pure 4D Radar SLAM: The authors utilized a point cloud registration technique without loop closures for baseline comparisons as well as an enhanced version leveraging loop closure constraints.
• 4D Radar-IMU SLAM: By adapting the Fast LiDAR-Inertial Odometry method, the research explores point feature-based odometry using 4D radar data.
• 4D Radar-Thermal SLAM: This approach projects radar data onto thermal images to perform odometry calculations using thermally-driven features, underscoring the potential of multi-sensory fusion in SLAM tasks.

Quantitative and qualitative evaluations substantiate the dataset's potential to foster advancements in SLAM technologies, with loop-closure-involved methods notably reducing trajectory errors.

Implications and Future Directions

The NTU4DRadLM dataset emerges as a crucial asset for the SLAM community, addressing the dearth of comprehensive radar and thermal imaging datasets designed for robust localization and mapping. Its breadth in sensory data and environmental conditions bolsters the exploration of SLAM in less than ideal scenarios, such as poor weather conditions where traditional LiDAR and visual methods falter.

The authors' commitment to extending the dataset to include adverse weather scenarios could further enhance its applicability, pushing the envelope in developing resilient autonomous systems. This dataset sets a precedent for future research, likely prompting the community to explore further integrations and novel sensor fusion strategies, potentially improving SLAM's effectiveness and reliability across varied contexts and platforms.