Customizable Perturbation Synthesis for Robust SLAM Benchmarking
Abstract: Robustness is a crucial factor for the successful deployment of robots in unstructured environments, particularly in the domain of Simultaneous Localization and Mapping (SLAM). Simulation-based benchmarks have emerged as a highly scalable approach for robustness evaluation compared to real-world data collection. However, crafting a challenging and controllable noisy world with diverse perturbations remains relatively under-explored. To this end, we propose a novel, customizable pipeline for noisy data synthesis, aimed at assessing the resilience of multi-modal SLAM models against various perturbations. This pipeline incorporates customizable hardware setups, software components, and perturbed environments. In particular, we introduce comprehensive perturbation taxonomy along with a perturbation composition toolbox, allowing the transformation of clean simulations into challenging noisy environments. Utilizing the pipeline, we instantiate the Robust-SLAM benchmark, which includes diverse perturbation types, to evaluate the risk tolerance of existing advanced multi-modal SLAM models. Our extensive analysis uncovers the susceptibilities of existing SLAM models to real-world disturbance, despite their demonstrated accuracy in standard benchmarks. Our perturbation synthesis toolbox, SLAM robustness evaluation pipeline, and Robust-SLAM benchmark will be made publicly available at https://github.com/Xiaohao-Xu/SLAM-under-Perturbation/.
- Task-aware risk estimation of perception failures for autonomous vehicles. arXiv preprint arXiv:2305.01870, 2023.
- Strategies for reducing speckle noise in digital holography. Light: Science and Applications, 7(1):48, 2018.
- The málaga urban dataset: High-rate stereo and lidar in a realistic urban scenario. Int. J. Robot. Research, 33(2):207–214, 2014.
- Robust slam systems: Are we there yet? In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5320–5327. IEEE, 2021.
- The EuRoC micro aerial vehicle datasets. The International Journal of Robotics Research, 2016.
- Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age. IEEE Transactions on Robotics, 32(6):1309–1332, 2016.
- Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam. IEEE Transactions on Robotics, 37(6):1874–1890, 2021.
- University of michigan north campus long-term vision and lidar dataset. The International Journal of Robotics Research, 35(9):1023–1035, 2016.
- Modeling camera effects to improve visual learning from synthetic data. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, September 2018.
- Robustnav: Towards benchmarking robustness in embodied navigation. In IEEE/CVF International Conference on Computer Vision, pages 15691–15700, 2021.
- Deep learning for visual localization and mapping: A survey. IEEE Transactions on Neural Networks and Learning Systems, pages 1–21, 2023.
- Kaist multi-spectral day/night data set for autonomous and assisted driving. IEEE Transactions on Intelligent Transportation Systems, 19(3):934–948, 2018.
- Pearson correlation coefficient. Noise reduction in speech processing, pages 1–4, 2009.
- Scannet: Richly-annotated 3d reconstructions of indoor scenes. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5828–5839, 2017.
- Procthor: Large-scale embodied ai using procedural generation. Advances in Neural Information Processing Systems, 35:5982–5994, 2022.
- Are we ready for autonomous drone racing? the UZH-FPV drone racing dataset. In IEEE Int. Conf. Robot. Autom. (ICRA), 2019.
- Blenderproc: Reducing the reality gap with photorealistic rendering. In International Conference on Robotics: Sciene and Systems, RSS 2020, 2020.
- Benchmarking robustness of 3d object detection to common corruption. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1022–1032, 2023.
- Carla: An open urban driving simulator. In Conference on robot learning, pages 1–16. PMLR, 2017.
- Present and future of slam in extreme environments: The darpa subt challenge. IEEE Transactions on Robotics, pages 1–20, 2023.
- Are we ready for autonomous driving? The KITTI Vision Benchmark Suite. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
- Samuel W Hasinoff. Photon, poisson noise. Computer Vision, A Reference Guide, 4(16):1, 2014.
- The hilti slam challenge dataset. IEEE Robotics and Automation Letters, 7(3):7518–7525, 2022.
- Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
- Driving in the matrix: Can virtual worlds replace human-generated annotations for real world tasks? In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 746–753, 2017.
- Benchmarking the robustness of semantic segmentation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8828–8838, 2020a.
- Increasing the robustness of semantic segmentation models with painting-by-numbers. In European Conference on Computer Vision, pages 369–387. Springer, 2020b.
- Champion-level drone racing using deep reinforcement learning. Nature, 620(7976):982–987, 2023.
- A survey of state-of-the-art on visual slam. Expert Systems with Applications, 205:117734, 2022.
- Splatam: Splat, track map 3d gaussians for dense rgb-d slam. arXiv, 2023.
- 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), 2023.
- Mimosa: A multi-modal slam framework for resilient autonomy against sensor degradation. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7153–7159, 2022.
- Tae Kyun Kim. T test as a parametric statistic. Korean journal of anesthesiology, 68(6):540–546, 2015.
- Robo3d: Towards robust and reliable 3d perception against corruptions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19994–20006, 2023.
- Robust referring video object segmentation with cyclic structural consensus. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22236–22245, 2023.
- Homogeneous multi-modal feature fusion and interaction for 3d object detection. In European Conference on Computer Vision, pages 691–707, 2022.
- A comprehensive survey of visual slam algorithms. Robotics, 11(1):24, 2022.
- 1 year, 1000 km: The oxford robotcar dataset. The International Journal of Robotics Research, 36(1):3–15, 2017.
- Benchmarking robustness in object detection: Autonomous driving when winter is coming. arXiv preprint arXiv:1907.07484, 2019.
- Universal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1765–1773, 2017.
- Extreme low-light environment-driven image denoising over permanently shadowed lunar regions with a physical noise model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6317–6327, 2021.
- Orb-slam: a versatile and accurate monocular slam system. IEEE transactions on robotics, 31(5):1147–1163, 2015.
- Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras. IEEE Transactions on Robotics, 33(5):1255–1262, 2017.
- Covins-g: A generic back-end for collaborative visual-inertial slam. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 2076–2082, 2023.
- Penncosyvio: A challenging visual inertial odometry benchmark. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 3847–3854. IEEE, 2017.
- A survey on active simultaneous localization and mapping: State of the art and new frontiers. IEEE Transactions on Robotics, 39(3):1686–1705, 2023.
- Infinite photorealistic worlds using procedural generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12630–12641, 2023.
- Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821–8831. PMLR, 2021.
- High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
- Kimera: From slam to spatial perception with 3d dynamic scene graphs. The International Journal of Robotics Research, 40(12-14):1510–1546, 2021.
- Nerf-slam: Real-time dense monocular slam with neural radiance fields. arXiv preprint arXiv:2210.13641, 2022.
- Visual-inertial navigation algorithm development using photorealistic camera simulation in the loop. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 2566–2573, 2018.
- The tum vi benchmark for evaluating visual-inertial odometry. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1680–1687, 2018.
- Dave Shreiner et al. OpenGL programming guide: the official guide to learning OpenGL, versions 3.0 and 3.1. Pearson Education, 2009.
- Alvy Ray Smith. Image compositing fundamentals. Microsoft Corporation, 5, 1995.
- Charles Spearman. The proof and measurement of association between two things. 1961.
- The replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1906.05797, 2019.
- A benchmark for the evaluation of rgb-d slam systems. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 573–580, 2012.
- imap: Implicit mapping and positioning in real-time. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6229–6238, 2021.
- Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. Advances in neural information processing systems, 34:16558–16569, 2021.
- Sebastian Thrun. Probabilistic robotics. Commun. ACM, 45(3):52–57, 2002.
- Kimera-multi: Robust, distributed, dense metric-semantic slam for multi-robot systems. IEEE Transactions on Robotics, 38(4):2022–2038, 2022.
- Resilient and distributed multi-robot visual slam: Datasets, experiments, and lessons learned. arXiv preprint arXiv:2304.04362, 2023.
- Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033, 2012.
- The raincouver scene parsing benchmark for self-driving in adverse weather and at night. IEEE Robotics and Automation Letters, 2(4):2188–2193, 2017.
- Simulating photo-realistic snow and fog on existing images for enhanced cnn training and evaluation. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pages 41–46. IEEE, 2019.
- Co-slam: Joint coordinate and sparse parametric encodings for neural real-time slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13293–13302, 2023.
- Tartanair: A dataset to push the limits of visual slam. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4909–4916. IEEE, 2020.
- Tartanvo: A generalizable learning-based vo. In Conference on Robot Learning, pages 1761–1772. PMLR, 2021.
- D2 slam: Decentralized and distributed collaborative visual-inertial slam system for aerial swarm. arXiv preprint arXiv:2211.01538, 2022a.
- Towards robust video object segmentation with adaptive object calibration. In Proceedings of the 30th ACM International Conference on Multimedia, pages 2709–2718, 2022b.
- Benchmarking augmentation methods for learning robust navigation agents: the winning entry of the 2021 igibson challenge. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1748–1755. IEEE, 2022.
- Nighttime single image dehazing via pixel-wise alpha blending. IEEE Access, 7:114619–114630, 2019.
- Multi-camera lidar inertial extension to the newer college dataset. arXiv preprint arXiv:2112.08854, 2021.
- Resilient robots: Concept, review, and future directions. Robotics, 6(4):22, 2017.
- Beyond nerf underwater: Learning neural reflectance fields for true color correction of marine imagery. IEEE Robotics and Automation Letters, 8(10):6467–6474, 2023.
- Go-slam: Global optimization for consistent 3d instant reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023.
- Subt-mrs: A subterranean, multi-robot, multi-spectral and multi-degraded dataset for robust slam. arXiv preprint arXiv:2307.07607, 2023.
- Structured3d: A large photo-realistic dataset for structured 3d modeling. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IX 16, pages 519–535. Springer, 2020.
- Nice-slam: Neural implicit scalable encoding for slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12786–12796, 2022.
- The uma-vi dataset: Visual–inertial odometry in low-textured and dynamic illumination environments. The International Journal of Robotics Research, 39(9):1052–1060, 2020.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.