NormalFlow: Fast, Robust, and Accurate Contact-based Object 6DoF Pose Tracking with Vision-based Tactile Sensors

Abstract: Tactile sensing is crucial for robots aiming to achieve human-level dexterity. Among tactile-dependent skills, tactile-based object tracking serves as the cornerstone for many tasks, including manipulation, in-hand manipulation, and 3D reconstruction. In this work, we introduce NormalFlow, a fast, robust, and real-time tactile-based 6DoF tracking algorithm. Leveraging the precise surface normal estimation of vision-based tactile sensors, NormalFlow determines object movements by minimizing discrepancies between the tactile-derived surface normals. Our results show that NormalFlow consistently outperforms competitive baselines and can track low-texture objects like table surfaces. For long-horizon tracking, we demonstrate when rolling the sensor around a bead for 360 degrees, NormalFlow maintains a rotational tracking error of 2.5 degrees. Additionally, we present state-of-the-art tactile-based 3D reconstruction results, showcasing the high accuracy of NormalFlow. We believe NormalFlow unlocks new possibilities for high-precision perception and manipulation tasks that involve interacting with objects using hands. The video demo, code, and dataset are available on our website: https://joehjhuang.github.io/normalflow.

• The paper introduces NormalFlow, an algorithm using surface normal maps from tactile sensors and Gauss-Newton optimization for real-time 6DoF pose tracking, unlike prior point cloud methods.
• Experiments show NormalFlow achieves 0.29 mm translation and 1.9 degrees rotation error at 70Hz, outperforming point cloud registration.
• NormalFlow is robust across various tactile sensors and opens new possibilities for high-fidelity robotic manipulation tasks.

Overview of NormalFlow: Tactile-based 6DoF Pose Tracking

This essay evaluates the "NormalFlow" algorithm introduced by Hung-Jui Huang, Michael Kaess, and Wenzhen Yuan. The paper proposes a tactile-based 6DoF pose tracking algorithm leveraging vision-based tactile sensors. Tackling occlusion issues that commonly impede vision-based methods, NormalFlow emphasizes contact-based object tracking to enhance robotic perception and manipulation. Unlike precedents relying on point cloud registration, NormalFlow directly uses surface normal maps derived from high-resolution tactile images to achieve real-time tracking, offering robustness and accuracy even in challenging scenarios like low-texture object tracking.

Key Contributions

The key innovation of the NormalFlow algorithm lies in its departure from utilizing point clouds. Instead, it minimizes discrepancies between surface normal maps, derived using photometric stereo methods, within the contact region. This shift not only circumvents inaccuracies induced by noisy integrations necessary for point cloud generation but also achieves superior tracking performance for a diverse array of objects.

1. Methodology: Inspired by the Lucas-Kanade optical flow method, NormalFlow employs Gauss-Newton optimization to determine the transformation between frames. This minimizes angular discrepancies and forms the basis of 6DoF pose estimation without needing extensive priors or object models.
2. Experimental Results: The efficacy of NormalFlow is validated against traditional point cloud registration methods such as ICP and probabilistic approaches like FilterReg. NormalFlow consistently exhibits higher accuracy and maintains real-time performance at 70Hz on a CPU, with an average translation error of 0.29 mm and a rotation error of 1.9 degrees across various tests. Notably, its ability to track low-texture objects and achieve a rotational tracking error of merely 2.5 degrees over long-distance paths demands attention.
3. Robustness: The algorithm demonstrates adaptability across various tactile sensors—such as the GelSight Mini and DIGIT—showcasing reduced computational demands by substituting height maps for direct normal comparisons. This choice significantly mitigates error accumulation typically engendered in point cloud scenarios.

Implications and Future Directions

NormalFlow potentially reshapes tactile-based robotic perception, introducing fresh approaches for high-fidelity control strategies in manipulation tasks. The method's proficiency in tracking elaborate surface textures enhances its applicability in precise tactile tasks, including complex in-hand manipulations and intricate surface reconstructions.

While the algorithm shows resilience to rapid rotations, its sensitivity to quick translational movements presents avenues for enhancing stability through hybrid approaches, possibly integrating the translation component via auxiliary ICP initialization. Further, by incorporating factor graph optimization, drifts encountered in long trajectory sequences can be mitigated, aligning with more intricate robotic systems' performance demands.

The method marks a pivotal inquiry into tactile sensor integration within robotic operations, potentially heralding advancements in real-time interaction frameworks. Given its scalability and generalization capabilities across different tactile sensing technologies and resolutions, the contributions from NormalFlow set a foundational benchmark for tactile-aware robotics, urging ongoing exploration into sensor-based perceptual algorithms.

Conclusively, NormalFlow's strategic deviation towards leveraging surface normal maps not only enhances tracking efficiency but also signifies a methodological advancement in tactile sensor utilization, broadening the horizon for precision-centric robotic interventions and tactile-based 3D reconstructions.