FusionForce: End-to-end Differentiable Neural-Symbolic Layer for Trajectory Prediction

Abstract: We propose end-to-end differentiable model that predicts robot trajectories on rough offroad terrain from camera images and/or lidar point clouds. The model integrates a learnable component that predicts robot-terrain interaction forces with a neural-symbolic layer that enforces the laws of classical mechanics and consequently improves generalization on out-of-distribution data. The neural-symbolic layer includes a differentiable physics engine that computes the robot's trajectory by querying these forces at the points of contact with the terrain. As the proposed architecture comprises substantial geometrical and physics priors, the resulting model can also be seen as a learnable physics engine conditioned on real sensor data that delivers $10^4$ trajectories per second. We argue and empirically demonstrate that this architecture reduces the sim-to-real gap and mitigates out-of-distribution sensitivity. The differentiability, in conjunction with the rapid simulation speed, makes the model well-suited for various applications including model predictive control, trajectory shooting, supervised and reinforcement learning, or SLAM.

• The paper introduces an end-to-end differentiable neural-symbolic layer that integrates a physics engine to predict complex robot trajectories.
• It employs image-conditioned terrain encoding using a Lift-Splat-Shoot framework to extract features like friction and stiffness.
• The approach simulates 10,000 trajectories per second, demonstrating real-time performance and enhanced generalization in diverse terrains.

"FusionForce: End-to-end Differentiable Neural-Symbolic Layer for Trajectory Prediction" (2502.10156)

Overview

"FusionForce" introduces a novel paradigm for predicting robot trajectories in challenging off-road environments using image data. The model leverages the robust principles of classical mechanics within a neural-symbolic framework, integrating a physics engine into an end-to-end differentiable architecture that can efficiently simulate 10,000 trajectories per second. The core innovation lies in merging data-driven approaches with symbolic reasoning, aiming to bridge the sim-to-real gap and enhance generalization across diverse terrains.

Architecture

The proposed architecture consists of a black-box component paired with a physics-aware neural symbolic layer. The image-conditioned component forecasts interaction forces between the robot and terrain, which the symbolic layer then queries to compute trajectory outcomes. This is facilitated by a differentiable physics engine, enhancing adaptability and the model's ability to backpropagate gradients for optimization. The model demonstrates reliability using a single onboard camera, simulating extensive sequences that can be utilized in model predictive control (MPC), trajectory shooting, SLAM, and other vision-based tasks.

Implementation Details

1. Terrain Prediction:
   • The system starts with a terrain encoder that generates essential environmental features from monocular images, projecting them to a virtual heightmap.
   • Geometry-aware Lift-Splat-Shoot architecture converts pixel depths into visual surface details, facilitating the extraction of terrain properties like friction and stiffness.
2. Differentiable Physics Engine:
   • The physics engine integrates forces calculated at contact points based on predicted terrain stiffness and damping properties.
   • Utilizes equations of motion dynamics, implemented via a differentiable ODE solver for efficient trajectory estimation.
   • Incorporates adaptive gradient computation to refine learning and inference procedures.
3. Learning Objectives:
   • Self-supervised learning minimizes trajectory loss, geometrical loss, and terrain loss, ensuring accuracy against ground truth lidar estimates and SLAM trajectories.

Comparison with Other Models

The study benchmarks the model against both data-driven and physics-based alternatives. Compared to black-box approaches, FusionForce exhibits reduced out-of-distribution risk and enhanced generalization due to its integrated physics layer. The method also surpasses conventional neural network models in trajectory accuracy, showing improved prediction in challenging terrains.

Computational Considerations

The computational efficiency of FusionForce is highlighted by its capacity for massive parallelization on GPUs, making it suitable for real-time deployment. A comparative analysis of CPU and GPU performances indicates significant speed-ups, reinforcing its practical feasibility for robotics applications.

Practical Applications

The model's implications extend to autonomous navigation where robots are required to traverse rough terrains. FusionForce leverages MPI-based command sampling for trajectory selection, optimizing robot paths while ensuring obstacle avoidance and terrain adaptability. It functions robustly in dynamic environments, demonstrating competence in varied tasks from control to SLAM.

Conclusion

FusionForce presents a significant step toward integrating physical intuition in data-driven machine learning models, marrying the strengths of symbolic reasoning with neural computation. By fostering robust generalization and minimizing sim-to-real disparities, it emerges as a promising tool for advanced robotics and vision-based navigation tasks. Future research directions may focus on exploring additional sensor modalities and refining terrain interaction modeling for enhanced performance across diverse robotic platforms.