Composable Action-Conditioned Predictors: Flexible Off-Policy Learning for Robot Navigation

Abstract: A general-purpose intelligent robot must be able to learn autonomously and be able to accomplish multiple tasks in order to be deployed in the real world. However, standard reinforcement learning approaches learn separate task-specific policies and assume the reward function for each task is known a priori. We propose a framework that learns event cues from off-policy data, and can flexibly combine these event cues at test time to accomplish different tasks. These event cue labels are not assumed to be known a priori, but are instead labeled using learned models, such as computer vision detectors, and then `backed up' in time using an action-conditioned predictive model. We show that a simulated robotic car and a real-world RC car can gather data and train fully autonomously without any human-provided labels beyond those needed to train the detectors, and then at test-time be able to accomplish a variety of different tasks. Videos of the experiments and code can be found at https://github.com/gkahn13/CAPs

• The paper introduces Composable Action-Conditioned Predictors (CAPs) enabling flexible robot navigation by training event predictors off-policy.
• Experiments show CAPs outperforms traditional RL in flexibility and generalization on simulated and real-world navigation tasks, adapting without retraining.
• This framework is promising for versatile autonomous robots adaptable to complex environments, improving decision-making without exhaustive retraining.

Composable Action-Conditioned Predictors: Flexible Off-Policy Learning for Robot Navigation

The paper "Composable Action-Conditioned Predictors: Flexible Off-Policy Learning for Robot Navigation" addresses a critical challenge in robotic autonomous systems—enabling robots to learn and execute multiple tasks without explicit, task-specific policies. Standard reinforcement learning (RL) methodologies often suffer from the limitations of learning isolated task-specific policies and requiring predefined extrinsic rewards. This paper introduces a novel framework, Composable Action-Conditioned Predictors (CAPs), that leverages off-policy learning and automatically detected event cues to provide adaptable, intelligent navigation capabilities in both simulated and real-world contexts.

Framework Overview

CAPs advances the RL landscape by integrating multi-task learning, off-policy training, and automatic event cue detection. The core concept involves predicting future event cues based on current observations and actions. At test time, these event predictors can be subjectively combined to fulfill specific user-defined goals, allowing for dynamic task execution without prior respecification during training. The framework essentially redefines the traditional value function approach by predicting and integrating multiple events to solve tasks using learned event cues.

Methodological Contributions

The CAPs framework introduces a multi-objective control model that scales to extensive datasets and utilizes deep neural networks alongside off-policy data training. The primary contributions include:

• Event Cue Prediction: The approach conditions action sequences to predict various events, which allows the model to devise policies based on the predicted likelihood of these events.
• Autonomous Labeling: It uses contemporary computer vision systems or self-supervision methods to autonomously label cues, minimizing human intervention beyond baseline detector training.
• Action Selection: The selection of optimal actions in the CAPs framework is driven by maximizing user-defined rewards, formulated from the predicted event cues. This is performed through a model predictive control (MPC) strategy, wherein a finite horizon solution at each timestep enhances robustness.
• Off-Policy Training and Deployment: Data collected in off-policy manner—through exploration or partially on-policy strategies—streamlines the acquisition of behaviors, which can then be flexibly repurposed at test time.

Experimental Evaluation

The CAPs framework was rigorously tested in complex environments to verify its effectiveness across different tasks. The experiments included:

1. Simulated Forest and City Benchmarks: These settings evaluated the framework's ability to perform path following and multi-objective navigation. When compared with goal-conditioned deep Q-learning (GC-DQL) variants, CAPs demonstrated superior generalization and flexibility on unseen tasks.
2. Real-World Indoor Navigation with RC Car: This experiment illustrated the utility of CAPs in a physical environment, demonstrating success in complex tasks like package delivery simulations via goal-heading and collision avoidance.

The experiments highlighted that CAPs outperformed traditional RL approaches in flexibility, particularly showcasing the capability to adapt through simple reward function modifications without necessitating retraining—emphasizing its applicability in real-world robotic applications.

Implications and Future Directions

The introduction of CAPs presents significant implications for both theoretical understanding and practical deployment of autonomous systems. The abstraction of task-specific policies into a flexible predictor model allows for expansive automation capabilities in uncertain environments. As robotic applications continue to evolve, this method provides a promising direction for improving decision-making in dynamic tasks without exhaustive retraining.

Future research could focus on extending CAPs to account for uncertainty in event detection, incorporating model-based RL approaches to enhance long-horizon planning, and broadening the horizons of integration with other sensory modalities. As the field of AI and robotics advances, frameworks like CAPs will play a fundamental role in enabling versatile, autonomous robotic systems capable of highly adaptive behavior in complex environments.