Integrate large-capacity models into real-time robot control frameworks

Identify principled ways to seamlessly incorporate large-capacity models, such as foundation models, into robotic control frameworks for real-time inference, including the design and evaluation of hierarchical learning or slow–fast control schemes that ensure responsiveness and reliability.

Background

The survey notes that foundational and large-capacity models are increasingly used in embodied AI, but integrating them into real-time control remains challenging due to latency, responsiveness, and reliability constraints.

As an explicit open problem, the authors call for principled integration strategies—potentially via hierarchical learning or slow–fast control—to enable seamless, real-time use of large models within control loops for embodied agents.

References

Open research problems and viable approaches include: 1) mixing different proportions of prior data distribution when fine-tuning on the latest data to alleviate catastrophic forgetting , 2) developing efficient prototypes from prior distributions or curricula for task inference in learning new tasks, 3) improving training stability and sample efficiency of online learning algorithms, 4) identifying principled ways to seamlessly incorporate large-capacity models into control frameworks, potentially through hierarchical learning or slow-fast control, for real-time inference.

— Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI  (2407.06886 - Liu et al., 2024) in Section 8, Challenges and Future Directions – Continual Learning