DaDu-E: Rethinking the Role of Large Language Model in Robotic Computing Pipeline

Abstract: Performing complex tasks in open environments remains challenging for robots, even when using LLMs as the core planner. Many LLM-based planners are inefficient due to their large number of parameters and prone to inaccuracies because they operate in open-loop systems. We think the reason is that only applying LLMs as planners is insufficient. In this work, we propose DaDu-E, a robust closed-loop planning framework for embodied AI robots. Specifically, DaDu-E is equipped with a relatively lightweight LLM, a set of encapsulated robot skill instructions, a robust feedback system, and memory augmentation. Together, these components enable DaDu-E to (i) actively perceive and adapt to dynamic environments, (ii) optimize computational costs while maintaining high performance, and (iii) recover from execution failures using its memory and feedback mechanisms. Extensive experiments on real-world and simulated tasks show that DaDu-E achieves task success rates comparable to embodied AI robots with larger models as planners like COME-Robot, while reducing computational requirements by $6.6 \times$. Users are encouraged to explore our system at: \url{https://rlc-lab.github.io/dadu-e/}.

• The paper introduces a closed-loop planning framework that integrates real-time visual feedback to dynamically adapt robotic actions.
• The paper leverages memory augmentation to reduce redundant computations, achieving a 6.6x reduction in computational cost compared to larger LLMs.
• The paper validates DaDu-E in real and simulated environments, demonstrating robust performance in fixed domains like warehouses and grocery stores.

An Analysis of DaDu-E: Rethinking the Role of LLMs in Robotics

The paper "DaDu-E: Rethinking the Role of LLM in Robotic Computing Pipeline" examines the limitations inherent in current LLM-based planners for robotic operations, typically manifested as inefficiency and inaccuracies in open-loop systems. The authors propose DaDu-E, a compact, closed-loop planning framework for embodied AI robots, enhancing traditional LLM utilization with memory augmentation and frequent visual feedback to foster robust adaptability in dynamic environments.

Key Contributions

1. Closed-loop Planning Architecture: DaDu-E innovatively shifts from an open-loop approach to a closed-loop system that allows the planner to adjust its strategies based on real-time environmental feedback, thereby improving task success rates while maintaining computational efficiency.
2. Memory Augmentation: The integration of a memory module serves to optimize cognitive load and minimize redundant computation by storing recently used objects, harmonizing its operations akin to episodic memory in humans.
3. Domain-specific Skill Sets: By limiting the operational scope to fixed domains, such as grocery stores and warehouses, DaDu-E delivers a streamlined skill set tailored for specific environments. This pragmatic approach promotes efficiency without sacrificing planning performance.
4. Experimental Validation: The deployment of DaDu-E in both real and simulated environments signals a reduction in computational demand by a factor of 6.6 times compared to larger model counterparts like GPT-4o, without detriment to task completion rates.

Methodological Approach

The framework amalgamates three primary components into a cohesive system: structured instruction sets, real-time feedback mechanisms, and a memory augmentation module. The LLM utilized, LLaMA 3.1-8B, is significantly smaller than models typically used in robotic planning but achieves comparable performance due to these optimizations. The authors achieve efficient task decomposition through visual and textual input integration via a VLM, further refined through feedback loops that re-evaluate planning sequences.

Numerical Results and Implications

The investigations demonstrate that DaDu-E retains strong performance metrics across a spectrum of long-horizon tasks despite reduced computational resource requirements. Notably, the system maintains a task success rate on par with larger models yet curtails computational costs significantly, with savings illustrated in both parameters utilized and FLOPs executed.

Limitations and Considerations

An inherent trade-off exists between the narrow operational focus and generalizability. While DaDu-E excels in fixed scenarios, its limited scope could curtail its adaptability to unforeseen environments. Future research could extend DaDu-E's principles to more generalized settings, potentially integrating additional multi-modal data sources to enhance situational awareness and decision-making capability.

Future Directions

The paper sets the stage for future explorations into resource-efficient AI-robotic integrations by reducing dependency on large-scale cloud-server operations. A promising trajectory lies in refining closed-loop feedback systems and further minimizing LLM model sizes without performance loss. Moreover, the integration of adaptive memory systems holds potential for further enhancing decision-making speed and accuracy.

In summary, DaDu-E stands as a testament to the evolving landscape of AI in robotics, advocating for strategic enhancements over brute computational power, thereby charting a path towards more agile and resource-efficient robotic systems.