DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)

Abstract: Recent LLM-driven visual agents mainly focus on solving image-based tasks, which limits their ability to understand dynamic scenes, making it far from real-life applications like guiding students in laboratory experiments and identifying their mistakes. Hence, this paper explores DoraemonGPT, a comprehensive and conceptually elegant system driven by LLMs to understand dynamic scenes. Considering the video modality better reflects the ever-changing nature of real-world scenarios, we exemplify DoraemonGPT as a video agent. Given a video with a question/task, DoraemonGPT begins by converting the input video into a symbolic memory that stores task-related attributes. This structured representation allows for spatial-temporal querying and reasoning by well-designed sub-task tools, resulting in concise intermediate results. Recognizing that LLMs have limited internal knowledge when it comes to specialized domains (e.g., analyzing the scientific principles underlying experiments), we incorporate plug-and-play tools to assess external knowledge and address tasks across different domains. Moreover, a novel LLM-driven planner based on Monte Carlo Tree Search is introduced to explore the large planning space for scheduling various tools. The planner iteratively finds feasible solutions by backpropagating the result's reward, and multiple solutions can be summarized into an improved final answer. We extensively evaluate DoraemonGPT's effectiveness on three benchmarks and several in-the-wild scenarios. The code will be released at https://github.com/z-x-yang/DoraemonGPT.

• The paper introduces a novel framework that leverages LLMs, task-related symbolic memory, and an MCTS planner to decompose dynamic video tasks.
• It demonstrates superior causal, temporal, and descriptive reasoning on benchmarks such as NEXT-QA, TVQA+, and Ref-YouTube-VOS.
• The approach partitions spatial and temporal features for efficient querying, paving the way for robust video scene understanding in complex real-world scenarios.

DoraemonGPT: Understanding Dynamic Scenes with LLMs

In the academic paper "DoraemonGPT: Toward Understanding Dynamic Scenes with LLMs," the authors propose a novel and systematic approach to handling dynamic video tasks through a LLM-driven system, DoraemonGPT. The system is designed to overcome limitations present in existing LLM-driven agents that are primarily focused on static image tasks by addressing spatial-temporal reasoning, larger planning space, and limited internal knowledge.

This work posits a structured methodology for decomposing dynamic video tasks into sub-tasks, leveraging a task-related symbolic memory controlled by a Monte Carlo Tree Search (MCTS) planner. Symbolic memory serves as an abstraction layer between the task and video data, enabling effective querying of spatial-temporal attributes crucial for video reasoning tasks. Precise memory construction involves partitioning dynamic features into space-dominant and time-dominant attributes, thus allowing for efficient access to pertinent information using symbolic languages like SQL. The incorporation of plug-and-play tools further augments the capacity of DoraemonGPT, providing the means to access external domain-specific knowledge, diverse foundation models, and a wide array of applications.

An MCTS planner underpins the decision-making process, enabling explorative search through the vast planning space inherent in dynamic scenes. This architecture allows DoraemonGPT to iteratively refine intermediate solutions and converge on a robust final solution, optimizing decisions across complex task landscapes.

Evaluations conducted across standard benchmarks, such as NEXT-QA, TVQA+, and Ref-YouTube-VOS, highlight the superiority of DoraemonGPT over recent competitors. The architecture demonstrates strong capabilities in causal, temporal, and descriptive reasoning, as well as in pixel-wise spatial-temporal segmentation tasks on videos. Notably, the system excels in unstructured, real-world scenarios where previous systems fall short, supporting more elaborate reasoning paths for anomaly detection and nuanced understanding of video content.

The introduction of a task-related symbolic memory—coupled with a tree search-inspired planner—marks a strategic shift from static to dynamic reasoning frameworks. The exploration of multiple solution paths and enhancement of decision-making through a structured memory model liken DoraemonGPT to a post-modern reasoning system. By isolating task-critical data through decoupling and structured memory, the paper provides crucial insights into managing complexity in dynamic spaces.

Despite its effective application, there are technological dependencies, such as the foundation models used in space-dominant and time-dominant scenarios. These dependencies could potentially introduce biases or inaccuracies in the processing of dynamic scenarios, akin to challenges faced generally in AI-driven systems utilizing pretrained models. Moreover, the additional computational overhead introduced by MCTS might restrict the implementation on resource-constrained systems.

Overall, DoraemonGPT opens compelling avenues for advancing video-based dynamic reasoning by harnessing the capabilities of LLMs, shaping new paradigms in autonomous and intelligent systems. Future research can explore extending this architecture's application beyond visual domains, into broader AI contexts such as open-world planning and interactive systems. As systems like DoraemonGPT evolve, they promise to progressively bridge the gap between modeled intelligence and the unpredictable intricacies of real-world dynamics.