BeliefNest: A Joint Action Simulator for Embodied Agents with Theory of Mind

Abstract: This paper introduces an open-source simulator, BeliefNest, designed to enable embodied agents to perform collaborative tasks by leveraging Theory of Mind. BeliefNest dynamically and hierarchically constructs simulators within a Minecraft environment, allowing agents to explicitly represent nested belief states about themselves and others. This enables agent control in open-domain tasks that require Theory of Mind reasoning. The simulator provides a prompt generation mechanism based on each belief state, facilitating the design and evaluation of methods for agent control utilizing LLMs. We demonstrate through experiments that agents can infer others' beliefs and predict their belief-based actions in false-belief tasks.