Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planning

Abstract: Navigation in unfamiliar environments presents a major challenge for robots: while mapping and planning techniques can be used to build up a representation of the world, quickly discovering a path to a desired goal in unfamiliar settings with such methods often requires lengthy mapping and exploration. Humans can rapidly navigate new environments, particularly indoor environments that are laid out logically, by leveraging semantics -- e.g., a kitchen often adjoins a living room, an exit sign indicates the way out, and so forth. LLMs can provide robots with such knowledge, but directly using LLMs to instruct a robot how to reach some destination can also be impractical: while LLMs might produce a narrative about how to reach some goal, because they are not grounded in real-world observations, this narrative might be arbitrarily wrong. Therefore, in this paper we study how the ``semantic guesswork'' produced by LLMs can be utilized as a guiding heuristic for planning algorithms. Our method, Language Frontier Guide (LFG), uses the LLM to bias exploration of novel real-world environments by incorporating the semantic knowledge stored in LLMs as a search heuristic for planning with either topological or metric maps. We evaluate LFG in challenging real-world environments and simulated benchmarks, outperforming uninformed exploration and other ways of using LLMs.