DM$^3$-Nav: Decentralized Multi-Agent Multimodal Multi-Object Semantic Navigation

Abstract: We present DM$^3$-Nav, a fully decentralized multi-agent semantic navigation system supporting multimodal open-vocabulary goal specification and multi-object missions. In our setting, decentralization implies operation without a central coordinator, global map aggregation, or shared global state at runtime. Robots operate autonomously and coordinate through ad-hoc pairwise communication, exchanging local maps, goal status, and navigation intent without synchronization. An implicit task allocation mechanism combining intent broadcasting and distance-weighted frontier selection reduces redundant exploration while preserving decentralized operation. Evaluations on HM3DSem scenes using the HM3Dv0.2 and GOAT-Bench datasets demonstrate that DM$^3$-Nav matches or exceeds centralized and shared-map baselines while eliminating single points of failure inherent in centralized architectures. Finally, we validate our approach in a real-world office environment using two mobile robots, demonstrating successful deployment relying entirely on onboard sensing and computation. A video of our real-world experiments is available online: https://drive.google.com/file/d/1QiUSCn5rIvtuTUqtuXLPgmt6S8x9-MCZ/view?usp=drive_link

• The paper introduces DM³-Nav, a decentralized system enabling independent multi-agent navigation with semantic goal matching using onboard sensing and computation.
• It integrates multimodal inputs via vision-language models and classical detectors to robustly map and navigate unknown environments.
• Empirical results demonstrate improved success rates and efficiency over centralized baselines, emphasizing the benefits of ad-hoc, pairwise coordination.

Decentralized Multi-Agent Multimodal Multi-Object Semantic Navigation: DM$^3$-Nav

Introduction and Motivation

The DM$^3$-Nav system addresses decentralized semantic navigation for multi-robot teams, supporting open-vocabulary, multimodal, multi-goal missions in unknown environments. Traditional approaches either handle multi-object, multimodal navigation in single-agent settings or restrict multi-agent coordination to categorical goals with centralized planning and global map sharing, introducing single points of failure and scalability bottlenecks. DM$^3$-Nav, by contrast, is architected for fully decentralized operation: each robot plans, perceives, and navigates independently, with coordination achieved exclusively via opportunistic, pairwise communication. Notably, the system enables real-world deployment without global state aggregation or reliance on infrastructure, using only onboard sensors and computation.

Figure 1: A multi-agent episode where two robots follow distinct paths, navigating an unexplored environment to locate heterogeneous semantic targets specified via language, images, or category.

Problem Formulation

DM$^3$-Nav defines the navigation episode as finding a set of $m$ goals $G = \{g_1, ..., g_m\}$, each specified via natural language, image, or category labels, within an unknown scene $S$ by $n$ robots starting at arbitrary positions $P = \{p_1, ..., p_n\}$. The robots are tasked with maximizing the number of goals found, minimizing navigational cost, and strictly obeying key decentralization constraints:

• No central server, global planner, or shared global state.
• Each robot maintains a local map in its own frame; coordination occurs through ad-hoc, asynchronous, pairwise communication of partial maps, goal status, and intent information when in communication range.

The primary success criteria are the number of goals found before episode termination (due to a step budget) and navigation efficiency.

Architecture and Core Methodology

Each robot employs an ego-centric, instance-aware semantic mapping and goal-matching pipeline built upon the GOAT backbone, further augmented for robust operation and multi-agent compatibility. The core modules are:

• Perception and Mapping: Robots build geometric and semantic 2D maps from their RGB-D sensors, using both classical MaskRCNN/Detic and CLIP/SuperGlue for multi-modal semantic grounding.
• Goal Matching: Matching is conducted via direct category lookup, vision-language similarity for free-form queries, and image feature matching for reference images.
• Frontier-Based Exploration: Robots select exploration frontiers based on distance-weighted coverage, factoring in peer agent positions when known.

Decentralized Coordination Mechanism

Decentralized coordination is achieved through two principal mechanisms:

1. Ad-Hoc Pairwise Communication: Upon coming within communication range, robots estimate a transformation between their local maps using ORB+RANSAC feature matching and semantic landmark alignment, merging high-confidence overlapping map regions into their respective coordinate frames.
2. Intent Broadcasting and Task Allocation: Robots communicate their current navigation intent and completed goals. Upon receiving a peer’s intent, robots avoid targeting the same goal, yielding to higher-priority agents in cases of simultaneous claim, resolved deterministically.

   Figure 2: The modular DM$^3$-Nav architecture, showing autonomous onboard perception, memory, and local planning, with decentralized coordination implemented through exchange of semantic maps, task status, and navigation intent.

Frontier Allocation

To maximize exploration efficiency, the system uses a distance-ratio-based frontier selection heuristic. Each robot independently scores candidate frontiers using the relative geodesic distances from itself and peers, favoring frontiers closest to itself and farthest from recently communicated agents. This fully distributed heuristic requires no global knowledge, no coordination rounds, and is robust to communication dropout.

Figure 3: Illustration of coverage-maximizing frontier selection under partial, local communication—showing both resolved and unresolved redundancy in exploration, harmonized as agents share updated information.

Evaluation and Empirical Results

DM$^3$0-Nav is evaluated across both synthetic (Habitat, HM3DSem, GOAT-Bench) and real-world (multi-robot office deployment) scenarios.

Single-Agent Baseline Performance: Architectural improvements over GOAT—probabilistic map fusion, robust goal region selection, and improved detection verification—produce substantial increases in both success rate and SPL, e.g., 44.0% SR and 30.7% SPL (predicted semantics) versus 26.3% SR and 17.5% SPL for the original GOAT.

Multi-Agent Comparison: Against centralized (Co-NavGPT) and semi-centralized/shared-map (MCoCo-Nav) baselines, DM$^3$1-Nav achieves a higher success rate (74.6% vs. 66.1%, 71.6%) and competitive SPL (38.2% vs. 33.1%, 38.7%) under predicted semantics, with pronounced gains under ground-truth conditions. Notably, disabling inter-agent communication in the two-robot configuration leads to a 6% absolute drop in SR, quantifying the coordination benefit.

Scaling to Multi-Object, Multi-Agent Setups: On GOAT-Bench, DM$^3$2-Nav demonstrates regular improvement in overall episode efficiency and goal completion as the number of agents increases, with a four-robot team achieving a 40% relative boost in SR compared to a single agent. The new MSPL metric is introduced to rigorously evaluate navigation efficiency in the presence of multiple agents and goals, normalizing against the optimal assignment of goals to robots.

(Figure 4 is not provided but would be suitably referenced here for real-robot deployment visualization if available.)

Empirical Validation in the Real World

DM$^3$3-Nav is deployed on two AgileX Scout Mini UGVs using only onboard sensing, computation, and ad-hoc networking. The team successfully locates all eight targets (various modalities) in a 200 m$^3$4 office space within 14 minutes, further validating real-world feasibility without infrastructure or centralized computation. Map merging visualizations confirm successful semantic alignment and coverage distribution.

Practical and Theoretical Implications

DM$^3$5-Nav systematically eliminates the architectural vulnerabilities of centralized multi-robot semantic navigation: it removes single points of failure, permits graceful degradation under network fragmentation, and maintains scalable performance as fleet size grows. The system's demonstrated ability to match or exceed centralized baselines with minimal loss in SPL—despite distributed local decisions and asynchronous communication—constitutes strong evidence that decentralized architectures are practical for real-world large-scale deployment.

From a theoretical standpoint, the introduction of the Multi-object Multi-agent SPL (MSPL) metric provides a strict, makespan-normalized generalization of SPL, properly characterizing the efficiency of decentralized exploration in multi-goal, multi-agent regimes. The work highlights an open research direction: bridging the gap between optimal centralized task allocation and robust decentralized heuristics under realistic communication constraints.

Future Directions

Potential avenues for further research include:

• Integration with lightweight perception models for real-time, resource-constrained hardware.
• Extension to heterogeneous agent teams (e.g., mixed aerial-ground assets, sensor specialization).
• Enabling temporally complex, logic-constrained mission specifications.
• Investigation of learning-based coordination strategies to approach optimality in MSPL.
• Systematic study of robustness to communication dropouts and localization drift.

Conclusion

DM$^3$6-Nav provides the first fully decentralized system for multi-agent multimodal multi-object semantic navigation, achieving performance competitive with or surpassing centralized baselines under both simulated and real-world constraints. Its modular, decentralized architecture and open-vocabulary, multi-modal mission support constitute a significant advance in scalable, robust multi-robot navigation, laying the groundwork for further progress in distributed embodied AI systems.