Do LLM-derived graph priors improve multi-agent coordination?

Abstract: Multi-agent reinforcement learning (MARL) is crucial for AI systems that operate collaboratively in distributed and adversarial settings, particularly in multi-domain operations (MDO). A central challenge in cooperative MARL is determining how agents should coordinate: existing approaches must either hand-specify graph topology, rely on proximity-based heuristics, or learn structure entirely from environment interaction; all of which are brittle, semantically uninformed, or data-intensive. We investigate whether LLMs can generate useful coordination graph priors for MARL by using minimal natural language descriptions of agent observations to infer latent coordination patterns. These priors are integrated into MARL algorithms via graph convolutional layers within a graph neural network (GNN)-based pipeline, and evaluated on four cooperative scenarios from the Multi-Agent Particle Environment (MPE) benchmark against baselines spanning the full spectrum of coordination modeling, from independent learners to state-of-the-art graph-based methods. We further ablate across five compact open-source LLMs to assess the sensitivity of prior quality to model choice. Our results provide the first quantitative evidence that LLM-derived graph priors can enhance coordination and adaptability in dynamic multi-agent environments, and demonstrate that models as small as 1.5B parameters are sufficient for effective prior generation.

• The paper presents a novel framework that employs LLMs to generate semantic coordination graphs for MARL.
• It integrates LLM-derived graph priors into a GNN with the QMIX algorithm, achieving over 60% performance boost in adversarial scenarios.
• Empirical evaluations across four MPE scenarios demonstrate the method’s robustness and adaptability with various LLM models.

LLM-Derived Coordination Graph Priors for Multi-Agent Reinforcement Learning

Introduction and Motivation

The coordination problem in cooperative Multi-Agent Reinforcement Learning (MARL) is central to AI systems deployed in distributed, heterogeneous, and adversarial environments such as multi-domain operations (MDO), UAV swarms, warehouse robotics, and sensor networks. Existing MARL approaches for capturing agent interdependencies—ranging from independent learners to centralized and graph-based models—often struggle with brittle, semantically uninformed, or data-intensive graph construction methods. Most either hand-specify graph topologies or rely on environment-driven heuristics without exploiting high-level semantic priors about agent relationships.

This paper investigates the hypothesis that LLMs, when provided with concise natural language descriptions of agent observations, can generate informative semantic priors over agent coordination, instantiated as coordination graphs. By injecting these LLM-derived priors into a GNN-based MARL pipeline, the methodology seeks to address inherent limitations of prior graph construction approaches and enhance both the adaptability and efficiency of multi-agent learning.

Figure 1: MARL application domains and the universal challenge of coordination under partial observability; LLM-derived graph priors are proposed as data- and semantics-efficient alternatives.

Methodological Contributions

Framework Overview

The proposed framework follows a structured pipeline:

1. Observation Description: Each agent's local observation is mapped via a deterministic, hand-authored function into a concise natural language summary encapsulating spatial, role, and state information.
2. LLM-Based Graph Prior Generation: The set of agent descriptions is passed as a prompt to an open-source instruction-tuned LLM. The model outputs a weighted adjacency matrix encoding pairwise coordination affinities.
3. Post-Processing: The resultant matrix is symmetrized, normalized, and augmented with self-loops to ensure message-passing suitability.
4. GNN Integration: The prior adjacency matrix is used in a Graph Neural Network, which aggregates local agent embeddings into expressive relational representations.
5. CTDE MARL Policy Optimization: These features are consumed by the downstream value decomposition MARL algorithm (instantiated as QMIX), with the prior fixed per episode.

   Figure 2: The framework: agent observations are summarized in natural language, LLMs generate a weighted coordination graph, a GNN aggregates agent features, and standard MARL optimizes the resulting policies.

LLM Model Selection

The study focuses on models in the 0.5B–3B parameter range (Qwen2.5, LLaMA-3, Gemma-2), assessing performance sensitivity to parameter count and model family. By containing the LLM invocation cost to episodic resets, the computational and wall-clock overhead remains practical even for high-throughput MARL scenarios.

Integration with GNN-MARL

The LLM-generated priors serve as persistent graph inductive biases, operationalizing structural semantic knowledge within the end-to-end differentiable GNN-MARL pipeline. The graph structure is not treated as a dynamic or learned variable during training, but rather as a knowledge-infused scaffold.

Empirical Evaluation

Environments

Assessment occurs on four increasingly complex cooperative scenarios in the Multi-Agent Particle Environment (MPE): Speaker-Listener, Reference, Cooperative Push, and Adversary—all requiring partial observability and role coordination. The environments probe the limits of implicit and explicit coordination under diverse team and adversary settings.

Baselines

Comparisons span:

• Independent Q-Learning (IQL)
• Value Decomposition Networks (VDN)
• QMIX (monotonic value decomposition)
• Deep Coordination Graphs (DCG, fixed graphs)
• Deep Implicit Coordination Graphs (DICG, soft attention-weighted graphs)

The LLM-prior method is pitted against these baselines, isolating the impact of semantic structural knowledge infusion.

Main Results and Numerical Highlights

• Across scenarios, the LLM-prior framework achieves the highest mean returns in three of four environments, outperforming state-of-the-art learned graph models.
• In the Adversary scenario, LLM priors provide a >60% performance boost over the strongest baseline, demonstrating the value of semantically-informed coordination under adversarial pressure.
• Even the smallest LLMs (e.g., Qwen2.5-0.5B) outperform non-graph methods. Qwen2.5-1.5B yields the best results overall, indicating diminishing returns with larger models at this coordination reasoning scale.
• Cross-family comparisons (Gemma-2, LLaMA-3) show robust performance, highlighting the generality of the approach.

Theoretical and Practical Implications

On the Power of Semantic Priors

LLMs encode a strategic and organizational prior rarely present in either simplistic or fully data-driven graph construction—reasoning over roles, intent, and task structure directly from language. The study quantifies for the first time that such knowledge, when operationalized as coordination priors, enhances coordination and accelerates the formation of effective joint policies.

Application Domains

Immediate beneficiary sectors include:

• MDO: Role-based asset coordination and adversary-aware maneuver planning.
• Robotics & Warehousing: Task and workflow-dependent collaboration, transcending spatial heuristics.
• UAV Swarms & Sensor Networks: Formation, delegation, and groupwise action partitioning.
• Human-Robot Teams: LLMs leverage institutional or doctrinal knowledge, assisting mixed-initiative teams.

Limitations and Scope

LLM-derived priors are not universally needed—environments with static structure or where spatial proximity suffices may not benefit from LLM inference overhead. The method does require hand-crafting natural language observation mappings and assumes the availability of sufficiently instructable open-source LLMs.

Future Directions

Extending the framework towards:

• Automated observation captioning, removing human engineering from the pipeline.
• Scalability via hierarchical prompting or modular graph construction.
• Integration with actor-critic methods and continuous action spaces.
• Deployment in real-world MDO/simulation settings and larger teams.

Conclusion

This work establishes that compact LLMs can be leveraged to inject effective coordination graph priors into MARL, bypassing the limitations of heuristic or environment-learnt structures. Integration with GNN-based pipelines yields measurable coordination and performance gains under partial observability and in the presence of adversaries. The results empirically demonstrate a new role for LLMs—not as policy generators, but as structural prior generators—bridging linguistic knowledge and symbolic multi-agent reasoning. This advances the frontier of semantics-informed cooperative AI, suggesting further convergence of LLMs and coordinated autonomous systems.