Multi$^2$: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments

Abstract: A central goal of LLM research is to build agentic systems that can plan, act, and adapt through sustained interaction with dynamic environments. While recent LLM-based agents exhibit impressive contextual reasoning, their long-horizon decision-making remains fragile, often suffering from objective drift, where goals and plans drift over extended interactions. We introduce Multi$^2$, a hierarchical multi-agent decision-making framework that explicitly decomposes agent behavior into complementary roles. A high-level agent (System 1) focuses on context-aware sub-goal generation using supervised fine-tuning (SFT), while a low-level agent (System 2) executes atomic actions through offline-to-online reinforcement learning (RL) in interactive environments. This separation enables stable long-horizon control, mitigates objective drift, and allows efficient adaptation. Across diverse interactive environments, Multi$^2$ consistently outperforms strong agentic baselines, demonstrating improved robustness and coordination in multi-turn interaction. Beyond performance, we introduce and release three hierarchical benchmark datasets, filling a long-standing gap in training and evaluating hierarchical decision-making for LLM-based agents.

• The paper introduces a hierarchical framework that decouples high-level sub-goal planning (System 1) and atomic action execution (System 2) using role-specialized LLM agents.
• It demonstrates significant improvements in performance and token efficiency across both in-distribution and out-of-distribution interactive environments.
• The approach leverages curated hierarchical datasets, combining SFT for planning and an offline-to-online RL pipeline for execution to mitigate drift and adapt robustly.

Multi$^2$: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments

Motivation and Challenges in Long-Horizon LLM-Based Agents

Multi-turn, long-horizon decision-making with LLM-based agents remains highly challenging due to persistent compounding errors, objective drift, and token inefficiency—especially in agentic settings where actions change the state of interactive environments. Previous approaches demonstrate limited robustness: as horizons lengthen, even hierarchical prompting schemes or fine-tuned models degrade, suffering alignment failures and productivity loss. The lack of formal role specialization and execution-side adaptation exacerbates these issues.

Figure 1: Challenges in long-horizon interaction on the ScienceWorld using Llama-3.1 8B backbone, comparing horizon robustness and token efficiency of ReAct, Glider, and Multi$^2$.

Multi$^2$ Framework: Hierarchical Role-Specialized Decision-Making

Multi$^2$ introduces a strictly hierarchical multi-agent framework, decoupling high-level sub-goal planning (System 1) and atomic action execution (System 2). These roles are instantiated as distinct LLM-based policies, each with a separate LoRA adapter while sharing the same LLM backbone. System 1 operates as a sub-goal generator trained via SFT on human-annotated data, prioritizing stable, context-aware semantic planning and intent preservation. System 2, by contrast, is exclusively responsible for action realization and is optimized via an offline-to-online RL pipeline. This decoupling supports explicit role specialization and targeted adaptation of execution, mitigating the bottlenecks of drift and token inefficiency.

Figure 2: Overview of Multi$^2$. System 1 and System 2 are trained on role-specific datasets, with System 2 undergoing both offline RL pretraining and online refinement for continual adaptation.

Hierarchical Dataset Construction and Training Paradigms

A major contribution is the creation and release of curated hierarchical datasets spanning ScienceWorld, ALFWorld, and TextCraft. These datasets partition experience for SFT-based sub-goal generation and RL-based action execution, enabling reproducible role alignment in both offline and online training settings.

• System 1 SFT: Cross-entropy loss is minimized over {task description, state observation} → sub-goal pairs, yielding consistent sub-goal proposals under distributional generalization.
• System 2 RL: Policy and value networks are initialized offline, minimizing a composite objective that integrates advantage-weighted imitation and critic-anchored regularization. During online RL, adaptation is governed by KL regularization to maintain close proximity to the offline reference policy, preventing mode collapse or value drift.

Experimental Analysis: Benchmarking and Ablations

Multi$^2$ demonstrates consistent and pronounced improvements over both prompt-based and fine-tuning hierarchical baselines in interactive, partially observable environments.

• Performance: Across all model scales and environment benchmarks, Multi$^2$ yields the highest pass@1 rates and score-based returns in both in-distribution (ID) and out-of-distribution (OOD) settings.
• Token Efficiency: Multi$^2$ produces substantial reductions in inference-time token usage, outperforming prompt-based methods that are heavily reliant on long demonstrations and extensive state histories.

  Figure 3: Inference-time token efficiency on ScienceWorld ID split with Llama-3.1 8B; Multi$^2$ achieves the highest performance for a given budget of tokens.

Failure analysis of baselines highlights the prevalence of invalid actions, unproductive loops, and constraint violations stemming from drift or execution misalignment. These modes are dramatically reduced by Multi$^2$’s hierarchical organization and role-specialized training.

Figure 4: Representative failure examples of baseline agents on the ScienceWorld Find-a-plant task, illustrating invalid actions and unproductive loops in non-hierarchical or non-role-specialized baselines.

Robustness to Long Horizons and Difficulty: Stratified analysis shows that whereas prompt-based agents deteriorate rapidly as task horizon increases, Multi$^2$0 sustains robust performance even on hard tasks—especially under distribution shift.

Figure 5: Objective drift robustness on the ScienceWorld ID split with Llama-3.1 8B, stratified by task difficulty; Multi$^2$1 maintains superior performance in hard/long-horizon scenarios.

Ablations: Component analyses confirm that joint use of role-specific LoRA adapters, role-specialized training assignments (SFT for System 1, RL for System 2), and both offline and online RL phases are critical. Removing any of these elements results in significant degradation.

Figure 6: Ablation results for training configurations, model structure, and adapter type on ScienceWorld, demonstrating necessity of hierarchical, role-specialized organization.

Loss design studies reveal that the policy-anchored advantage term and KL regularization in System 2 training yield tighter, less brittle performance distributions and are essential for stable and transferable adaptation.

Figure 7: Ablation results for offline and online RL loss functions, showing that KL regularization and policy anchoring are necessary for stable performance.

Model Scale: Performance on OOD splits increases with model capacity, highlighting the importance of backbone scale in supporting generalization under distributional mismatch.

Figure 8: Effect of backbone model scale on Multi$^2$2 performance in ScienceWorld with Qwen-2.5 backbones; OOD gains are monotonic with respect to scale.

Online RL Adaptation and Continual Self-Improvement

Online RL updates contribute further improvement, especially in the OOD setting and for medium/hard tasks, by correcting subtle execution errors and reducing compounding failures accumulated during multi-turn interaction. The explicit regularization in objective design ensures sample-efficient and stable adaptation without mode collapse.

Figure 9: Online RL adaptation on ScienceWorld with Qwen-2.5 3B; within-task adaptation yields further performance improvements, particularly OOD.

Theoretical and Practical Implications

Multi$^2$3 establishes an operational template for LLM-centric decision agents in dynamic environments, offering a clear path by which hierarchical decomposition, formal role specialization, and differentiated optimization objectives yield concrete, measurable gains. The paradigm generalizes to a wide array of agentic benchmarks and supports practical engineering by enabling fast adaptation, stateful interaction, and stringent budget-constrained inference. On the theoretical front, it motivates future research on hierarchical RL for large sequence models, decomposed adaptation under distribution shift, and continual online improvement.

Conclusion

The Multi$^2$4 hierarchical role-specialized agent framework resolves key limitations of existing LLM-agent methods in long-horizon, dynamic environments by combining explicit sub-goal planning, token-efficient execution, and robust, sample-efficient RL adaptation. The combination of structured datasets, modular training regimes, and cross-domain evaluation establishes a reproducible foundation for agentic LLM research and deployment.

Reference: "Multi$^2$5: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments" (2606.03698).