Multi-Agent Reflective Policy Optimization

• The paper introduces a trajectory reflection mechanism that leverages subsequent step data to effectively double the information used for policy updates.
• MARPO implements an adaptive, KL-divergence-matched asymmetric clipping strategy to ensure robust, data-dependent learning stability.
• Empirical results on benchmarks like SMAC-Hard and Google Research Football show MARPO outperforms standard baselines in speed, efficiency, and convergence.

Multi-Agent Reflective Policy Optimization (MARPO) is an on-policy reinforcement learning algorithm developed for cooperative multi-agent tasks in partially observable, decentralized environments. MARPO is designed to address sample inefficiency and instability characteristic of standard policy gradient methods in multi-agent reinforcement learning (MARL). It introduces a trajectory-reflection mechanism that utilizes subsequent trajectory data and an adaptive, KL-divergence-matched asymmetric clipping strategy to achieve both improved sample efficiency and training robustness (Wu et al., 28 Dec 2025).

1. Problem Setting: Decentralized Multi-Agent Cooperation

MARPO operates within the Decentralized Partially Observable Markov Decision Process (Dec-POMDP) framework. A Dec-POMDP is a tuple

$G = (\mathcal{N}, S, \mathcal{A}, P, O, r, \gamma)$

where $\mathcal{N} = \{1, \ldots, n\}$ indexes agents, $S$ is the global state space, $\mathcal{A} = A_1 \times \cdots \times A_n$ is the joint action space, $P$ specifies the transition kernel, $O$ is the observation function, $r$ is the team reward, and $\gamma$ is the discount factor. Each agent $i$ implements a local policy $\pi^i_\theta(a_i | o_i)$ parameterized by the global vector $\mathcal{N} = \{1, \ldots, n\}$0 but acting only on its observation $\mathcal{N} = \{1, \ldots, n\}$1. The objective is to maximize

$\mathcal{N} = \{1, \ldots, n\}$2

with cumulative return $\mathcal{N} = \{1, \ldots, n\}$3 for trajectory $\mathcal{N} = \{1, \ldots, n\}$4. Advantage estimation uses Generalized Advantage Estimation (GAE) for on-policy rollouts.

2. Core Algorithmic Innovations

2.1. Reflection Mechanism

Traditional PPO-style objectives use only immediate state–action pairs for policy improvement. MARPO's reflection mechanism incorporates information from the next time step $\mathcal{N} = \{1, \ldots, n\}$5. Define the per-agent per-timestep probability ratios:

$\mathcal{N} = \{1, \ldots, n\}$6

The surrogate objectives are: $\mathcal{N} = \{1, \ldots, n\}$7

$\mathcal{N} = \{1, \ldots, n\}$8

where $\mathcal{N} = \{1, \ldots, n\}$9. The total reflective surrogate is

$S$0

with $S$1 balancing one-step and two-step terms.

2.2. Asymmetric Clipping via KL Divergence

Unlike PPO's symmetric clipping $S$2, MARPO chooses clipping bounds $S$3 such that the expected surrogate

$S$4

matches a target KL divergence $S$5. The clipping interval $S$6 is derived by solving $S$7 for $S$8. The target is exponentially averaged: $S$9 This adaptive, theory-grounded trust region accommodates changing policy drift, yielding principled and data-dependent learning stability.

3. Optimization Workflow and Loss Composition

The total optimization objective is: $\mathcal{A} = A_1 \times \cdots \times A_n$0 with $\mathcal{A} = A_1 \times \cdots \times A_n$1 denoting policy entropy and $\mathcal{A} = A_1 \times \cdots \times A_n$2 its exploration bonus coefficient. The procedure for each iteration is as follows:

1. Collect on-policy rollouts under $\mathcal{A} = A_1 \times \cdots \times A_n$3.
2. Compute actual KL divergences and update $\mathcal{A} = A_1 \times \cdots \times A_n$4 with EMA.
3. Find asymmetric clipping roots for both step and next-step KLs.
4. For each mini-batch/epoch:
   • Compute surrogate losses $\mathcal{A} = A_1 \times \cdots \times A_n$5.
   • Back-propagate and update parameters.
   • Update the reference policy.

Default hyperparameters are $\mathcal{A} = A_1 \times \cdots \times A_n$6, $\mathcal{A} = A_1 \times \cdots \times A_n$7, $\mathcal{A} = A_1 \times \cdots \times A_n$8, $\mathcal{A} = A_1 \times \cdots \times A_n$9, $P$0 epochs per iteration.

4. Empirical Evaluation and Baseline Comparisons

MARPO was evaluated on classic cooperative multi-agent benchmarks:

• SMAC-Hard (StarCraft II Multi-Agent Challenge, six maps)
• SMACv2 (with stochasticity and delayed rewards)
• Google Research Football (GRF) cooperative tasks

Baselines included on-policy methods (MAPPO, HAPPO), sequence-modeling (MAT), and factored value-based approaches (QMIX, QPLEX, LDSA). Architectural and GAE hyperparameters were controlled across all methods.

MARPO attained average win rates of 94–100% on SMAC-Hard, outperforming all baselines: MAPPO (53–99%), HAPPO (0–73%), QMIX (34–99%), QPLEX (0–78%), LDSA (13–99%), MAT (10–99%) (Wu et al., 28 Dec 2025). Learning curves demonstrated that MARPO achieved faster initial improvement, earlier performance plateaus, and lower across-seed variance. Ablations demonstrated that removing either reflection or asymmetric clipping resulted in degraded performance and reduced sample efficiency. Hyperparameter sensitivity was low, supporting robustness.

5. Theoretical Properties and Practical Implications

The reflection mechanism doubles the effective per-update information utilized by the policy gradient, thereby reducing gradient variance and boosting sample usage. The KL-derived asymmetric clipping ties the learning step to a principled, convex surrogate, ensuring that trust region adaptations respond to empirical policy divergence rather than fixed heuristic intervals. The function $P$1 is convex, non-negative, and possesses unique roots for all practical KL levels.

Monotonic improvement properties, as inherited from Reflective Policy Optimization (RPO), yield empirically stable updates with reduced variance relative to both single-step and fixed-clip baselines. This foundation supports the practical advantages observed in the benchmark domains.

6. Comparison of Algorithmic Components

The following table summarizes the primary differences between MARPO and key baselines:

Aspect	MARPO	PPO/MAPPO	HAPPO/QMIX/MAT
Surrogate Objective	One-step + two-step (reflection)	One-step only	Varies
Clipping Strategy	Adaptive, KL-matched asymmetric	Fixed symmetric ($P$2)	Varies
Update Stability	Self-adjusting, data-driven	Fixed, hand-tuned	Varies

Inclusion of both a reflective, next-step surrogate and adaptive trust region is distinctive to MARPO and underpins its observed advantages.

7. Empirical Analysis and Ablative Findings

Ablation experiments demonstrated that eliminating the reflection term ($P$3) or reverting to symmetric clipping ($P$4) degrades both the convergence rate and ultimate performance. These results indicate the necessity of both algorithmic innovations for realizing MARPO's gains in sample efficiency and robustness. Performance on stochastic and delayed-reward environments further confirms the generality and resilience of the approach.

8. Summary and Significance

MARPO integrates trajectory-level reflection and a theoretically motivated, KL-controlled asymmetric clipping strategy to advance on-policy deep multi-agent RL. It achieves greater sample efficiency and stability over both standard and state-of-the-art baselines in benchmark cooperative environments. These characteristics are supported by both formal analysis and empirical evidence. A plausible implication is that further extensions of trajectory-level reflection or KL-adaptive trust-region control could benefit broader classes of MARL algorithms (Wu et al., 28 Dec 2025).