Bayesian Adversarial Robust Dec-POMDP

• The paper introduces BARDec-POMDP, a novel framework for robust MARL under adversarial conditions using Bayesian inference and equilibrium analysis.
• It employs a two-timescale actor-critic algorithm and robust Markov perfect Bayesian equilibria to adapt policies and achieve convergence in adversarial settings.
• Empirical evaluations in benchmarks like SMAC and LBF demonstrate significant performance gains and resilience against diverse adversarial threat models.

A Bayesian Adversarial Robust Decentralized Partially Observable Markov Decision Process (BARDec-POMDP) is a formalism for multi-agent reinforcement learning (MARL) in environments where agents may be subject to Byzantine failures or adversarial control. BARDec-POMDP is designed to ensure robust cooperation among benign agents, while minimizing vulnerabilities to agents that may deviate arbitrarily from prescribed policies due to malfunction or attack. The framework conceptualizes agent types as nature-dictated, latent variables and employs Bayesian inference over these types, allowing each agent to adapt its policy based on posterior beliefs about which teammates are benign and which behave adversarially. The principal solution is an ex interim robust Markov perfect Bayesian equilibrium (RMPBE), which is guaranteed to exist in mixed strategies and to weakly dominate prior robust MARL equilibria in the worst case. A two-timescale actor-critic algorithm achieves convergence to the equilibrium under classical stochastic approximation conditions, facilitating practical implementation and empirical validation in diverse benchmark domains (Li et al., 2023).

1. Formal Model and Bayesian Type Framework

The classical cooperative Decentralized Partially Observable Markov Decision Process (Dec-POMDP) is generalized to include agent types that capture the possibility of adversarial (Byzantine) actions. The standard Dec-POMDP is given by:

$$\mathcal G = \bigl\langle \mathcal N,\,\mathcal S,\,\mathcal O,\,O,\,\mathcal A,\,\mathcal P,\,R,\,\gamma\bigr\rangle,$$

where $\mathcal N$ is the set of agents; $\mathcal S$ the global state space; $\mathcal O$ and $O$ specify observations; $\mathcal A$ is the joint action space; $\mathcal P$ is the transition kernel; $R$ is the shared reward function; and $\gamma$ is the discount factor.

BARDec-POMDP introduces a type space $\Theta = \times_{i \in \mathcal N} \Theta^i$, with each $\Theta^i = \{0, 1\}$ denoting whether agent $i$ is benign ($\theta^i = 0$) or Byzantine ($\theta^i = 1$). The tuple

$$\hat{\mathcal G} = \bigl\langle \mathcal N,\,\mathcal S,\,\Theta,\,\mathcal O,\,O,\,\mathcal A,\,\mathcal P^\alpha,\,\mathcal P,\,R,\,\gamma\bigr\rangle$$

incorporates an action perturbation kernel $$\mathcal P^\alpha(\overline{\mathbf a} \mid \mathbf a, \hat{\pi}, \theta)$$, such that for each agent $i$, if $\theta^i = 1$ the prescribed action $a^i$ is replaced with an adversarial choice according to a policy $\hat{\pi}^i(\cdot \mid H^i, \theta)$. The system’s evolution then follows the perturbed joint action $\overline{\mathbf a}$.

Each agent maintains a posterior belief $b^i_t(\theta) = p(\theta \mid H^i_t)$, updated recursively using Bayes’ rule given observed trajectories and the action perturbation model.

2. Solution Concepts: Robust Bayesian Equilibria

The optimality criterion for BARDec-POMDP is formulated as Robust Markov Perfect Bayesian Equilibrium (RMPBE), with distinctions between ex ante and ex interim robustness.

• Ex Ante RMPBE: Policies are optimized under the prior $p(\theta)$, unconditioned on observed behavior. The solution is:

$$(\pi^{EA}_*, \hat{\pi}^{EA}_*) \in \arg\max_{\pi} \mathbb{E}_{p(\theta)}\left[\min_{\hat{\pi}} V_\theta^\pi(s)\right].$$

• Ex Interim RMPBE: Policies condition on current posterior beliefs $b^i$ over types. For each agent $i$, candidate policies achieve:

$$\left(\pi_*^{EI}(\cdot \mid H, b), \hat{\pi}_*^{EI}(\cdot \mid H, \theta)\right) \in \arg\max_{\pi(\cdot \mid H,b)} \mathbb{E}_{b(\theta)}\left[\min_{\hat{\pi}(\cdot \mid H, \theta)} V_\theta(s)\right].$$

Standard fixed-point theory (Kakutani’s theorem) establishes existence of mixed-strategy ex ante and ex interim robust MBEs, even when pure-strategy equilibrium fails. As $t\to\infty$, the posterior $b^i_t$ concentrates on the true type vector $\theta$, implying that the ex interim equilibrium policy weakly dominates ex ante policy against the worst-case adversary:

$$\lim_{t\to\infty} V_\theta(s; \pi_*^{EI}, \hat{\pi}_*) \geq V_\theta(s; \pi_*^{EA}, \hat{\pi}_*) \quad \forall \theta.$$

3. Algorithmic Realization: Two-Timescale Actor–Critic

The algorithmic approach centers on robust Harsanyi–Bellman equations for value functions $Q^i(s, \mathbf a, b^i)$ and $Q^i(s, \overline{\mathbf a}, b^i)$, encoding the minimax outcome over perturbed actions and Bayesian beliefs. The operator is a $\gamma$-contraction, ensuring unique fixed points for these Q-functions.

Defender and adversarial policies are parameterized respectively as $\pi^i_{\phi^i}$ and $\hat{\pi}^i_{\hat{\phi}^i}$, with a critic $Q^i_\psi(s, \overline{\mathbf a}, b^i)$ and a belief network $p_\xi(\theta \mid H^i)$.

Policy optimization employs a two-timescale stochastic gradient procedure:

• Adversary (fast timescale):

$$\hat{\phi}^i \leftarrow \hat{\phi}^i + \alpha_t \nabla_{\hat{\phi}^i} J^i(\hat{\phi}).$$

• Defender (slow timescale):

$$\phi^i \leftarrow \phi^i + \beta_t \nabla_{\phi^i} J^i(\phi), \quad \frac{\beta_t}{\alpha_t} \to 0.$$

• Critic and belief network are updated according to temporal-difference and belief cross-entropy losses, respectively. The sequence of updates is embedded in the Generalized Differential Actor (GDA) update framework.

Convergence to a mixed-strategy ex interim robust MBE is guaranteed almost surely under standard conditions (Borkar-Meyn): step sizes $\alpha_t, \beta_t$ sum to $\infty$, their squares sum to $<\infty$, and all relevant state/action/type triples are visited infinitely often.

4. Empirical Evaluation and Micromanagement Performance

BARDec-POMDP and its implementation ("EIR-MAPPO") were evaluated in three benchmark environments: an iterative matrix game, Level-Based Foraging (LBF), and the StarCraft Multi-Agent Challenge (SMAC). Four distinct types of adversarial/threat models are considered:

Attack Type	Description	Evaluation Metric
Non-oblivious adversary	Learns explicit worst-case policy	Average episodic return
Random failure	Uniform random actions
Observation-based attack	$\ell_\infty$-bounded PGD noise ($\epsilon$)
Transferred adversary	Trained on surrogate algorithm then deployed

EIR-MAPPO achieved near-optimal robust returns under non-oblivious adversary scenarios (Toy $\approx$50, LBF $\approx$1.0, SMAC $\approx$20), closely matching the “True Type” oracle and surpassing prior robust MARL baselines by 5–25% in average return under all threats. Qualitative examples in SMAC demonstrate that, even under maximal adversarial disruption, the learned policy exhibits intricate skills such as kiting, focused fire, and dynamic role adaptation, whereas baselines fail to coordinate or are easily distracted by adversaries (Li et al., 2023).

5. Practical Implications and Limitations

BARDec-POMDP establishes a rigorous foundation for robust MARL in safety-critical domains where Byzantine agents may arise. By treating adversarial failures as latent types and leveraging Bayesian posterior updates, agents can:

• Reliably discriminate between allies and adversaries,
• Engage in collaboration when trust is justified,
• Default to minimax-randomized strategies under threat.

Potential application areas include robotic swarms, autonomous traffic management, critical infrastructure control (e.g., power grids), and distributed systems requiring resilience to misbehaving nodes.

Several limitations should be noted:

• The convergence guarantees are predicated on strict assumptions, such as finite state, action, and type spaces, and the requirement that all scenarios are visited infinitely often.
• The methodology currently addresses only action perturbations; robustness to observation/reward/transition modifications remains an open problem.
• The exponential scaling of the type space with agent count challenges tractability. Approaches such as factorized representations (QMIX-style) or continuous priors may provide relief.
• The supervised binary cross-entropy approach to belief estimation may benefit from advances in opponent modeling or “aware” online adaptation.

6. Future Directions and Research Opportunities

Work remains in extending function approximation guarantees to BARDec-POMDP, enabling sample-efficient robust MARL with scalable architectures. Improved modeling of type structure (e.g., via hierarchical or continuous latent spaces) could mitigate the exponential complexity associated with enumerating all possible ally/adversary configurations. Faster and more precise belief adaptation strategies, possibly informed by opponent learning or meta-learning modules, hold promise for accelerating discrimination between benign and adversarial agents. Finally, broadening the adversarial model to include perturbations of observation, reward, or environment dynamics may yield frameworks of greater practical relevance for heterogeneous and adversarial multi-agent systems (Li et al., 2023).