Population-Aware Imitation Learning in Mean-field Games with Common Noise

Abstract: Mean Field Games (MFGs) provide a powerful framework for modeling the collective behavior of large populations of interacting agents. In this paper, we address the problem of Imitation Learning (IL) in MFGs subject to common noise, where the population distribution evolves stochastically. This stochasticity compels agents to adopt population-aware policies to respond to aggregate shocks. We formulate two distinct learning objectives: recovering a Nash equilibrium and maximizing performance against an expert population. We investigate two imitation proxies: Behavioral Cloning (BC) and Adversarial (ADV) divergence. We then establish finite-sample error bounds showing that minimizing these proxies effectively controls both the policy's exploitability and its performance gap relative to the expert. Furthermore, we propose a numerical framework using generalized Fictitious Play and Deep Learning to compute expert population-aware policies. Through experiments on three environments we demonstrate that standard population-unaware policies fail to capture the equilibrium dynamics. Our results highlight that learning population-aware policies is crucial to avoid being misled by the randomness inherent in common noise.

• The paper presents the first theoretical framework for population-aware imitation learning in stochastic mean-field games with common noise.
• It establishes performance guarantees using Behavioral Cloning and Adversarial divergence proxies, with ADV minimization offering robust equilibrium recovery.
• Experimental analysis shows that adaptive policies consistently outperform vanilla approaches in dynamic, noisy environments, achieving lower exploitability and better reward-matching.

Population-Aware Imitation Learning in Mean-field Games with Common Noise

Overview and Motivation

The paper "Population-Aware Imitation Learning in Mean-field Games with Common Noise" (2605.03357) develops the first theoretical and algorithmic foundation for imitation learning (IL) in stochastic mean-field games (MFGs) with common noise. Classical MFGs model agent interactions via deterministic population distribution flows, while common noise introduces systemic shocks that render the mean-field stochastic. This stochasticity fundamentally alters the notion of optimality, making vanilla (population-unaware) policies provably suboptimal. The work identifies the necessity for population-aware (adaptive) policies and provides a quantitative framework for recovering equilibria and high-performing policies from expert data in unknown reward environments.

Problem Formulation and Theoretical Framework

The authors formalize MFGs with common noise as a tuple consisting of state/action spaces, distributional kernels, reward functions, and a stochastic process governing common noise. A Nash equilibrium in this setting is a fixed point under social reactivity and environmental coupling. Two core IL objectives are targeted:

• Recovering a Nash equilibrium (minimizing exploitability when all agents adopt the learned policy).
• Matching expert performance (maximizing reward when the population follows the expert while the learner deviates).

Behavioral Cloning (BC) and Adversarial (ADV) divergence proxies are proposed to quantify imitation quality: $\delta^{BC}$ measures local policy discrepancy, while $\delta^{ADV}$ quantifies global deviations in state-action distributions.

Theoretical Guarantees

Theoretical bounds are provided linking BC and ADV minimization to equilibrium quality and performance gap. Key results include:

• BC minimization yields reward-matching guarantees in unknown reward environments, regardless of discontinuity or environmental irregularity.
• ADV minimization provides more robust equilibrium approximation, with polynomial rather than exponential error propagation in BC.

Both bounds are sensitive to the population's social reactivity ($L_E$) and environmental coupling ($L_P$), with explicit scaling quantified for exploitability and performance gap.

Algorithmic Framework: Population-Aware Policy Learning

The authors generalize Fictitious Play for MFGs with common noise, leveraging deep learning for best-response computation, and propose methods for policy distillation from Fictitious Play sequences. The adaptive policy is parameterized as a neural network mapping from state and mean-field distribution, trained via interactive imitation against a population of expert agents. Vanilla policies, which ignore the population distribution, are contrasted as baselines.

Experimental Analysis

Three environments demonstrate the necessity and efficacy of population-aware policies:

1. 2-state, 2-action congestion model: Adaptive policies consistently outperform vanilla, matching equilibrium dynamics and reward.

Figure 1: Performance metrics for 5 different runs, with $\eta = 0.75$.

2. Beach Bar congestion environment: Adaptive policies react to mean-field fluctuations, successfully reconstructing expert Nash adaptation; vanilla policies baseline to average actions.

Figure 2: Performance metrics for 5 different runs, with $\eta = 0.3$.

Figure 3: One realization of the mean-field trajectory generated by the expert, $(\alpha, \eta) = (1, 0.3)$.

3. Night Clubs social choice environment: The adaptive policy again aligns with expert population-aware responses, while vanilla policy fails to capture dynamic reactivity.

Figure 4: Performance metrics for 5 different runs, with $\alpha = 1.05$.

Figure 5: One realization of the mean-field trajectory generated by the expert, $(\alpha, \eta) = (0.1, 1)$.

Strong Claims and Quantitative Findings

A pronounced numerical gap is observed between population-aware (adaptive) and population-unaware (vanilla) policies across all environments. The adaptive policies consistently achieve lower exploitability and better reward-matching in stochastic settings. The BC proxy allows performance matching, but only ADV minimization delivers robust equilibrium recovery. Theoretical bounds are shown to scale polynomially with ADV, contrasting the exponential sensitivity in BC (notably, equilibrium recovery is fragile while exploitation minimization is robust).

Implications and Future Directions

The paper’s analysis reveals that vanilla policies are fundamentally misleading under common noise; population-aware strategies are indispensable. This has implications for practical systems—financial networks, crowd management, energy infrastructures—where aggregate shocks and stochastic fluctuations drive collective behavior.

The population-aware deep IL pipeline (generalized Fictitious Play + deep policy distillation) establishes a robust methodology for equilibrium computation and learning in high-dimensional stochastic environments. The framework opens avenues for further research into state-action mean field interactions, real-world data deployments, and the exploration of improved adversarial proxies.

On a theoretical level, the work advances the understanding of policy approximation in stochastic MFGs, providing finite-sample guarantees for imitation learning under unknown reward environments and stochastic mean fields.

Conclusion

The paper establishes the first IL framework for mean-field games with common noise, proving that only population-aware policies can reproduce equilibrium dynamics and expert-level performance under stochastic aggregates. Population-unaware policies will always be suboptimal when common noise is present. Imitation metrics (BC, ADV) are quantitatively linked to equilibrium quality and reward-matching, with ADV delivering robust equilibrium recovery. The deep learning pipeline enables high-dimensional population-aware policy learning, and the results recommend adaptive strategies for any stochastic population system.