Mean-Field Control with a Common Hidden State under Decentralized Observations

Abstract: We study optimal control of a system with multiple decision makers who share a common hidden state and receive fully decentralized observations through identical channels. The dynamics of the hidden state and the cost incurred by the agents depend on the agents' actions only through their empirical distribution. In the limit problem with infinitely many agents, the problem reduces to a single agent control problem where the agent affects the hidden state dynamics via the conditional law of the actions given the past values of the hidden state process. We formulate this problem as a deterministic measure valued control problem over the space of policies and provide a dynamic programming recursion. We first show that for the limiting problem randomization over the control actions is necessary for optimality. However, randomization over the selection of policies (i.e., mixture policies) is not required. We then show that the optimal symmetric policies designed for the infinite population problem are near optimal for the finite population problem. In particular, we establish convergence rates that decay with number of agents as $\frac{1}{\sqrt{N}}$, and grow exponentially with the memory length used in the policy.

• The paper formulates the infinite-population limit and derives a measure-valued dynamic programming recursion for decentralized control with a common hidden state.
• The paper demonstrates that optimality in the infinite-agent limit requires action randomization, achieving near-optimality for large finite populations with a convergence gap of O(N⁻¹/²).
• The paper establishes structural properties and provides explicit numerical bounds, paving the way for tractable approximation methods in large-scale decentralized systems.

Mean-Field Control with a Common Hidden State under Decentralized Observations: An Expert Summary

Problem Formulation and Motivation

This paper addresses the decentralized stochastic control of multi-agent systems where $N$ agents cooperatively control the evolution of a shared hidden state process $x_t$ under local, noisy, and decentralized observation constraints. The state dynamics and cost functional are driven by the empirical distribution of the agents’ control actions, formalized as $x_{t+1} = f(x_t, \mu_{\mathbf{u}_t}, w_t)$, with $\mu_{\mathbf{u}_t}$ representing the empirical distribution over the agents’ actions at time $t$, and observations $y_t^i = g(x_t, v_t^i)$ provided via symmetric channels per agent. The agents’ information sets are strictly local, comprising their own observation and control histories.

The agents' objective is to minimize a finite-horizon expected team cost: $$J^N(P_0, \gamma) = \sum_{t=0}^{T-1} \mathbb{E}_\gamma \left[c(x_t, \mu_{\mathbf{u}_t}) \right]$$ by selecting admissible decentralized policies $\gamma^i_t$ measurable with respect to each agent’s information set $I_t^i$. The setting is highly relevant for large-scale networked systems, distributed sensing, and smart infrastructure, where decentralized partial observation and mean-field coupling are inherent.

Infinite Population Limit and Measure-Valued Control Formulation

Assuming symmetry and exchangeability among agents, the paper investigates the infinite population limit ($N \to \infty$) and demonstrates the reduction to a single-agent randomized control problem, where the empirical distribution of actions transforms into a conditional law dependent on the past hidden state trajectory. The infinite-agent objective becomes: $x_t$0 with $x_t$1 representing the conditional law of the action $x_t$2 under the policy mixture $x_t$3. The control space is the set of measure-valued randomized policies, and the state space is the measure over the joint trajectories of states, observations, and actions. A dynamic programming recursion is formulated on this space, allowing for recursive computation of optimal policies.

Structural Properties and Policy Optimality

A rigorous structural analysis is conducted, establishing several key results:

• Necessity of Action Randomization: Optimality for the infinite population problem requires agents to randomize their actions, but not to randomize over policy sequences (mixture policies).
• Replication Lemma: Any mixture policy over policy sequences can be equivalently represented as a deterministic sequence of policies, showing that the search space for optimality can be restricted to pure (deterministic) policies that randomize actions.
• Dynamic Programming Principle: The infinite-agent measure-valued formulation enables a DP recursion over the state distribution and policy kernels, which is inherently more tractable than the finite-agent case where symmetric policies are not generally optimal.

The proof leverages disintegration, measure-theoretic structural results, and functional analytic arguments, confirming that the infinite-agent optimal policy can be implemented without centralized randomness.

Convergence Analysis and Numerical Bounds

Under regularity assumptions (compactness, Lipschitz continuity) for the action space, transition kernel, observation kernel, and cost, explicit finite-sample bounds on the convergence gap between the finite and infinite-population optimal costs are established: $x_t$4 where $x_t$5 is explicitly characterized in terms of the system parameters and increases exponentially with policy memory length.

Key Claims:

• Symmetric policies designed for the infinite-agent problem are provably near-optimal in the finite-agent regime for large $x_t$6, with performance gaps decaying at a rate $x_t$7.
• The optimality gap grows exponentially with the length of history used in the policy, highlighting a trade-off between policy expressivity and convergence.
• The results extend to higher-dimensional action spaces with appropriately modified rates.

These results generalize earlier work on mean-field teams with partial observations but fundamentally differ by including common randomness (through the hidden state) and by coupling the state evolution with the entire empirical action distribution.

Theoretical and Practical Implications

The theoretical contributions include a precise characterization of the infinite-agent limit for partially observed mean-field control under decentralized policies and a rigorous quantification of the near-optimality of infinite-population symmetric policies in practical finite-agent settings. This advances the understanding of decentralized stochastic team control, especially where common noise and policy memory are present.

Practically, the results justify designing symmetric randomized policies for large decentralized networks and quantify the expected sub-optimality. The measure-valued DP formulation, though infinite-dimensional, opens avenues for approximation algorithms, including policy gradient and truncation-based methods. The explicit convergence rates guide the choice of memory and population size for practical implementation.

Future work is suggested in developing computational methods for policy optimization, specifically by parameterizing policy spaces and considering finite-memory truncations to make DP tractable in high-dimensional settings.

Conclusion

This paper provides a rigorous framework for decentralized mean-field team control with a common hidden state and decentralized observations. Through measure-valued dynamic programming, structural policy analysis, and explicit convergence bounds, it deepens both the theoretical foundations and the practical methodologies for decentralized cooperative control in large systems with partial information. The findings have implications for scalable distributed control design, and motivate future research in tractable approximation schemes, finite-memory policies, and learning-based control for such settings.