Cooperative Hybrid Diffusion Policies (CHDP)

• CHDP is a reinforcement learning framework that employs cooperative diffusion policies to model both discrete and continuous actions in hybrid action spaces.
• It separates action selection into two agents that share a common Q-function and leverage codebook embeddings to align multi-modal action distributions.
• CHDP demonstrates state-of-the-art performance and improved sample efficiency on diverse PAMDP benchmarks compared to traditional hybrid policy approaches.

Cooperative Hybrid Diffusion Policies (CHDP) define a class of algorithms for reinforcement learning in environments characterized by hybrid action spaces, where actions comprise both discrete choices and continuous parameters. The CHDP framework introduces two expressive, diffusion-based policies—one over discrete actions and one over continuous parameters—configured to operate as fully cooperative agents, sharing a single Q-function and optimized via a sequential update mechanism. This formulation captures multi-modal action distributions, leverages codebook-based embeddings for efficient discrete action selection, and aligns both policy outputs through Q-function–driven guidance. These innovations collectively enable CHDP to overcome the key limitations of traditional hybrid action policy architectures regarding expressiveness, scalability, and sample efficiency, while attaining state-of-the-art results on standardized parameterized-action benchmarks (Liu et al., 9 Jan 2026).

1. Challenges in Hybrid Action Spaces

Hybrid action spaces are prevalent in domains such as robotics and game AI, where each decision involves both a categorical choice (e.g., which actuator to trigger or which strategy to employ) and a vector of continuous parameters (e.g., control inputs, timings). Two central challenges are identified:

• Multi-modality: Tasks often admit multiple (discrete, continuous) action pairs with equivalent or comparable rewards. Standard unimodal policy architectures, such as Gaussian or deterministic networks, typically collapse the distribution to a single mode, discarding valid alternatives and yielding suboptimal policies.
• Combinatorial Explosion: If the discrete action space has cardinality $K$, exploration complexity increases as $O(K)$, making naive exhaustive search infeasible in high-dimensional settings, especially when each discrete action is associated with distinct continuous parameters.

CHDP models these settings via the Parameterized-Action MDP (PAMDP): with state space $S$, discrete action set $A_d$, and, for each $a^d \in A_d$, a conditional continuous set $A_c(a^d)$. The agent seeks a policy $\pi(a^d, a^c \mid s)$ to maximize expected discounted return. CHDP recasts this single-agent hybrid action selection problem as a fully cooperative two-agent game, separating the discrete and continuous selection into distinct cooperative processes (Liu et al., 9 Jan 2026).

2. CHDP Framework and Architecture

The CHDP architecture comprises two diffusion policies:

• Discrete Agent: The discrete action policy $\pi_{\theta_d}(e\mid s)$ employs a diffusion process to generate a latent continuous encoding $e$, which is subsequently quantized to its nearest codebook entry $e_k$ associated with discrete action $a^d$.
• Continuous Agent: The continuous action policy $\pi_{\theta_c}(a^c\mid s, e_k)$ conditions on the state and the codeword $e_k$, producing the action parameters via its own conditioned diffusion process.

Both agents are trained to maximize the expected return computed by a shared double Q-function $Q_\phi(s, e, a^c)$, ensuring joint policy alignment. The score networks $\epsilon_{\theta_d}$ and $\epsilon_{\theta_c}$ serve as denoisers in the respective reverse diffusion steps. The coupling of discrete and continuous policy stages—via codebook quantization—enforces a dependency structure between the selected discrete mode and the continuous parameters, capturing the full joint action distribution (Liu et al., 9 Jan 2026).

3. Diffusion Processes and Score Network Details

Each agent implements a conditional diffusion process:

• Forward Kernels (Noising):
  • Discrete: $$q(e_t\mid e_{t-1}) = \mathcal{N}(e_t; \sqrt{\alpha_t}e_{t-1}, \beta_t I)$$
  • Continuous: $$q(a^c_t\mid a^c_{t-1}) = \mathcal{N}(a^c_t; \sqrt{\alpha_t}a^c_{t-1}, \beta_t I)$$
• Reverse Kernels (Denoising):
  • Discrete: $$p_{\theta_d}(e_{t-1}\mid e_t, s) \approx \mathcal{N}(\mu_{\theta_d}(e_t, s, t), \sigma_t^2 I)$$ with score function $\epsilon_{\theta_d}$
  • Continuous: $p_{\theta_c}(a^c_{t-1}\mid a^c_t, s, e_k)$ accordingly, using $\epsilon_{\theta_c}(a^c_t, s, e_k, t)$

The score networks predict the added Gaussian noise at each diffusion step, enabling the sampling of complex, multi-modal distributions over actions. The explicit conditioning on codebook entries enforces structured correlations between discrete selections and continuous parameterizations (Liu et al., 9 Jan 2026).

4. Sequential Update Mechanism and Codebook Strategy

To prevent detrimental cross-policy interference during learning, CHDP adopts a sequential gradient update protocol per training step:

1. Discrete Policy Update: Minimize score prediction loss $L_d(\theta_d)$ and maximize expected Q-value $L_q(\theta_d)$, using fixed continuous actions from replay.
2. Continuous Policy & Codebook Update: Minimize score loss $L_d(\theta_c)$ and maximize Q-value $L_q(\theta_c, \zeta)$, with gradients propagating through codebook embeddings $e_k$.
3. Critic Update: Employ Double Q-learning with value target $y = r + \gamma \min_j Q'_{\phi'_j}(s', e', a'^c)$.

The codebook $E_\zeta \in \mathbb{R}^{K \times d_e}$ bridges the discrete agent’s continuous output $e$ with the discrete action index $k$. Unlike standard vector quantization (VQ) approaches based on reconstruction, CHDP aligns codebook entries via downstream Q-function gradients, embedding semantics of high-value actions into a compact latent space. The stop-gradient operation $sg(e_k)$ ensures that learning signals propagate correctly through the continuous stage only (Liu et al., 9 Jan 2026).

Component	Representation	Update Mechanism
Discrete policy agent	$e \in \mathbb{R}^{d_e}$	Score + Q-guidance
Codebook	$E_\zeta \in \mathbb{R}^{K \times d_e}$	Q-aligned embedding
Continuous policy agent	$a^c$	Score + Q-guidance

5. Q-Function Guidance and Combined Objectives

Both policies integrate Q-learning–style guidance to bias actions toward regions of maximal expected value:

• Discrete guidance: $$L_q(\theta_d) = -\mathbb{E}_{s,e,a^c}[Q_\phi(s, e, a^c)]$$
• Continuous guidance: $$L_q(\theta_c) = -\mathbb{E}_{s, sg(e_k), a^c}[Q_\phi(s, sg(e_k), a^c)]$$

The overall objectives combine score matching (denoising accuracy) with Q-driven policy improvement:

• Discrete: $$L(\theta_d) = L_d(\theta_d) - \alpha \mathbb{E}[Q_\phi(s, e, a^c)]$$
• Continuous: $$L(\theta_c, \zeta) = L_d(\theta_c) + \alpha \mathbb{E}[-Q_\phi(s, sg(e_k), a^c)]$$
• Critic: $$L(\phi_i) = \mathbb{E}[(Q_{\phi_i}(s, e, a^c) - y)^2]$$

These terms obviate the need for additional kernel shaping, relying instead on the learned value structure to guide both discrete and continuous denoising toward optimal regions (Liu et al., 9 Jan 2026).

6. Empirical Performance and Benchmark Analysis

CHDP has been evaluated on eight PAMDP benchmark environments, including Platform, Goal, Catch Point, Hard Goal, and Hard Move tasks with $n \in \{4,6,8,10\}$ actuators. Compared to prior methods (HPPO, PA-TD3, PDQN-TD3, HHQN-TD3, HyAR-TD3), CHDP achieves the highest mean success rate in all environments, e.g., surpassing HyAR-TD3 by up to $19.3\%$ in the Hard Goal setting (CHDP: $79.5\%$ vs. HyAR-TD3: $60.2\%$). In the Hard Move tasks ($|A_d| = 2^8 = 256$), CHDP maintains $>90\%$ success rate, whereas baseline methods suffer catastrophic failure.

Sample efficiency is also enhanced: CHDP demonstrates faster convergence and higher asymptotic performance, as evidenced by learning curves on all test domains (Liu et al., 9 Jan 2026).

7. Discussion, Limitations, and Extensions

The success of CHDP is attributed to:

• Expressiveness: Diffusion-based policies are capable of representing multi-modal distributions over joint discrete-continuous action spaces.
• Co-adaptation: Sequential updates and judicious use of gradients prevent destabilization due to cross-policy interactions.
• Scalability: The codebook mechanism embeds large discrete action sets into structured, compact latent spaces oriented by reward semantics.

Identified limitations include computational overhead from diffusion sampling (e.g., $15$ denoising steps per action), practical codebook configuration for extremely large $|A_d|$, and sensitivity to codebook dimension choices. Prospective research directions involve adaptive or hierarchical codebooks, integrating model-based rollouts into the diffusion process, extending CHDP to offline reinforcement learning settings, and incorporating entropy-based policy regularization (Liu et al., 9 Jan 2026).

A plausible implication, evident from related work in human-robot collaboration, is that diffusion-based policies naturally promote emergent cooperative behaviors such as mutual adaptation, leadership switching, and temporally consistent multimodal planning, even in complex joint or hybrid action spaces (Ng et al., 2023). The success of CHDP on standardized benchmarks suggests generalizability to collaboration-centric domains requiring rich action compositionality.

---

References:

• "CHDP: Cooperative Hybrid Diffusion Policies for Reinforcement Learning in Parameterized Action Space" (Liu et al., 9 Jan 2026)
• "Diffusion Co-Policy for Synergistic Human-Robot Collaborative Tasks" (Ng et al., 2023)