Conservative Q-Learning (CQL) Model

• Conservative Q-Learning (CQL) is an offline reinforcement learning method that employs a conservative regularizer to penalize unsupported Q-values and mitigate prediction errors.
• It adapts to both discrete and continuous control scenarios, with extensions like SA-CQL and CFCQL enabling robust multi-agent and domain-specific policy enhancements.
• Empirical studies demonstrate that CQL improves policy safety, data efficiency, and performance across real-world applications such as autonomous driving, healthcare, and transportation.

Conservative Q-Learning (CQL) is a class of offline reinforcement learning (RL) algorithms designed to address critical overestimation errors induced by distributional shift between static datasets and learned policies. CQL introduces a conservative regularization to the standard Bellman error minimization in Q-function estimation, enforcing pessimism on actions not well-covered in the offline data, and thereby producing reliable policies that avoid catastrophic extrapolation to out-of-distribution (OOD) actions. Its foundations, theoretical guarantees, algorithmic implementations, extensions, and domain-specific successes have established CQL as a leading framework for robust offline RL across discrete and continuous control, multi-agent systems, autonomous systems, and healthcare.

1. Theoretical Foundations of Conservative Q-Learning

CQL modifies the canonical Bellman error loss for Q-function learning in the offline RL setting by introducing a conservative penalty. Given a static dataset $\mathcal{D}$ of transitions $(s,a,r,s')$, the standard Bellman objective is

$$\mathcal{L}_{\mathrm{Bellman}}(Q) = \mathbb{E}_{(s,a,r,s') \sim \mathcal{D}} \bigl[ Q(s,a) - (r + \gamma\, \mathbb{E}_{a' \sim \pi(\cdot|s')} [Q(s', a')]) \bigr]^2.$$

CQL augments this with a regularizer penalizing high Q-values on OOD actions via log-sum-exp, yielding (for discrete actions): $$\mathcal{L}_{\mathrm{CQL}}(Q) = \mathcal{L}_{\mathrm{Bellman}}(Q) + \alpha \,\mathbb{E}_{s \sim \mathcal{D}} \Bigl[ \log \sum_{a} \exp(Q(s,a)) - \mathbb{E}_{a \sim \mathcal{D}}[Q(s,a)] \Bigr].$$ This regularizer induces a pointwise lower bound, ensuring $Q^{\mathrm{CQL}}(s,a) \leq Q^{\pi}(s,a)$ for all $(s,a) \in \mathcal{D}$, and provably depresses the estimated policy value below the true value in expectation, controlling overestimation error due to distributional shift (Kumar et al., 2020). The parameter $\alpha > 0$ regulates the strength of pessimism; its choice can be fixed or tuned via dual gradient descent.

2. Algorithmic Implementation and Practical Extensions

CQL is composable with both discrete (CQL-DQN) and continuous (CQL-SAC) Q-learning architectures. The log-sum-exp penalty over actions is efficiently implemented as either an explicit sum or Monte Carlo approximation in continuous spaces. Typical pipelines, as seen in large-scale autonomous driving (Guillen-Perez, 9 Aug 2025), employ Transformer-based feature extractors, deep multi-layer perceptrons for Q and policy networks, and automated temperature tuning for entropy regularization.

In practice:

• The critic is updated by minimizing $\mathcal{L}_{\mathrm{CQL}}$ using stochastic gradient descent.
• The actor is updated (when applicable) to maximize the conservative critic, possibly regularized with an entropy term.
• Target networks are maintained for stabilizing bootstrapped updates.

Extensions include Reweighted Distribution Support (CQL-ReDS), which replaces the standard penalty with a reweighted mixture to enforce per-state support constraints in heteroskedastic datasets, enhancing policy flexibility (Singh et al., 2022), and State-Aware CQL (SA-CQL), which further modulates the degree of pessimism via discounted state occupancies using DualDICE ratio estimation, resulting in strictly improved suboptimality bounds and empirical performance (Chen et al., 2022).

3. Multi-Agent and Distributional Conservative Q-Learning

In multi-agent contexts, CQL's direct application to the joint action space is exponentially unscalable. Counterfactual Conservative Q-Learning (CFCQL) distributes conservative penalties across agents, introducing per-agent regularization computed by counterfactual sampling (holding other agents' actions at observed values and varying the agent of interest) (Shao et al., 2023). This strategy retains the underestimation and safe policy improvement guarantees of single-agent CQL, but the induced regularization scales only linearly with the number of agents. Further, Conservative Quantile Regression (CQR) fuses distributional RL objectives with CQL penalties to address both epistemic (data-induced) and aleatoric (stochasticity-induced) uncertainties, allowing for risk-sensitive designs in multi-agent settings (Eldeeb et al., 2024).

Multi-agent CQL loss structure: $$\begin{aligned} L_{\mathrm{CFCQL}}(Q) &= \frac{1}{2} \mathbb{E}_{(s, \mathbf{a}, r, s') \sim D} \bigl[ Q(s, \mathbf{a}) - (r + \gamma \mathbb{E}_{\mathbf{a}' \sim \pi}[Q(s', \mathbf{a}')]) \bigr]^2 \ &\quad + \alpha \sum_i \lambda_i \left( \mathbb{E}_{s, a^i \sim \mu^i, \mathbf{a}^{-i} \sim \beta^{-i}}[Q(s, \mathbf{a})] - \mathbb{E}_{s, \mathbf{a} \sim \beta}[Q(s, \mathbf{a})] \right) \end{aligned}$$

4. Robust Policy Learning in Real-World Domains

CQL has demonstrated substantial gains over both behavioral cloning and other offline RL methods in diverse domains:

• Autonomous Driving: CQL applied to entity-centric, Transformer-based architectures yields policies with dramatically improved success rates and safety metrics relative to BC, leveraging conservative value estimation to mitigate compounding model errors in long-horizon planning (Guillen-Perez, 9 Aug 2025).
• Healthcare: In sepsis treatment, CQL aligns RL-generated dosing policies more closely with expert practice, reducing mortality rates and discouraging overprescription in severely ill patient cohorts (Kaushik et al., 2022).
• Transportation: RG-CQL combines a conservative Double-DQN with supervised reward estimators to enable efficient offline tuning and safe online adaptation in large-scale ride-pooling and transit systems, achieving significant improvements in both data efficiency and operational rewards (Hu et al., 24 Jan 2025).

5. Bi-Level IRL and Inverse Reinforcement Learning Integration

The BiCQL-ML framework integrates CQL into a bi-level inverse RL architecture, alternately optimizing a conservative Q-function under current rewards and updating the reward parameters to maximize expert action likelihood (Park, 27 Nov 2025). The use of CQL at the lower level prevents overgeneralization of rewards to OOD actions, yielding more robust reward recovery and downstream policy performance. Theoretical contraction mappings guarantee convergence to a soft-optimal reward and corresponding conservative Q-function under standard assumptions.

6. Empirical Results, Ablation Studies, and Theoretical Guarantees

CQL consistently achieves state-of-the-art results across standard offline RL benchmarks including MuJoCo, Atari, Adroit manipulation, AntMaze navigation, and multi-agent environments. In Mujoco and Atari, SA-CQL yields normalized scores exceeding CQL by substantial margins, and performs first or second in the majority of tested scenarios (Chen et al., 2022). In offline multi-agent benchmarks, CFCQL outperforms variants that regularize joint actions by avoiding the exponential scaling of pessimism, maintaining high performance even as the agent count grows (Shao et al., 2023).

Key empirical findings:

• Conservative regularization suppresses extrapolation error and policy value overestimation on OOD actions.
• Per-state and per-agent modulation of regularization (SA‐CQL, CFCQL) further improves both safety and performance in high-dimensional, heterogeneous, and multi-agent settings.
• Data efficiency, reward reliability, and safety (as measured by mortality in healthcare or collision and road departure in driving) are strictly improved under CQL.

Ablation studies highlight the necessity of both the regularization term and its state- or agent-adaptive structure to avoid collapse or excessive conservatism. Tuning of $\alpha$ is less critical in the presence of advanced variants (ReDS, SA-CQL) due to their adaptive penalty scaling.

7. Limitations, Open Questions, and Future Directions

CQL's efficacy hinges on sufficient coverage of relevant state-action pairs in $\mathcal{D}$ and accurately calibrated regularization strength. Current variants address some limitations of uniform pessimism via support constraints and per-state density ratio estimates, but further research is warranted in:

• Adaptive and uncertainty-aware regularization (dynamic $\alpha$, integration with OPE protocols).
• Robust extension to continuous and high-dimensional action spaces (e.g., Conservative Soft Actor-Critic).
• Dealing with severe covariate shift and limited behavioral policy support.
• Evaluation and extension in nonstationary or adversarial environments.

Integration with reward learning, risk-sensitive optimization, and large-scale multi-agent coordination represent active areas of development for conservative offline RL methodologies.

---

References:

• "Conservative Q-Learning for Offline Reinforcement Learning" (Kumar et al., 2020)
• "From Imitation to Optimization: A Comparative Study of Offline Learning for Autonomous Driving" (Guillen-Perez, 9 Aug 2025)
• "A Conservative Q-Learning approach for handling distribution shift in sepsis treatment strategies" (Kaushik et al., 2022)
• "Conservative and Risk-Aware Offline Multi-Agent Reinforcement Learning" (Eldeeb et al., 2024)
• "BiCQL-ML: A Bi-Level Conservative Q-Learning Framework for Maximum Likelihood Inverse Reinforcement Learning" (Park, 27 Nov 2025)
• "Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning" (Shao et al., 2023)
• "Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraints" (Singh et al., 2022)
• "State-Aware Proximal Pessimistic Algorithms for Offline Reinforcement Learning" (Chen et al., 2022)
• "Coordinating Ride-Pooling with Public Transit using Reward-Guided Conservative Q-Learning" (Hu et al., 24 Jan 2025)