Embedding MPC in RL

• Embedding MPC in RL is a hybrid approach that replaces conventional approximators with parameterized MPC schemes to enforce safety and constraint adherence.
• It leverages differentiable optimization techniques like KKT sensitivity analysis to enable effective policy and value gradient computations in RL.
• Applications in robotics and industrial process control demonstrate enhanced sample efficiency, safety guarantees, and robust performance compared to pure RL or MPC methods.

Embedding Model Predictive Control (MPC) as a function approximator in Reinforcement Learning (RL) is a principled framework that integrates the structure, safety, and constraint-handling abilities of MPC with the adaptivity and data-driven optimization of RL. In this paradigm, the classical neural or linear function approximators for value functions and policies are partially or wholly replaced by parameterized MPC schemes. These MPC schemes can serve as value function approximators, policy approximators, or both, resulting in hybrid architectures with distinct theoretical and practical advantages. Below, key dimensions of this approach are outlined with reference to foundational and recent research.

1. Mathematical Foundations of MPC-Embedded RL

The core principle involves defining the value function $Q_\theta(s, a)$ or the policy $\pi_\theta(s)$ implicitly through the solution of a finite-horizon MPC optimal control problem, parameterized by a tunable vector $\theta$. At each RL step, instead of propagating gradients or Q-values through a learned network, the RL agent evaluates the cost-to-go of taking action $a$ in state $s$ by solving:

$$Q_\theta^N(s,a) = \min_{u_{1:N-1},x, \sigma} J_\mathrm{MPC}(x_0=s, [a,u_{1:N-1}], \sigma; \theta)$$

where $J_\mathrm{MPC}$ incorporates:

• Stage costs $\ell_\theta(x_k, u_k)$ and terminal costs $V^f_\theta(x_N)$, parameterized by $\theta$.
• Dynamics constraints $x_{k+1} = f_\theta(x_k, u_k)$, potentially with model parameters included in $\theta$.
• State/input constraints $h_\theta(x_k, u_k) \leq \sigma_k$, possibly softened with slack variables $\sigma_k$.
• Additional terms handling arrival costs or constraint tightening as needed (Zanon et al., 2019, Airaldi et al., 2022).

The greedy policy is obtained by $\pi_\theta(s) = \arg\min_a Q_\theta^N(s,a)$, and value-based methods use the solution $V_\theta^N(s) = \min_a Q_\theta^N(s,a)$. For actor-critic and policy gradient methods, the actor is likewise parameterized via the argmin of a (possibly mixed-integer) MPC scheme (Gros et al., 2020).

2. Integration with RL Algorithms

RL integration is carried out by adapting the Bellman update or policy gradient computations to account for the implicit dependence of the value/policy on the MPC parameters:

Q-Learning and Value-Based Methods

For a transition $(s_k, a_k, s_{k+1}, \ell_k)$, a TD error is computed as:

$$\delta_k(\theta) = \ell(s_k, a_k) + \gamma \min_{a'} Q_{\tilde\theta}^N(s_{k+1}, a') - Q_\theta^N(s_k, a_k)$$

A stochastic gradient or Gauss-Newton batch update adjusts $\theta$ to minimize the squared TD error, with optional constraints to preserve positiveness and stabilizing properties of the costs (Zanon et al., 2019).

Policy Gradient and Actor-Critic Methods

In these settings, the deterministic policy gradient is evaluated as:

$$\nabla_\theta J(\pi_\theta) = \mathbb{E}_{s \sim d^{\pi_\theta}}[ \nabla_\theta \pi_\theta(s) \cdot \nabla_a A_{\pi_\theta}(s, a)\vert_{a = \pi_\theta(s)} ]$$

where $\pi_\theta(s)$ is obtained as the solution to the MPC problem. The gradient $\nabla_\theta \pi_\theta(s)$ is extracted efficiently via sensitivity analysis or implicit differentiation of the KKT system governing the MPC subproblem (Amos et al., 2018, Kordabad et al., 2021, Esfahani et al., 14 Jul 2025).

For mixed-integer MPC policies, exploration in the discrete and continuous action spaces is achieved by softmax sampling and linear perturbations, ensuring that all exploration actions remain feasible under the original constraints (Gros et al., 2020).

3. Structural Properties, Constraints, and Safety

Embedding MPC as a function approximator ensures that all policies considered during learning are generated by the solution of a well-posed, typically convex, constrained optimization problem. As a direct consequence:

• State and input constraints are always enforced (or only mildly violated via slack variables).
• Stability and feasibility are preserved if the MPC quadratic cost and terminal settings satisfy Lyapunov-based or positive definiteness conditions throughout learning updates (Zanon et al., 2019, Airaldi et al., 2022).
• Parameter updates can be safety-constrained via chance constraints modeled using Gaussian Process surrogates or by incorporating Lyapunov-type penalty terms into the learning objective, providing probabilistic guarantees on closed-loop constraint satisfaction (Airaldi et al., 2022, Esfahani et al., 14 Jul 2025, Dzhumageldyev et al., 4 Dec 2025).

Innovations such as the integration of Control Barrier Functions (CBFs) into the MPC constraints allow the learned policy to guarantee forward invariance of safety sets, even in the face of learning-based tuning of both MPC costs and CBF decay parameters (Dzhumageldyev et al., 4 Dec 2025).

4. Differentiable and Efficient MPC Integration

Computing policy gradients or value gradients through MPC requires differentiation through the nonlinear (or quadratic) optimization problem. This can be accomplished by:

• Differentiating the KKT optimality conditions at the solution—a procedure that shares computational primitives with Riccati or Newton backward passes in DDP/iLQR algorithms (Amos et al., 2018, Esfahani et al., 14 Jul 2025).
• Using "soft" or fictitious MPC rollout surrogates, replacing the hard argmin by a differentiable closed-form or learned controller (e.g., MPCritic), enabling standard backpropagation over batched data with significant computational acceleration (Lawrence et al., 1 Apr 2025).
• Employing implicit function theorem-based differentiation for mixed-integer or parameterized NLP/QP-based MPC layers (Gros et al., 2020, Kordabad et al., 2021).

These approaches are crucial for the practical deployment of such schemes in actor–critic, policy-gradient, or batch RL algorithms where batched and parallel computing through PyTorch or TensorFlow is desired (Lawrence et al., 1 Apr 2025).

5. Sample Efficiency, Safety, and Explainability

MPC-based function approximation imposes strong structural priors, leading to:

• Reduced number of learnable parameters compared to deep neural policy/value representations, yielding dramatic improvements in sample efficiency (Amos et al., 2018, Airaldi et al., 2022).
• Enhanced explainability, as policies can be inspected via their MPC weights or constraints, and are amenable to formal analysis of stability, reachability, and safety specifications (Zanon et al., 2019, Sawant et al., 2022).
• Robustness to model error via online value correction. Frameworks blending MPC local value estimates with learned global value functions (e.g., TD($\lambda$)-style mixing) yield performance close to that of perfect-model MPC, even when the internal MPC model is biased (Bhardwaj et al., 2020).
• In multi-agent and distributed RL, MPC-based value and policy approximators can be parallelized using consensus/ADMM schemes, enabling nonstationarity-free learning updates and scalability (Mallick et al., 2023, Kordabad et al., 2021).

6. Applications and Empirical Results

Empirical validation spans domains including:

• Nonlinear process control (evaporation process): Closed-loop cost reductions of ≈14% vs. naïve MPC; learning shifts the safe-operating region under constraints (Zanon et al., 2019).
• Atari Pong and inverted pendulum: MPC-augmented RL outperforms both pure MPC and pure RL in terms of game reward, safety, and learning speed (Rathi et al., 2019).
• Robotic manipulation and multi-agent energy systems: Embedded MPC architectures demonstrate sample-efficient convergence while maintaining strict state/input constraints (Kordabad et al., 2021, Mallick et al., 2023).
• Industrial process control: Multi-objective Bayesian optimization of MPC parameters enables both performance improvement and formal Lyapunov stability margins, outperforming DNN-RL baselines in sample efficiency and robustness (Esfahani et al., 14 Jul 2025).
• Safe navigation and obstacle avoidance: MPC–CBF policies guarantee collision-free motion and learn optimal trade-offs between task performance and safety constraint satisfaction, including with dynamic obstacles (Dzhumageldyev et al., 4 Dec 2025).

A summary table of approaches and key concepts is provided below.

Paper (arXiv ID)	MPC Usage in RL	Safety/Structure	Differentiability	Example Domain
(Zanon et al., 2019)	Q-function, policy	Hard constraints, PSD	KKT sensitivities	Nonlinear process
(Amos et al., 2018)	Policy, value	Box constraints	Implicit diff/KKT	Pendulum, Cartpole
(Airaldi et al., 2022)	Policy	GP-based chance safety	NLP sensitivities	Quadrotor
(Gros et al., 2020)	Mixed-int. policy	Feasible stoch. policy	Parametric NLP diff	Toy MI-problems
(Bhardwaj et al., 2020)	Q-value, policy	Model bias robustness	Sampling/mixing	Manipulation
(Esfahani et al., 14 Jul 2025)	Policy (CDPG+BO)	Lyapunov, slacks	KKT sensitivities	CSTR, Industrial
(Dzhumageldyev et al., 4 Dec 2025)	Policy, value	CBF constraints	KKT/NN/RNN diff	Obstacle avoid.
(Lawrence et al., 1 Apr 2025)	Critic, policy	Penalty constraints	Automatic diff	LQR, CSTR

7. Limitations, Open Questions, and Extensions

Recent work indicates several challenges and directions:

• Scalability: Solving and differentiating through large QPs/NLPs per RL step can be computationally intensive, especially in high-dimensional, long-horizon, or multi-agent settings (Sawant et al., 2022).
• Parameterization: Over-restricting to the MPC structure may limit policy expressivity in highly nonlinear or nonconvex tasks, whereas relaxing structure risks loss of explainability and safety guarantees.
• Adaptivity vs. Safety: The design of state- or uncertainty-dependent mixing coefficients in blended schemes (e.g., TD($\lambda$)-type) remains an active research area (Bhardwaj et al., 2020).
• Safe Exploration: While MPC naturally enforces safety in action generation, exploration in parameter space (e.g., via Bayesian optimization or GP safety sets) is required to guarantee safety during online learning (Airaldi et al., 2022, Esfahani et al., 14 Jul 2025).
• Data-Driven Modeling: Integrating data-enabled predictive control (DeePC) or learned dynamics models into MPC–RL frameworks is promising for systems with limited first-principles models (Vahidi-Moghaddam et al., 5 Oct 2025).

In sum, embedding MPC as a function approximator within RL offers a mature and mathematically rigorous route to safe and sample-efficient learning in constrained decision-making. Current and emerging research focuses on expanding scalability (via differentiable approximations and parallelization), enhancing robustness to model errors, and establishing theoretical convergence and safety guarantees in both centralized and distributed multi-agent contexts.