Dyna-Style Safety Augmented Reinforcement Learning: Staying Safe in the Face of Uncertainty

Abstract: Safety remains an open problem in reinforcement learning (RL), especially during training. While safety filters are promising to address safe exploration, they are generally poorly suited for high-dimensional systems with unknown dynamics. We propose Dyna-style Safety Augmented Reinforcement Learning (Dyna-SAuR), a novel algorithm that learns both a scalable safety filter and a control policy using a learned uncertainty-aware dynamics model, while requiring minimal domain knowledge. The filter avoids failures and high uncertainty regions. Thus, better models expand the set of safe and certain states, reducing filter conservatism. We present the effectiveness of Dyna-SAuR on goal-reaching CartPole as well as MuJoCo Walker, reducing failures compared to state-of-the-art methods by 2 orders of magnitude.

• The paper's main contribution is a Dyna-style RL algorithm that jointly learns an uncertainty-aware dynamics model and safety filter to minimize training-time failures.
• It employs a novel hyperplane parametrization for the safety filter, enabling efficient and expressive action space search in high-dimensional settings.
• Experimental results show a two order-of-magnitude reduction in safety violations while maintaining competitive control performance.

Dyna-Style Safety Augmented Model-Based Reinforcement Learning: The DYNASAUR Framework

Introduction

Ensuring safety during exploration and learning has remained a central technical bottleneck in deploying Reinforcement Learning (RL) on real-world platforms. While safety filters and constrained RL have seen significant research, existing techniques are typically limited by dependence on high-fidelity models, restrictive assumptions, or insufficient scalability for high-dimensional control. "Dyna-Style Safety Augmented Reinforcement Learning: Staying Safe in the Face of Uncertainty" (2604.25508) introduces DYNASAUR, a Dyna-style model-based RL algorithm that addresses these challenges by integrating joint learning of both scalable safety filters and control policies via an uncertainty-aware dynamics model with minimal domain assumptions.

The core contributions lie in: a principled RL formulation for learning viability-based safety filters in the presence of model uncertainty, a novel hyperplane parametrization for efficient and expressive action-space search, and an integrated training paradigm aligning model, filter, and control policy learning to minimize training-time failures without sacrificing asymptotic performance.

Figure 1: Schematic of the DYNASAUR mechanism—data-driven model, safety filter, and controller learning with iterative data/model refinement expand the set of safely-explorable states.

Technical Framework

Safety in Model-Based Reinforcement Learning

DYNASAUR addresses sequential decision-making in MDPs $M = \{ S, A, p, r \}$ with safety defined via a user-provided failure set $S_F \subset S$ and an initial dataset. The framework focuses on training-time safety, formally minimizing failures accrued during exploration, not just deployment. The RL agent learns:

• A predictive uncertainty-aware ensemble model: capturing both epistemic and aleatoric uncertainty, and defining an "accurate set" $E$ of state-action pairs $(s, a)$ where the model provides certifiable reliability via predictive entropy bounds.
• A scalable, learned safety filter: parameterized as a state-dependent hyperplane in action space, projecting proposed actions to a viable subset with high probabilistic guarantees of avoiding both explicit failures and high-uncertainty regions.
• A performant control policy: trained under the safety filter, balancing robust exploratory behavior and return maximization within the expanding certain/viable region as the model improves.

Viability kernels are extended into the model-based setting to guard against failure in both unsafe and uncertain regions, with all safety statements made over finite horizons under process noise.

Safety Filter Parametrization and Learning

The filter is realized as a neural parametric mapping

$$a^V(s) = \arg\min_a \|a - \pi(s)\|_2 \,\, \text{s.t.} \quad w(s)^\top a \geq b(s)$$

with $(w, b)$ produced via a bijective transformation from a unit hyperball parameter space, ensuring non-redundant, bounded, and expressive hyperplane representations intersecting the bounded action domain $A$. This parametrization addresses non-identifiability and search-space issues prevalent in previous work [black-box-lavankul].

Learning the safety filter is framed as an RL problem itself (the "filter MDP"), with reward design and rollout strategies informed by the need for maximal "least-restrictive" viable sets. Value-based RL, using actor-critic (FastTD3 variant) with high-throughput, model-generated rollouts, and prioritized state distributions, achieves tractable convergence.

(Figure 2)

Figure 2: Illustration of the hyperplane action space—the filter parametrization ensures expressive yet computationally stable mapping to viable-region projections.

Iterative Dyna-Style Procedure

DYNASAUR alternates between model, filter, and control policy learning in a Dyna-style loop:

1. Model Update: Retrain the uncertainty-aware ensemble model on data from the previous safe interactions, expanding the set $E$.
2. Filter Update: Retrain the safety filter MDP from scratch, ensuring generalization and avoidance of local minima.
3. Control Policy Training: Train under the fixed filter, collecting new data while monitoring both control return and safety violations.
4. Data Aggregation: Add all new safe interactions to the experience buffer, further closing uncertainty gaps.

This loop drives safe exploration and coverage of the task domain while minimizing the cumulative probability of failure at each step.

Experimental Results

The empirical evaluation focuses on two standard but challenging RL settings: goal-reaching CartPole with explicit state constraints and high-dimensional MuJoCo Walker with safety constraints on joint angles and torso position.

Figure 3: Depiction of the goal-reaching CartPole setup with explicit unsafe regions (red) and target goal (blue).

DYNASAUR demonstrated two primary outcomes:

• Control Performance: Matches or slightly outperforms strong model-free safe RL methods (PPO-Lagrangian [Ray2019SafeExploration], DH-RL [black-box-lavankul]) on both domains, and reaches competitive asymptotes compared to unconstrained, safety-agnostic model-based RL (Infoprop-Dyna).
• Training-Time Failures: Reduces safety violations by at least two orders of magnitude compared to other scalable safe RL baselines. The safety filter leads to near-elimination of catastrophic failures throughout the control training phases, a claim verified under multiple random seeds on both tasks.

  Figure 4: Learning curves—control return (top) and log-scaled accumulated failures (bottom); DYNASAUR (green) achieves strong return with two orders of magnitude fewer failures than baselines.

  

  Figure 5: Distributional shift in explored states—initial model data (blue) vs. final filtered policy coverage (green); DYNASAUR effectively expands safe/certain coverage over training.

Ablation and Design Insights

Additional ablation on CartPole reveals critical dependence on the proposed hyperplane action space and sampling strategies:

• Direct $w,b$ parametrization (as in [black-box-lavankul]): Fails to produce reliable or broad safety filters, leading to frequent training failures.
• Omitting filter action regularization: Results in overly conservative filtering, limiting control performance.
• Ignoring state-distribution shaping: Increases failures and reduces robustness across seeds.

  Figure 6: Ablation on key filter policy design decisions; the proposed parametrization and distribution shaping are necessary for joint safety and performance.

Analysis of the final safety filter reveals expected structure—the filter admits a wide viable set near the goal and upright pole configurations but becomes sharply restrictive near boundaries or at large velocities.

Figure 7: Visualization of state-conditioned safety filter—regions of white (pass-through actions) shrink as the agent approaches high-risk state-space edges.

Theoretical and Practical Implications

This work establishes that viability-based safety can be scaled to high-dimensional deep RL with learned models, with no need for explicit domain dynamics, prior safe sets, or terminal state design. The filter parametrization admits stable, expressive neural RL, unlike previous formulations. Empirical results show safety and return are compatible in model-based RL, contradicting some prior findings which reported substantial trade-offs or sample inefficiency.

Practically, DYNASAUR opens the door to safe RL in hardware-closed-loop settings (e.g., robotics, nonlinear process control, autonomous vehicles) with only minimal expert input (knowledge of failure states and initial data), avoiding the prohibitive manual engineering or computational detours of Hamilton–Jacobi, CBF, or reachability analyses. Theoretically, it demonstrates that RL-based safety filter learning can be robustified via uncertainty-aware models and principled action-space design, even under epistemic uncertainty.

Limitations and Future Directions

While DYNASAUR scales well on moderately high-dimensional tasks (MuJoCo Walker), exploration and expansion of the certain set $E$ become less efficient with increasing task complexity or dimensionality. Further advances in uncertainty quantification, targeted model learning, and efficient coverage of rare but critical state-action regions are required for more complex systems. Integrations with additional model-based planning, improved epistemic uncertainty metrics, or adaptive filter learning objectives are promising avenues.

Conclusion

DYNASAUR provides a unified, scalable solution for safe RL with hard constraints during both training and deployment, leveraging joint uncertainty-aware modeling and principled safety filter learning. It achieves practical and theoretical safety performance with strong sample efficiency and minimal prior knowledge, making it a compelling candidate for real-world safe RL deployments and further AI safety research.