$λ$-Reachability: Geometric-Horizon Safety Bellman Equations for Humanoid Safety

Abstract: We introduce $λ$-Reachability, a scalable approach to Hamilton--Jacobi safety analysis for high-dimensional robotic systems. Unlike prior discounted formulations that rely on fixed one-step Bellman updates, $λ$-Reachability employs a stochastic multi-step estimator of the safety value, using a geometrically distributed rollout horizon together with a randomly absorbed terminal. Conceptually analogous to TD($λ$), $λ$-Reachability interpolates between local self-consistency updates and long-horizon max-over-trajectory safety targets via an interpretable horizon-control parameter. Unlike TD($λ$), where the terminal value is always incorporated in learning targets, the terminal safety value in $λ$-Reachability is only used at a probability controlled by parameter $δ$. We formally show that for $δ<1$, the update induces a contraction mapping that allows temporal-difference learning; as $λ\to 1$, the estimator recovers the undiscounted reachability objective. We apply $λ$-Reachability to high-dimensional safety learning problems with both simulated and real humanoid robots under balance and collision avoidance constraints. Experimental results demonstrate that $λ$-Reachability significantly improves both safe-set boundary classification and safety margin estimation compared to single-step temporal-difference baselines.

• The paper introduces λ-Reachability, a geometric-horizon safety Bellman approach that quantifies future safety in high-dimensional humanoid robots.
• It demonstrates significant improvements in early violation prediction and reduced false positives in balance-critical and collision-avoidance tasks through multi-step value propagation.
• The method unifies elements of control barrier functions and Hamilton-Jacobi reachability, enabling scalable, efficient, and real-time safety certification in perception-rich environments.

$\lambda$-Reachability for High-Dimensional Humanoid Safety

Motivation and Problem Formulation

The paper "$\lambda$-Reachability: Geometric-Horizon Safety Bellman Equations for Humanoid Safety" (2606.16022) introduces a scalable safety value learning approach for high-dimensional robotic systems, with an emphasis on humanoid robots in balance-critical and collision-avoidance tasks. The central problem addressed is the synthesis and inference of safety value functions, $V^\pi(x)$, that quantify the future safety of a policy $\pi$ with respect to a state-wise safety signal $\ell(x)$, typically defined so that $\ell(x)>0$ corresponds to a violation.

Persistent safety, or the guarantee that a robot remains in the safe set $X_S = \{ x \mid \ell(x) \leq 0 \}$ for all future trajectory steps, is foundational in safety-critical control. The safety value function formalizes this as a worst-over-future metric:

$V^\pi(x_t) = \sup_{k \ge t} \ell(x_k)$

which is the principal object in Hamilton-Jacobi (HJ) reachability and control barrier function (CBF) synthesis.

Limitations of Traditional Bellman Approaches

Standard approaches—such as sum-of-squares (SOS) programming, neural barrier function fitting, and PDE-based HJ solvers—scale poorly in high dimensions. Neural CBF approaches do not guarantee global satisfaction of barrier conditions without expensive continuous-state verification. Grid-based HJ methods are intractable due to exponential scaling. Discounted safety Bellman operators, enabling TD/Q-learning [fisac2019bridging], overcome the contraction deficiency of the undiscounted safety Bellman but propagate information inefficiently because updates are one-step bootstraps, leading to slow and potentially unstable learning in high-dimensional, perception-rich tasks.

The paper argues that Monte Carlo supervision or multi-step targets would increase value accuracy and propagation efficiency, but complete trajectories are impractical in online learning and real hardware environments.

$\lambda$-Reachability: Stochastic Multi-Step Safety Bellman

To address these deficiencies, the authors propose $\lambda$-Reachability, a stochastic multi-step estimator analogous to TD($\lambda$0) [sutton1988td], but tailored to the safety value context. The approach defines a geometric-horizon Bellman operator with a randomly absorbed terminal, controlled by parameters $\lambda$1 (effective lookahead) and $\lambda$2 (probability of bootstrapping from the value function at terminal):

$\lambda$3

where $\lambda$4 is geometrically distributed ($\lambda$5), and the terminal is randomly absorbed with probability $\lambda$6. This interpolates between one-step TD ($\lambda$7), long-horizon bootstrapping ($\lambda$8), and Monte Carlo evaluation. The approach is formalized and proven to induce a contraction mapping under suitable parameter choices, enabling temporal-difference learning convergence.

Theoretical Properties

The stochastic multi-step Bellman operator admits a contraction under $\lambda$9, $V^\pi(x)$0, and as $V^\pi(x)$1, recovers the undiscounted reachability objective. Critically, the decoupling of contraction and value approximation accuracy via independent $V^\pi(x)$2 and $V^\pi(x)$3 enables annealing towards unbiased high-horizon estimates without sacrificing training stability, unlike classic discounted or bootstrapped approaches.

Experimental Evaluation: Simulation and Hardware

The method is benchmarked on three safety tasks (G1Flat, G1Rough, G1Collision) using a high-dimensional Unitree G1 humanoid, with complex balance and collision-avoidance objectives. Policies are trained in RL environments with rich observations, including proprioceptive and perception-based inputs.

Key quantitative outcomes:

• For $V^\pi(x)$4, $V^\pi(x)$5-Reachability achieves temporal recall $V^\pi(x)$6 nearly matching supervised oracle baselines, minimal value error, and rare false positives across all tasks.
• Ablation on $V^\pi(x)$7 shows dramatic degradation as the horizon shrinks ($V^\pi(x)$8), confirming the necessity of long-horizon backup for accurate safety margin and boundary inference.
• Discounted Policy Evaluation (DPE) baselines, reliant on one-step updates, cannot propagate unsafe event information efficiently, resulting in substantially higher error and missed early warnings.

  Figure 1: Safety value inference on hardware; $V^\pi(x)$9 predicts impending violations in advance, with the predictive horizon visualized by the gap between $\pi$0 and $\pi$1.

In hardware deployment, $\pi$2-Reachability, trained in simulation and applied to physical data, maintains superior early warning and low error metrics compared to DPE, even under domain shifts and sensor noise, though optimal $\pi$3 value must be traded off against robustness.

Figure 2: Humanoid safety tasks visualized; color coding indicates proximity to violation, with $\pi$4-Reachability enabling accurate margin prediction.

Geometric-Horizon Control and Safety Set Classification

$\pi$5 provides an explicit horizon control, directly mapping to a policy's predictive safety horizon. The approach unifies CBF-style invariance supervision and HJ local consistency, allowing interpolation from offline, global safety certificates to efficient online learning.

Figure 3: G1Flat task setup illustrating high-dimensional proprioceptive state for balance-critical locomotion.

Implications for Scalable Safe Robotics

The proposed method substantially improves margin and boundary estimation in high-dimensional, perceptual robotic systems, enabling scalable safety certification and monitoring suitable for both simulated and real-world deployment. The geometric-horizon mechanism supports tractable online learning from partial trajectories and robust value propagation for complex robot-environment interactions.

Practically, $\pi$6-Reachability can be integrated into actor-critic RL algorithms to explicitly maximize safety value, thereby refining safe sets and failure recovery triggers in policy optimization. The accurate safety margins derived facilitate the correct scaling of safety-constrained operations (e.g., barrier gradients for safe control, collision alarms, robust constraint enforcement), critical for real-time safety in dynamic manipulation, navigation, and human-robot interaction.

Figure 4: Additional hardware experiment results demonstrate predictive safety margin estimation under external disturbances.

Conclusion

$\pi$7-Reachability introduces a theoretically sound and empirically validated approach to scalable safety value learning for high-dimensional robotic systems. By enabling geometric-horizon interpolation between local and global max-backups, it achieves efficient, accurate propagation of safety information and early warning of violations, outperformed traditional one-step bootstrapped baselines both in simulation and hardware. The methodology opens new directions for generalizing safe control and RL in humanoids and complex perceptual environments, with further research needed on bias-robustness tradeoffs in truncated horizon sampling and adaptation to noisy online settings.