Sensitivity Shaping for Latent Modeling

Abstract: Generative dynamics models enable planning in challenging robotic systems, but safe deployment requires reliably detecting policy-induced out-of-distribution (OOD) transitions. Existing methods typically treat the learned dynamics as fixed and attach post hoc support surrogates. We show that these surrogates can fail when the dynamics are locally insensitive to critical action choices: unsupported control actions may produce latent predictions that resemble demonstrated transitions, suppressing OOD signals despite large true predictive errors. To address this, we introduce support-conditioned control-sensitivity regularization, which promotes sensitive local response to control input changes in learned dynamics in high-support training regions. This preserves control-induced variation while limiting unstable extrapolation due to weak empirical support. Experiments in vision-based obstacle avoidance, manipulation, and real-robot navigation show improved OOD detection and safer closed-loop planning.

• The paper proposes a support-conditioned sensitivity regularizer that enforces a lower bound on the control-Jacobian norm, restoring latent responsiveness near safety boundaries.
• Empirical evaluations in obstacle avoidance, block plucking, and navigation demonstrate improved OOD detection, higher trajectory success rates, and reduced failure rates.
• The study highlights the trade-off between latent smoothness and sharp control sensitivity, emphasizing the need for scalable support quantification in generative dynamics.

Sensitivity Shaping in Latent Generative Dynamics Models: A Formal Essay

Introduction

"Sensitivity Shaping for Latent Modeling" (2606.14585) systematically analyzes the impact of control sensitivity within generative dynamics models, particularly focusing on safety-critical robotics applications. The authors elucidate that traditional latent dynamics, such as those used in world models, often exhibit local insensitivity to control inputs—especially near task boundaries or in failure-avoiding regimes. This phenomenon undermines the effectiveness of out-of-distribution (OOD) detection, as unsupported actions produce latent transitions that are spuriously similar to those induced by demonstrated controls, thereby masking true predictive errors.

Theoretical Framework and Problem Analysis

The paper frames the problem by considering discrete-time dynamical systems modeled via generative latent representations. The mapping from high-dimensional observations to latent states, coupled with parametric latent transition dynamics (e.g., RSSMs within Dreamer architectures), enables sample-efficient planning and control. However, deployment-time distribution shifts induce state-action queries outside the empirical training support, causing reliability issues—particularly when control diversity in critical regions is limited due to the necessity of safety (e.g., collision-free data collection).

The authors formally define control sensitivity using the Frobenius norm of the dynamical model's control-Jacobian. They demonstrate via vision-based obstacle avoidance that, in regions near safety boundaries, the model's learned dynamics respond weakly to control input perturbations. This insensitivity results from collapsed latent geometry: distinct controls yield nearly indistinguishable predicted transitions, contrary to the true system response.

Figure 1: Control-insensitive dynamics generate in-distribution-looking predictions for unsupported actions, suppressing OOD signals; regularization restores responsiveness for improved OOD detection.

Empirical results substantiate these findings, showing substantial divergence between model-predicted and ground-truth transitions under small control perturbations. Standard OOD surrogates (e.g., $k$NN, flow-matching densities, ensemble disagreement) fail to track predictive error reliably in these collapsed regimes.

Figure 2: Sensitivity analyses reveal two low-sensitivity regimes—sparse outer regions and near-obstacle areas with limited control diversity; vanilla dynamics deviate from true responses, while regularization reduces mismatch.

Methodology: Support-Conditioned Control Sensitivity Regularization

To mitigate control insensitivity, the authors introduce a support-conditioned sensitivity regularizer. This regularizer is applied only in high-support regions—identified using $k$NN latent distances—where empirical data adequately constrains the latent dynamics. The objective enforces a lower bound on the Frobenius norm of the control-Jacobian for these samples:

$$\mathcal{L}_{\mathrm{reg}} = \mathbb{E}_{(o_t, u_t) \sim \bar{\mathbb{D}},\, z_t \sim \phi_\theta(\cdot | o_t)} \left[ \left( g - \left\| \frac{\partial F_\theta(z_t, u_t)}{\partial u_t} \right\|_F^2 \right)_+ \right]$$

where $g$ is a task-dependent sensitivity margin and $(\cdot)_+$ denotes the positive part function. Hutchinson’s stochastic estimator is utilized to compute Frobenius norms efficiently, circumventing explicit Jacobian construction.

Empirical Evaluation: OOD Detection and Planning

Obstacle Avoidance

Sensitivity-regularized models recover substantive control responsiveness in the near-obstacle regime, leading to greater geometric separation between demonstrated and undemonstrated transitions in latent space.

Figure 3: Sensitivity-regularized dynamics exhibit nontrivial control responsiveness, producing clear geometric separation of in-distribution vs. OOD transitions in t-SNE/PCA projections.

Filtering random policies via OOD surrogates and safety thresholds show marked improvements in both trajectory success rates and reduced OOD violations, particularly for flow-matching-based surrogates. Ensemble disagreement, while responsive, is susceptible to spurious inter-model agreement and corresponding ranking errors.

Block Plucking Manipulation and Real-Robot Navigation

On a Franka block-plucking task, reachability-based filters trained on sensitivity-regularized dynamics achieve lower failure rates and reconstruction errors for both mediocre and reward-optimized policies. Conservative thresholding (higher $\alpha$) reduces failures but increases task incompletion, indicating the importance of calibrating OOD conservatism.

In real-robot static-obstacle avoidance, sensitivity shaping yields improvement under practical hardware constraints. Flow-matching surrogates paired with regularized dynamics deliver optimal safety performance; ensemble reachability filtering, although intervention-heavy, benefits significantly from the improved latent responsiveness.

Figure 4: Latent transition separation improves with increasing support fraction $\beta$; $\beta=0.3$ achieves geometric separation comparable to $\beta=0.6$.

Robust Numerical Results and Claims

Across all experiments, numerical results strongly support the claim that sensitivity-regularized dynamics recover robust control-induced variation in high-supported regions, yielding superior OOD detectability and safer planning. For instance, success rates in obstacle avoidance improve from 0.60 (vanilla, $k$NN) to 0.80 (regularized, $k$0NN), and from 0.91 (vanilla, flow-matching) to 0.96 (regularized, flow-matching). Block-plucking failure rates decrease from 0.196 (vanilla) to 0.150 (regularized), with corresponding reductions in reconstruction error.

Implications and Future Directions

Practically, these findings mandate sensitivity shaping as a prerequisite for reliable deployment of generative latent dynamics in safety-critical robotics. Theoretically, the work formalizes the latent geometry requirements for OOD surrogates, providing a structured approach for enforcing the local variability necessary to distinguish in-distribution from OOD transitions.

Future development avenues include:

• Scalable and adaptive support identification beyond $k$1NN, especially in high-dimensional observation spaces
• Extending sensitivity shaping to continuous visual-linguistic control policies, exploiting foundation model priors while retaining calibrated responsiveness
• Broader hardware validation for diverse robot platforms
• Theoretical analysis of the trade-offs between latent smoothness and sharp control sensitivity, particularly for adversarial extrapolation regimes

Conclusion

This paper formally identifies control insensitivity as a critical limitation within generative latent dynamics affecting OOD analysis and safe planning. The proposed support-conditioned sensitivity regularizer restores locally responsive latent geometry without destabilizing weakly-supported regions, leading to measurable improvements in OOD detection and downstream policy safety. Limitations include reliance on $k$2NN-based support estimates and periodic support updates, motivating further research in scalable representation learning and robust support quantification.