Learning Smooth Time-Varying Linear Policies with an Action Jacobian Penalty

Abstract: Reinforcement learning provides a framework for learning control policies that can reproduce diverse motions for simulated characters. However, such policies often exploit unnatural high-frequency signals that are unachievable by humans or physical robots, making them poor representations of real-world behaviors. Existing work addresses this issue by adding a reward term that penalizes a large change in actions over time. This term often requires substantial tuning efforts. We propose to use the action Jacobian penalty, which penalizes changes in action with respect to the changes in simulated state directly through auto differentiation. This effectively eliminates unrealistic high-frequency control signals without task specific tuning. While effective, the action Jacobian penalty introduces significant computational overhead when used with traditional fully connected neural network architectures. To mitigate this, we introduce a new architecture called a Linear Policy Net (LPN) that significantly reduces the computational burden for calculating the action Jacobian penalty during training. In addition, a LPN requires no parameter tuning, exhibits faster learning convergence compared to baseline methods, and can be more efficiently queried during inference time compared to a fully connected neural network. We demonstrate that a Linear Policy Net, combined with the action Jacobian penalty, is able to learn policies that generate smooth signals while solving a number of motion imitation tasks with different characteristics, including dynamic motions such as a backflip and various challenging parkour skills. Finally, we apply this approach to create policies for dynamic motions on a physical quadrupedal robot equipped with an arm.

• The paper introduces an action Jacobian penalty that directly reduces high-frequency control signals for smoother policy outputs.
• It proposes a Linear Policy Net architecture that decouples Jacobian computation, improving training efficiency and reducing tuning needs.
• Empirical results demonstrate improved stability, rapid convergence, and effective sim-to-real transfer across diverse robotic tasks.

Learning Smooth Time-Varying Linear Policies with an Action Jacobian Penalty: A Technical Synthesis

Introduction

The construction of smooth, physically plausible control policies in reinforcement learning for physics-based character animation and robotics is a persistent challenge. In "Learning Smooth Time-Varying Linear Policies with an Action Jacobian Penalty" (2602.18312), the authors propose a novel approach to mitigate the emergence of high-frequency, unnatural control signals in DRL-derived policies. Central to their method is the introduction of an action Jacobian penalty combined with a Linear Policy Net (LPN) architecture, yielding efficient, smooth controllers for both simulated humanoids and physical quadrupeds.

Action Jacobian Penalty: Direct Smoothness Regularization

Traditional approaches to enforce smoothness, such as action change penalties in the reward function or Lipschitz constraints, suffer from task sensitivity and significant tuning requirements. The action Jacobian penalty proposed in this work directly penalizes the Frobenius norm of the Jacobian of the policy's actions with respect to input state. This penalty regularizes the policy by reducing its sensitivity to high-frequency state fluctuations without requiring extensive per-task tuning or indirect reward balancing.

The total loss function for policy optimization is augmented as follows:

$$\mathcal L_\text{total} = \mathcal L_\text{PPO} + w_\text{Jac} \mathcal L_\text{Jac}$$

where $\mathcal L_\text{Jac} = \lVert \mathbf{J} \rVert^2$ with $\mathbf{J}$ being the Jacobian $\partial \bm{a}/\partial \bm{s}$. Penalizing this norm diminishes spurious high-frequency control signals and yields smoother action trajectories.

Linear Policy Net: Architecture and Benefits

To ameliorate the computational overhead introduced by the Jacobian penalty, the authors introduce the Linear Policy Net (LPN). In this architecture, a two-layer MLP, conditioned only on the reference state, outputs both a time-varying linear feedback matrix $K_t$ and a feedforward vector $\bm{k}_t$. The control action is then given by:

$$\bm{a}_t = K_t \bm{s}_t + \bm{k}_t + \hat{\bm{a}}_t$$

where $\bm{s}_t$ is the current character state and $\hat{\bm{a}}_t$ is the reference joint action. Crucially, in LPN the Jacobian $\partial \bm{a}_t / \partial \bm{s}_t$ reduces to $K_t$, decoupling the computation of the penalty from the state and leading to significant improvements in training efficiency.

Figure 1: Architecture of the Linear Policy Net, leveraging an MLP to generate a time-varying feedback matrix and feedforward action from the reference state.

Empirical Evaluation

The authors present an extensive empirical evaluation across a range of challenging motion imitation tasks—locomotion, gymnastics (e.g., backflip), complex environment interactions (e.g., parkour), and extended motion sequences (e.g., table tennis footwork). LPNs with the Jacobian penalty consistently outperform or match feedforward (FF) baselines, both in final reward and in smoothness metrics.

Figure 2: Comparative learning curves illustrating the training efficiency and final performance of various architectures and smoothness regularization approaches.

Quantitative smoothness metrics include average action change, high frequency spectral content (>10 Hz), and joint jerk. LPNs frequently attain best-in-class or near-best performance, notably requiring no custom per-task reward tuning and providing rapid policy convergence.

Figure 3: Visualization of ankle joint action trajectories for backflip and table tennis motions, highlighting the reduction in high-frequency oscillation when using smoothness regularization.

Notably, although LPNs restrict the expressiveness of the policy class to time-varying linear feedbacks, this structural constraint does not degrade— and sometimes improves—performance, especially for previously intractable or hard-to-tune tasks.

Sim-to-Real Transfer: Quadruped Robot Deployment

The utility of smooth time-varying linear controllers is further demonstrated by successful deployment on a physical Boston Dynamics Spot robot with an arm. Leveraging the LPN-derived feedback matrices, the physical controller achieves stable pacing with concurrent arm movements and dynamic hopping/arm-swinging behaviors. Precomputing and running the linear feedback matrices at lower frequencies (\SI{15}{Hz}) suffices, reducing onboard inference costs relative to FF DRL policies deployed at higher frequencies.

Theoretical and Practical Implications

From a theoretical viewpoint, this work highlights the sufficiency of time-varying linear feedback control policies for a broad set of challenging dynamic and contact-rich tasks. This finding has implications for model-based control, suggesting that, when coupled with DRL for robustness and data efficiency, simple control structures can match or surpass the performance of significantly more expressive networks.

On the practical front, the architecture's computational efficiency and smooth action outputs improve the viability of sim-to-real policy transfer, given bandwidth and actuation constraints.

Future Directions

Several avenues for advancement are identified:

• Robust linear policy generators: As LPN policies can be represented compactly, generative models (e.g., diffusion in policy space) could be used to automate skill synthesis for large motion libraries.
• Piecewise linear policies: Segmenting state space into regions with dedicated linear feedback could enhance expressivity and robustness, especially via ReLU-based neural partitioning.
• Hybrid model-based/DRL policy initialization: Combining the sample efficiency of DDP and the robustness of DRL by initializing LPNs from model-based solutions merits exploration.
• Generalization beyond motion imitation: Adapting these policies to adversarial imitation or domains lacking reference motion data remains largely unexplored.

Conclusion

This work demonstrates that action Jacobian penalties, when paired with Linear Policy Nets, enable efficient learning of smooth, time-varying linear feedback control policies for complex motion imitation. The resulting policies combine training efficiency, reduced computational overhead, smoother and more robust control signals, and strong sim-to-real transfer characteristics. The architectural findings challenge the necessity of highly expressive neural policies for many tasks, opening a path toward more structured, interpretable, and computationally efficient policy representations in deep RL for control (2602.18312).