SenCache: Accelerating Diffusion Model Inference via Sensitivity-Aware Caching

Abstract: Diffusion models achieve state-of-the-art video generation quality, but their inference remains expensive due to the large number of sequential denoising steps. This has motivated a growing line of research on accelerating diffusion inference. Among training-free acceleration methods, caching reduces computation by reusing previously computed model outputs across timesteps. Existing caching methods rely on heuristic criteria to choose cache/reuse timesteps and require extensive tuning. We address this limitation with a principled sensitivity-aware caching framework. Specifically, we formalize the caching error through an analysis of the model output sensitivity to perturbations in the denoising inputs, i.e., the noisy latent and the timestep, and show that this sensitivity is a key predictor of caching error. Based on this analysis, we propose Sensitivity-Aware Caching (SenCache), a dynamic caching policy that adaptively selects caching timesteps on a per-sample basis. Our framework provides a theoretical basis for adaptive caching, explains why prior empirical heuristics can be partially effective, and extends them to a dynamic, sample-specific approach. Experiments on Wan 2.1, CogVideoX, and LTX-Video show that SenCache achieves better visual quality than existing caching methods under similar computational budgets.

• The paper introduces a principled caching strategy based on the denoiser’s local sensitivity, enabling dynamic and sample-specific cache reuse.
• It leverages Jacobian norms of latent variables and timesteps to trigger caching, thereby substantially reducing the number of expensive forward passes.
• Empirical results show that SenCache achieves significant reductions in NFE and improves visual quality compared to traditional heuristic-based approaches.

Sensitivity-Aware Caching for Efficient Diffusion Model Inference

Overview

The paper "SenCache: Accelerating Diffusion Model Inference via Sensitivity-Aware Caching" (2602.24208) presents a theoretically grounded framework for cache-based acceleration of diffusion model inference. Standard diffusion generators, especially video diffusion transformers, require numerous sequential denoising steps, each involving costly forward passes through large neural networks. Existing caching approaches select cache/reuse timesteps using empirical heuristics, but these lack theoretical justification, require extensive hyperparameter tuning, and are statically applied to all samples. SenCache introduces a principled, dynamic policy: caching is triggered based on the denoiser's local sensitivity to input perturbations, offering stronger guarantees and improved fidelity across a range of video diffusion models.

Sensitivity as the Foundation for Caching Decisions

SenCache's central premise is that the denoiser’s local sensitivity—characterized by the Jacobian norms with respect to the noisy latent variable $\mathbf{x}_t$ and timestep $t$—acts as a proxy for estimating output change between denoising steps. The framework formalizes a sensitivity score:

$$S_t = \|J_x\| \, \|\Delta \mathbf{x}_t\| + \|J_t\| \, |\Delta t|,$$

where $J_x$ and $J_t$ are the Jacobians of the network outputs with respect to the latent and timestep, and $\Delta \mathbf{x}_t$, $\Delta t$ are their respective perturbations between steps. Cache reuse occurs whenever $S_t$ is below a user-defined tolerance $\varepsilon$, thus dynamically adapting cache decisions to each sample according to the network's local stability.

This avoids pitfalls of methods relying on single-term proxies (e.g., magnitude or time embedding differences), which can fail when unmodeled sensitivity components are substantial. The approach is agnostic to model architecture, sampling schedule, and can be calibrated using a small, diverse video set.

Figure 1: SenCache utilizes sensitivity scores as the criterion for cache-based inference acceleration in diffusion models.

Empirical Analysis of Sensitivity Dynamics

To validate the importance of both latent and timestep sensitivities, extensive sensitivity analyses across SiT-XL/2 checkpoints show substantial and varying Jacobian norms for both $\mathbf{x}_t$ and $t$ as a function of the denoising trajectory. This substantiates the necessity of a joint criterion for cache decisions.

Figure 2: Left: Norm of the Jacobian w.r.t. the noisy latent. Right: Norm w.r.t. the timestep—both contribute significantly to output variation.

Calibration with only 8 diverse samples offers sensitivity estimates highly comparable to those obtained from much larger datasets (4096 videos). This highlights the practical efficiency and stability of sensitivity-based caching.

Figure 3: Sensitivity profiles from calibration sets of 8 and 4096 videos; small sets suffice for accurate estimates.

Quantitative Results and Comparative Evaluation

SenCache is evaluated on Wan 2.1, CogVideoX, and LTX-Video—state-of-the-art video diffusion models—against recent caching baselines (TeaCache and MagCache). Both efficiency (NFE, Cache Ratio) and visual quality (LPIPS, SSIM, PSNR) metrics are reported. Under identical compute budgets, SenCache achieves consistently superior visual quality, especially in aggressive caching regimes, where the gap in LPIPS and SSIM widens compared to MagCache and TeaCache.

The sensitivity-aware approach also brings interpretability: prior heuristics are shown to be special cases that can either over-cache challenging samples or under-cache easy ones due to static, monolithic triggers. In contrast, SenCache’s dynamic adaptation per sample leverages actual input-output responsiveness, minimizing both over-caching and cumulative approximation error.

Figure 4: Mean absolute error between denoiser outputs at consecutive timesteps, illuminating model-specific sensitivity differences.

Ablation and Cross-Model Insights

Parameter ablations reveal sharp accuracy-efficiency trade-offs: increasing the cache lifetime $n$ and error threshold $\varepsilon$ reduces NFE, but over-extension leads to quality degradation due to less accurate local approximations. Small calibration sets suffice for robust sensitivity estimation, and the sensitivity profile varies significantly across models and timesteps, suggesting that dynamic scheduling of $\varepsilon$ and cache lifetime could further optimize speed-quality tradeoff.

Additional efficiency metrics indicate substantial wall-clock speedups and compute reductions compared to prior baselines. Visualizations show that in critical denoising regimes, certain models exhibit markedly higher sensitivity, necessitating higher tolerance for comparable NFE reductions but potentially incurring more visual artifacts.

Practical and Theoretical Implications

SenCache’s framework enhances practical deployability of diffusion generators without retraining or architecture modification, making it suitable for latency-critical applications and modalities beyond video (e.g., audio, motion). Theoretically, it provides a foundation for adaptive caching policies grounded in the model's local Lipschitz behavior, bridging the gap between empirical heuristics and robust decision-making. The approach is extensible: higher-order sensitivity estimators or learned surrogates could further refine accuracy, and per-timestep or globally optimized tolerances can be integrated for even finer control.

Conclusion

SenCache establishes a principled mechanism for cache-based diffusion model acceleration, leveraging sensitivity both to the noisy latent and timestep as the criterion for safe reuse. This results in dynamic, sample-specific caching schedules that robustly minimize approximation error, directly improving the speed-quality envelope for generative architectures. By unifying and generalizing prior approaches, the framework sets a foundation for future research on adaptive inference schedules, model-agnostic acceleration, and cross-modal diffusion system optimizations.