DeepCache: Accelerating Diffusion Models for Free

Abstract: Diffusion models have recently gained unprecedented attention in the field of image synthesis due to their remarkable generative capabilities. Notwithstanding their prowess, these models often incur substantial computational costs, primarily attributed to the sequential denoising process and cumbersome model size. Traditional methods for compressing diffusion models typically involve extensive retraining, presenting cost and feasibility challenges. In this paper, we introduce DeepCache, a novel training-free paradigm that accelerates diffusion models from the perspective of model architecture. DeepCache capitalizes on the inherent temporal redundancy observed in the sequential denoising steps of diffusion models, which caches and retrieves features across adjacent denoising stages, thereby curtailing redundant computations. Utilizing the property of the U-Net, we reuse the high-level features while updating the low-level features in a very cheap way. This innovative strategy, in turn, enables a speedup factor of 2.3$\times$ for Stable Diffusion v1.5 with only a 0.05 decline in CLIP Score, and 4.1$\times$ for LDM-4-G with a slight decrease of 0.22 in FID on ImageNet. Our experiments also demonstrate DeepCache's superiority over existing pruning and distillation methods that necessitate retraining and its compatibility with current sampling techniques. Furthermore, we find that under the same throughput, DeepCache effectively achieves comparable or even marginally improved results with DDIM or PLMS. The code is available at https://github.com/horseee/DeepCache

• The paper introduces a training-free approach that caches high-level U-Net features to reduce redundant computations in diffusion models.
• It achieves up to 2.3× speedup on Stable Diffusion and 4.1× on LDM models while maintaining near-original quality.
• The method is compatible with fast samplers and suggests a modular acceleration pathway for efficient generative modeling.

DeepCache: Training-Free Acceleration of Diffusion Models via Feature Caching

Introduction

Diffusion models (DMs) have established themselves as state-of-the-art generative models for tasks ranging from image synthesis to text-to-image and beyond. However, their prevalent sequential reverse denoising process, inherent to models such as DDPM, LDM, and Stable Diffusion, imposes substantial computational costs and latency during inference, presenting challenges to their adoption in resource-constrained and real-time scenarios. Traditional strategies for alleviating this computational burden include sampling step reduction, model pruning, distillation, and quantization, most of which introduce supplementary training or require access to large datasets and compute resources. The paper "DeepCache: Accelerating Diffusion Models for Free" (2312.00858) introduces a distinct, training-free paradigm for accelerating DMs by exploiting temporal redundancy in U-Net-based diffusion architectures.

Methodology

The fundamental insight motivating DeepCache is the high temporal similarity observed in high-level features of the U-Net across adjacent denoising steps. Empirical analysis conducted across multiple diffusion models demonstrates that certain upsampling block features remain nearly invariant between consecutive timesteps. DeepCache leverages this by caching the outputs of selected upsampling blocks during one full forward pass (the "cache step") and reusing these cached features during subsequent denoising iterations (the "retrieve steps"), performing only shallow partial inference for the intervening steps.

The acceleration process is formalized as alternating full and partial U-Net inferences, where for an interval $N$, the main branch features (high-level) are recomputed only once and then reused for $N-1$ steps, during which only the lightweight downsampling blocks and relevant skip connections are computed afresh. This enables skipping the majority of the U-Net's computational graph across a substantial fraction of the denoising trajectory. The approach also introduces a non-uniform interval selection mechanism to address the empirical observation that temporal similarity decays non-uniformly across the denoising process, adjusting the cache refresh schedule accordingly to balance computational efficiency with sample fidelity.

Empirical Evaluation

Acceleration and Quality Retention

DeepCache achieves a speedup of up to $2.3\times$ on Stable Diffusion v1.5 (50 PLMS steps) with a negligible 0.05 decline in CLIP Score, and $4.1\times$ on LDM-4-G (250 DDIM steps) with only a marginal increase (0.22) in FID on ImageNet. On CIFAR-10, LSUN-Bedroom, and LSUN-Churches, DeepCache surpasses post-training pruning and distillation approaches, maintaining superior FID and efficiency without any retraining overhead.

Compatibility and Comparisons

The method is inherently compatible with existing fast samplers (e.g., DDIM, PLMS). When controlling for throughput rather than step count, DeepCache produces comparable or marginally improved generation metrics relative to DDIM and PLMS. For example, at the same throughput, DeepCache with $N=3$ matches DDIM with 91 steps on LDM-4-G in terms of FID, while offering higher acceleration and lower inference cost.

Structural Analysis and Ablations

DeepCache's efficacy is shown to be highly sensitive to the choice of skip branch for caching; selecting shallow branches offers greater speedups but incurs minimal fidelity degradation for moderate caching intervals ($N<5$). Ablation studies verify that cached high-level features are nontrivially beneficial for denoising, and that shallow partial inferences with cached features significantly outperform zero-initialized baselines. The trade-off between speed and sample quality is controllable via $N$ and the branch selection.

Limitations

DeepCache's acceleration upper bound is constrained by the structure of the underlying U-Net. If the shallowest skip branch accounts for a large proportion of network FLOPs, achievable speedups are limited. Performance degrades for large cache intervals ($N\gg10$), as high-level feature staleness becomes significant, particularly in later denoising stages with larger feature variations.

Theoretical and Practical Implications

DeepCache delineates a new axis for efficient inference in diffusion models: exploiting intra-trajectory feature redundancy rather than modifying model weights or architecture via retraining. This differs from distillation and pruning pipelines, making it particularly suitable for deployment with large, fixed pre-trained models (e.g., widely distributed Stable Diffusion checkpoints) where retraining is undesirable or infeasible. The approach interacts synergistically with fast solvers and step reduction methods, suggesting a modularity in DM acceleration pipelines that is orthogonal to prior art.

Future Directions

Given the robustness of DeepCache across models and datasets, prospective extensions include automated search for optimal cache schedules and branch selection, dynamic feature similarity analysis for adaptive caching, or hardware-aware implementations to maximize batch throughput. Integration with quantization or other lightweight architectures may push acceleration further in real-time and edge settings. The caching principle itself may inspire analogous mechanisms in other sequential generative frameworks or even autoregressive transformers.

Conclusion

DeepCache introduces a mechanism-level acceleration technique for diffusion model inference, achieving significant computational gains without retraining or architectural modification. By strategically caching and reusing temporally local high-level U-Net features, DeepCache realizes fast, high-fidelity generation and demonstrates compatibility with other acceleration methodologies. It represents an advance in practitioner-friendly, training-free DM acceleration and opens up new possibilities for efficient generative modeling and deployment (2312.00858).