From Reusing to Forecasting: Accelerating Diffusion Models with TaylorSeers

Abstract: Diffusion Transformers (DiT) have revolutionized high-fidelity image and video synthesis, yet their computational demands remain prohibitive for real-time applications. To solve this problem, feature caching has been proposed to accelerate diffusion models by caching the features in the previous timesteps and then reusing them in the following timesteps. However, at timesteps with significant intervals, the feature similarity in diffusion models decreases substantially, leading to a pronounced increase in errors introduced by feature caching, significantly harming the generation quality. To solve this problem, we propose TaylorSeer, which firstly shows that features of diffusion models at future timesteps can be predicted based on their values at previous timesteps. Based on the fact that features change slowly and continuously across timesteps, TaylorSeer employs a differential method to approximate the higher-order derivatives of features and predict features in future timesteps with Taylor series expansion. Extensive experiments demonstrate its significant effectiveness in both image and video synthesis, especially in high acceleration ratios. For instance, it achieves an almost lossless acceleration of 4.99$\times$ on FLUX and 5.00$\times$ on HunyuanVideo without additional training. On DiT, it achieves $3.41$ lower FID compared with previous SOTA at $4.53$$\times$ acceleration. %Our code is provided in the supplementary materials and will be made publicly available on GitHub. Our codes have been released in Github:https://github.com/Shenyi-Z/TaylorSeer

• The paper presents TaylorSeer’s cache-then-forecast strategy that uses Taylor series expansion to accurately predict future features in diffusion models.
• It demonstrates near-lossless acceleration ratios (up to 5×) in models like FLUX, HunyuanVideo, and DiT-XL/2, maintaining high image and video quality.
• The study paves the way for efficient real-time applications and further research in sequence modeling and computational forecasting.

Accelerating Diffusion Models with TaylorSeers

Introduction

The paper "From Reusing to Forecasting: Accelerating Diffusion Models with TaylorSeers" (2503.06923) addresses the computational inefficiency of Diffusion Transformers (DiT) in high-fidelity image and video synthesis. Diffusion Models (DMs) have achieved remarkable advancements in generative AI but are hindered by their resource-intensive nature, especially for real-time applications. The prevalent strategy, feature caching, aims to expedite DMs but faces limitations due to decreased feature similarity over substantial timestep intervals. This study introduces TaylorSeer, an innovative approach that utilizes Taylor series expansion to forecast features, thus enabling high-ratio acceleration without compromising quality.

Methodology

Cache-then-Forecast Paradigm

TaylorSeer marks a departure from the conventional "cache-then-reuse" methodology by proposing a "cache-then-forecast" strategy. This approach leverages the smooth, continuous evolution of features across timesteps to predict future states more accurately. By employing Taylor series expansion, which approximates features using higher-order derivatives, TaylorSeer can effectively estimate features at upcoming timesteps based on preceding data.

Linear Prediction and Higher-Order Extensions:

• Linear Prediction: TaylorSeer initially adopts a linear prediction model, caching both feature values and their temporal differences, which aids in capturing linear trends in feature trajectories between timesteps.
• Higher-Order Prediction: To further refine accuracy, TaylorSeer incorporates higher-order Taylor expansions. Utilizing multi-step features to approximate derivatives, the model captures nonlinear feature dynamics across significant intervals, enhancing prediction accuracy.

  Figure 1: An overview of TaylorSeer, detailing its capabilities from naive feature caching to higher-order finite difference modeling.

Experimental Results

The empirical studies span across various models, including FLUX, HunyuanVideo, and DiT-XL/2, demonstrating TaylorSeer's prowess in maintaining output quality at impressive acceleration ratios.

Image and Video Synthesis Evaluation

• FLUX and HunyuanVideo: TaylorSeer achieves a near-lossless acceleration of 4.99× and 5.00×, respectively, on these platforms. Performance metrics such as FID, PSNR, and SSIM indicate that TaylorSeer maintains or even improves image quality compared to baseline full-resolution runs.

  Figure 2: Detailed visualization results on FLUX. TaylorSeer's output preserves detail and color fidelity where competitors falter under acceleration constraints.

• DiT-XL/2: In class-conditional image generation, TaylorSeer reduces inference latency while sustaining high FID scores even at a 4.53× acceleration. Higher-order Taylor expansions contribute significantly to this success, offering a balance between the number of computational steps and the model's fidelity and diversity of outputs.

Theoretical Implications and Future Directions

TaylorSeer's introduction of Taylor series into the field of feature caching uncovers a novel mechanism for addressing the computational inefficiencies intrinsic to diffusion models. This paradigm not only extends the practicality of DMs in real-time applications but also opens pathways for further theoretical exploration in sequence modeling and forecasting. Future research might explore integrating machine learning models to dynamically adjust the order of Taylor expansions based on real-time computational constraints or feature evolution patterns.

Conclusion

TaylorSeer represents a pivotal innovation in accelerating diffusion models, transitioning from merely caching features to forecasting them with mathematical rigor. By expanding upon traditional caching methodologies, TaylorSeer significantly optimizes computational efficiency without sacrificing quality, establishing a new benchmark for future developments in diffusion-based generative models.