Adaptive Spectral Feature Forecasting for Diffusion Sampling Acceleration

Abstract: Diffusion models have become the dominant tool for high-fidelity image and video generation, yet are critically bottlenecked by their inference speed due to the numerous iterative passes of Diffusion Transformers. To reduce the exhaustive compute, recent works resort to the feature caching and reusing scheme that skips network evaluations at selected diffusion steps by using cached features in previous steps. However, their preliminary design solely relies on local approximation, causing errors to grow rapidly with large skips and leading to degraded sample quality at high speedups. In this work, we propose spectral diffusion feature forecaster (Spectrum), a training-free approach that enables global, long-range feature reuse with tightly controlled error. In particular, we view the latent features of the denoiser as functions over time and approximate them with Chebyshev polynomials. Specifically, we fit the coefficient for each basis via ridge regression, which is then leveraged to forecast features at multiple future diffusion steps. We theoretically reveal that our approach admits more favorable long-horizon behavior and yields an error bound that does not compound with the step size. Extensive experiments on various state-of-the-art image and video diffusion models consistently verify the superiority of our approach. Notably, we achieve up to 4.79$\times$ speedup on FLUX.1 and 4.67$\times$ speedup on Wan2.1-14B, while maintaining much higher sample quality compared with the baselines.

• The paper's main contribution is the use of Chebyshev polynomial expansions to forecast latent features, enabling significant acceleration in diffusion sampling.
• It employs adaptive scheduling and ridge regression to balance speed and fidelity, drastically reducing expensive denoiser evaluations.
• Empirical results show up to 4.79× speedup in text-to-image and text-to-video tasks while maintaining high quality compared to prior baselines.

Adaptive Spectral Feature Forecasting for Diffusion Sampling Acceleration

Motivation and Context

Diffusion models have become the principal paradigm for high-fidelity image and video synthesis, but their practical deployment is bottlenecked by high latency due to iterative denoiser evaluations, especially in transformer-based architectures. Feature caching and local approximation methods promise training-free acceleration by skipping expensive network evaluations and reusing cached intermediate features. However, current designs remain fundamentally constrained by rapidly compounding prediction errors and degraded sample quality under aggressive acceleration. This paper presents a global spectral forecasting approach—Spectrum—which approximates diffusion features as functions over time using Chebyshev polynomials, yielding superior long-range prediction accuracy and robust sample fidelity even at extreme speedups.

Methodology

The core innovation is the application of spectral approximation for forecasting diffusion features. In detail, each latent feature channel of the denoiser output is viewed as a function over time along the diffusion trajectory. Spectrum fits each such function via a truncated Chebyshev polynomial expansion, with coefficients determined through efficient ridge regression on the feature cache. The theoretical rationale is that Chebyshev bases provide a global, well-conditioned representation with error bounds independent of forecasting step size, thus addressing the error amplification seen in Taylor and finite-difference (local) forecasters.

Algorithmically, Spectrum alternates between actual network evaluations at selected timesteps—where features are cached and coefficients recomputed—and inexpensive spectral forecasts at skipped timesteps, dramatically reducing total denoiser calls. Adaptive scheduling is employed to concentrate full evaluations in early diffusion steps, mitigating error accumulation and optimizing the tradeoff between speed and quality.

Theoretical Analysis

Spectrum's forecasting error is analyzed in terms of uniform approximation bounds (Bernstein ellipses), showing exponential decay with respect to the Chebyshev degree. Critically, the error does not scale with the forecasting horizon, contrasting with the $(j-k)^{P+1}$ scaling identified in local Taylor expansion (see Theorem 1 and Theorem 2). This implies that feature forecasting can be reliably applied across long temporal skips, enabling high acceleration ratios with bounded quality loss.

The regularization in fitting (ridge regression) is shown to be necessary for numerical stability and avoidance of overfitting, as confirmed by ablation studies.

Empirical Evaluation

Extensive experiments are conducted on state-of-the-art text-to-image (FLUX.1, Stable Diffusion 3.5-Large) and text-to-video (Wan2.1-14B, HunyuanVideo) diffusion models. The evaluation employs both classical reconstruction metrics (PSNR, SSIM, LPIPS), semantic fidelity scores (ImageReward, CLIPScore), and video-specific metrics (VBench).

Spectrum achieves up to 4.79$\times$ speedup on FLUX.1 and 4.67$\times$ speedup on Wan2.1-14B, consistently maintaining high sample quality relative to reference samplers and outperforming prior caching baselines such as TaylorSeer, FORA, ToCa, and TeaCache, especially as the number of network evaluations is further reduced.

Figure 1: Qualitative comparison on text-to-image generation with FLUX.1: Spectrum nearly matches the reference quality at $4.79\times$ acceleration, while baselines fail on color and prompt consistency.

Figure 2: Qualitative comparisons on text-to-video generation (HunyuanVideo): Spectrum delivers high fidelity at much higher speedup than local predictors.

Figure 3: Text-to-video generation with Wan2.1-14B: Spectrum aligns with the reference using only 14 network evaluations; TaylorSeer introduces artifacts.

Strong numerical results validate the theoretical error claims: Spectrum's forecasted latent features are significantly closer to oracle values (see RMSE tables), and adaptive scheduling further enhances quality compared to uniform cache intervals.

Ablation and Analysis

Comprehensive ablations demonstrate the impact of Chebyshev degree $M$, regularization weight $\lambda$, and scheduling strategies. Increasing $M$ improves forecast fidelity up to a moderate threshold ($M=4$), after which returns diminish. The regularization weight must be finely tuned to balance overfitting and stability.

Caching only the last attention block (as opposed to layer-wise caching) yields equal or improved sample quality with reduced memory and computational overhead.

Figure 4: Ablation study on $\lambda$ demonstrates critical impact on stability and quality.

Figure 5: Chebyshev degree ablation shows rapid improvement up to $M=4$.

Implications and Future Directions

Spectrum establishes a new direction in diffusion sampling acceleration: global spectral forecasting enables higher throughput with negligible degradation in sample quality and is applicable across diverse diffusion architectures (including U-Net-based SDXL). Its training-free, model-agnostic design allows seamless combination with other efficiency techniques (pruning, quantization, token reduction, parallelization). The theoretical bounds suggest feasibility for real-time generation and deployment in interactive systems.

In terms of future development, further exploration of spectral function bases, sparse adaptive caching, multi-modal integration, and hardware-aware algorithm design may yield additional improvements. Combining Spectrum with distillation-based and ODE-solver acceleration techniques remains promising for even more aggressive inference cost reduction.

Conclusion

Spectrum introduces adaptive spectral feature forecasting using Chebyshev polynomials for diffusion model sampling acceleration, achieving superior speed/quality tradeoffs compared to previous local caching forecasters. Theoretical analysis and empirical results confirm tightly controlled approximation errors, enabling robust sample fidelity under extreme inference budget constraints. The approach provides a solid foundation for highly efficient, real-time generative modeling.

Figure 6: Spectrum generates high-quality text-to-video samples at $3.5\times$ speedup using only 14 network evaluations, without quality degradation.