DDiT: Dynamic Patch Scheduling for Efficient Diffusion Transformers

Abstract: Diffusion Transformers (DiTs) have achieved state-of-the-art performance in image and video generation, but their success comes at the cost of heavy computation. This inefficiency is largely due to the fixed tokenization process, which uses constant-sized patches throughout the entire denoising phase, regardless of the content's complexity. We propose dynamic tokenization, an efficient test-time strategy that varies patch sizes based on content complexity and the denoising timestep. Our key insight is that early timesteps only require coarser patches to model global structure, while later iterations demand finer (smaller-sized) patches to refine local details. During inference, our method dynamically reallocates patch sizes across denoising steps for image and video generation and substantially reduces cost while preserving perceptual generation quality. Extensive experiments demonstrate the effectiveness of our approach: it achieves up to $3.52\times$ and $3.2\times$ speedup on FLUX-1.Dev and Wan $2.1$, respectively, without compromising the generation quality and prompt adherence.

• The paper introduces DDiT, a dynamic patch scheduling approach that adjusts patch sizes per denoising step to optimize diffusion transformer efficiency.
• The paper leverages finite difference metrics on latent dynamics to achieve up to 3.52× speedup while preserving perceptual quality in image and video synthesis.
• The paper demonstrates seamless integration with pre-trained models using minimal modifications, enabling scalable control over the quality-speed tradeoff.

Dynamic Patch Scheduling for Efficient Diffusion Transformers: An Expert Summary

Motivation and Context

Diffusion Transformers (DiTs) have become the backbone for high-quality image and video synthesis, exploiting iterative denoising in VAE latent spaces via transformer-powered architectures. Despite their wide adoption for photorealistic generation, DiTs suffer considerable compute bottlenecks, largely due to fixed patch tokenization—every diffusion timestep processes the latent at the same granularity, regardless of structural complexity or prompt-specific detail requirements. Previous works have focused on acceleration using static token reduction, pruning, caching, quantization, and distillation, but often degrade output quality by indiscriminately discarding critical computation, or failing to adapt resource allocation to prompt or denoising stage complexity.

DDiT proposes a paradigm shift: exploiting content and temporal complexity by dynamically varying the patch size at each denoising step during inference. Early steps prioritize global structure with coarse granularity; later steps incrementally refine local details using smaller patches, yielding efficiency gains with controllable quality preservation.

Figure 1: Main idea: dynamic tokenization during denoising—DDiT adapts patch size at each denoising timestep based on latent complexity, as opposed to fixed patch sizes used in standard protocols.

Architectural Innovations

DDiT retrofits pre-trained DiT models with minimal modifications, leveraging a revised patch-embedding layer that supports multi-resolution patching. Each patch size (multiples of the base $p$) is associated with a dedicated embedding branch, implementable using LoRA adapters. The patch-size embedding enables patch-size identification within the transformer block, and positional embeddings for new patch sizes are obtained via bilinear interpolation.

Crucially, a residual connection from pre-embedding to post-de-embedding stabilizes latent manifold transitions. The LoRA branch is fine-tuned with a distillation loss against the frozen base model, maintaining perceptual output robustness.

Figure 2: Revised patch-embedding layer supports patches of varied resolutions, allowing seamless latent processing across timesteps for different patch sizes.

Inference speed is tightly coupled to patch granularity: larger patches decrease token count quadratically, leading to orders-of-magnitude acceleration with minimal patch size increases.

Figure 3: Inference speed vs. patch size—showing substantial acceleration as patch size increases during denoising, highlighting the scalability gains of dynamic scheduling.

Dynamic Patch Scheduling Mechanism

DDiT introduces a test-time, training-free Patch Scheduler driven by latent manifold evolution. Using finite difference (first, second, and third order) against the latent trajectory, patch size is chosen per-timestep according to acceleration and spatial variance:

• Latent acceleration estimation: The third-order temporal difference ($\Delta^{(3)}$) is empirically most predictive for identifying generative transitions (coarse to fine structure).
• Spatial variance estimation: Within each latent patch, the standard deviation of acceleration ($\boldsymbol{\sigma}_{t-1}^{p_{i}}$) is computed; a high variance signals need for finer granularity.
• Patch schedule aggregation: Rather than mean aggregation, the $\rho$-th percentile across spatial patches offers robust prompt-sensitive scheduling, mitigating bias from mixed-content images.

Patch selection uses a threshold $\tau$; the largest valid patch is chosen if its variance is below $\tau$, otherwise defaulting to the finest granularity. This provides explicit control over quality-speed tradeoff.

Figure 4: Computation of within-patch standard deviation $\boldsymbol\sigma_{t-1}^{p_i}$ for latent acceleration enables data-driven scheduling.

Prompt complexity is directly reflected by patch schedule dynamics over diffusion timesteps.

Figure 5: Prompts with different spatial requirements (e.g., zebras vs. apple) elicit distinct patch variance profiles, validating scheduler adaptability.

Experimental Results

DDiT is evaluated on FLUX-1.Dev [text-to-image] and Wan-2.1 [text-to-video]. It achieves up to $3.52\times$ (image) and $3.2\times$ (video) speedup, retaining quality as measured by FID, CLIP, ImageReward, SSIM, and LPIPS, with negligible degradation compared to the base model. Quality preservation is validated for both simple and complex prompts.

Figure 6: DDiT preserves fine spatial layout and detail against base and competitive methods at comparable speedups.

Figure 7: DrawBench qualitative comparisons indicate robust handling of semantically complex prompts, outperforming TaylorSeer baseline.

Video generation is similarly accelerated, with VBench scores demonstrating competitive quality under substantially reduced compute budgets.

Figure 8: DDiT generates videos of comparable quality to baseline with marked inference speedup.

Patch schedule trajectories are prompt-sensitive: detailed prompts (e.g., “sketch of a city street”) result in more fine-grained patch allocation during late denoising, whereas simple prompts switch to coarse patches earlier.

Figure 9: Patch schedules adapt to prompt complexity, allocating computation where generative detail is required.

Analysis

DDiT’s efficacy hinges on third-order finite difference for temporal scheduling, confirmed via empirical ablation: higher-order latent evolution metrics yield improved FID and CLIP. Human visual preference studies show DDiT outputs are indistinguishable from baseline in the majority of cases.

Patch scheduling threshold ($\tau$) adjustment allows smooth control over the speed-quality frontier; increased speedup induces only minor quality reduction, demonstrating scheduling robustness.

Implications and Future Directions

DDiT demonstrates that content- and step-adaptive computation unlocks substantial efficiency gains in DiTs with minimal architectural modification, enabling practical high-resolution content generation on limited resources. The technique is generic—applicable to diffusion models for both images and videos, and synergistic with existing acceleration strategies (e.g., caching).

From a theoretical perspective, the work deepens understanding of diffusion feature evolution, relating generative complexity to latent dynamics. Practically, explicit control of the computational budget per prompt offers scalability for real-world deployment, mobile inference, and multi-modal generative applications. Extension to intra-step adaptive patching (varying patch sizes within a timestep) remains future work.

Conclusion

DDiT introduces dynamic patch scheduling for diffusion transformers, providing granular control over inference computation by adapting patch sizes per denoising step based on latent complexity. The method preserves perceptual image and video quality under significant speedup, generalizes across tasks and models, and is straightforward to integrate with off-the-shelf pre-trained DiTs. DDiT establishes a new benchmark for efficient diffusion-based generation, with implications for scalable generative AI deployment and theoretical advances in latent manifold dynamics.

(2602.16968)