Learning When to Denoise: Optimizing Asynchronous Schedules for Latent Diffusion

Abstract: Multi-representation diffusion models can improve visual synthesis by denoising complementary views of an image, but their performance depends critically on the asynchronous schedule that determines when each representation is denoised. We propose to learn this schedule. Our method formulates asynchronous flow matching over multiple representation spaces and uses a schedule-corrected objective that keeps each representation's local noising-time weights fixed as the schedule changes. We instantiate the schedule with a flexible parametric class that is convex and monotone by construction, and learn it using a fast joint probe with less than 1% additional training compute. On ImageNet 256x256, the learned schedule substantially improves both convergence speed and final quality under a matched 675M-parameter XL backbone. With AutoGuidance, our 200-epoch model reaches FID 1.05, matching the 800-epoch SFD-XL baseline with 4x less training. Training to 600 epochs further improves to FID 1.02, outperforming the 1B-parameter SFD-XXL result of FID 1.04 while using a smaller model. In the unguided setting, our 200-epoch model reaches FID 2.37, already below the best 800-epoch SFD-XL result (2.54) at 4x less training, and improves to FID 2.14 at 600 epochs. Code is available at https://github.com/bsq532087/LWD

• The paper introduces a joint schedule learning method that adapts asynchronous denoising for semantic and texture latent spaces.
• It leverages analytic corrections and kinetic-energy regularization to enforce convex, monotone scheduling for enhanced convergence.
• Empirical results on ImageNet show significantly faster training and superior FID scores compared to hand-tuned approaches.

Learning When to Denoise: Optimizing Asynchronous Schedules for Latent Diffusion (2606.19662)

Motivation and Problem Statement

The paper addresses a fundamental design question in multi-representation latent diffusion models: the optimal asynchronous schedule for denoising distinct image representation spaces. Existing multi-latent diffusion systems such as Semantic-First Diffusion (SFD) utilize a hand-selected temporal offset to coordinate denoising between latent groups—typically semantic and texture. However, this manual selection becomes increasingly suboptimal as the space of schedules and representation pairs broadens, especially with variable model architectures and loss functions. The authors propose a principled methodology for asynchronously scheduling denoising in latent diffusion: jointly learning the schedule as part of the main optimization process.

Methodological Contributions

Asynchronous Flow Matching and Schedule Learning

The method formulates asynchronous flow matching for composite latents by positing group-wise local time schedules. Specifically, the semantic branch proceeds synchronously with the global denoising trajectory, whereas the texture branch follows a learned schedule with enforced monotonicity and convexity, ensuring semantic-leading behavior:

Figure 1: Learning the relative denoising schedule between semantic and texture latents; the semantic schedule leads, guiding texture refinement.

The authors theoretically characterize the population optimum of the schedule-corrected loss in two equivalent ways: (i) via the continuity equation for distributional interpolation, and (ii) as a group-wise transformation of the score field (see Theorem 1). This dual characterization yields a change-of-variables correction to maintain invariant marginal weighting over noising times during schedule adaptation.

Schedule-Corrected Objective and Kinetic Regularization

A component-wise flow-matching loss is crafted for both semantic and texture branches. Schedule learning induces a confounding effect—changing the schedule alters local-noising time distributions. To counteract this, a necessary and sufficient Jacobian correction is analytically derived (see Lemma 1). Moreover, a kinetic-energy regularizer penalizes trajectories that compress denoising into short intervals, favoring schedules amenable to discretized ODE integration.

Parametric Schedule Class and Bilevel Optimization

The texture schedule is parameterized as a convex, monotone polynomial family, automatically enforcing semantic-first conditions. Joint optimization proceeds via a two-phase probe: a short, regularized schedule-learning phase (~1% compute budget) with a temporary denoiser, followed by main training with the fixed, learned schedule.

Figure 2: $\lambda$-sweep of schedule probe; weak regularization leads to collapse, stricter penalization stabilizes the learned schedule.

Empirical Results and Performance

Quality and Efficiency Gains

On class-conditional ImageNet $256 \times 256$, the learned schedule outperforms hand-tuned SFD schedules under matched architecture, latent representations, sampler, and evaluation protocols. Unguided FID improves from 2.54 (SFD-XL, 4M iterations) to 2.37 (Ours, 1M iterations) and further to 2.14 at 3M iterations with the same backbone. Critically, convergence speed is accelerated: Ours matches SFD-XL’s best FID using five times fewer diffusion-model updates.

Under AutoGuidance, the procedure achieves FID 1.05 with the XL backbone at 200 epochs versus SFD-XL’s 1.06 at 800 epochs; extending training to 600 epochs reaches FID 1.02, surpassing the 1B-parameter SFD-XXL baseline (FID 1.04) despite lower parameterization.

Figure 3: Qualitative samples from Ours-XL at $256\times256$; class-conditional outputs under AutoGuidance.

Robustness to Representation Variants

Schedule learning generalizes across semantic encoder choices: results consistently improve for DINO-PCA and CLIP-PCA latents relative to SFD's hand-tuned schedules. The shape of the learned schedule adapts to encoder informativeness, yielding more aggressive semantic-leading curves for weaker semantic latents.

Figure 4: Cockatoo (class 89). Ours-XL samples under AutoGuidance, $w=1.5$.

Figure 5: Husky (class 250). Ours-XL samples under AutoGuidance, $w=1.5$.

Figure 6: Lion (class 291). Ours-XL samples under AutoGuidance, $w=1.5$.

Figure 7: Balloon (class 417). Ours-XL samples under AutoGuidance, $w=1.5$.

Figure 8: Coral reef (class 973). Ours-XL samples under AutoGuidance, $w=1.5$.

Figure 9: Volcano (class 980). Ours-XL samples under AutoGuidance, $w=1.5$.

Implications and Future Directions

The proposed framework establishes asynchronous schedule learning as an essential component for scalable, representation-aware latent diffusion. By showing robustness across encoder variants and architectures, the approach paves the way for generalized multi-group latent diffusion—extending beyond semantic and texture to auxiliary representations such as depth or segmentation. Bilevel schedule learning with analytic correction enables systematic exploration of optimal denoising orderings, which may be critical for generator interpretability and transferability.

Future research may focus on:

• Scaling to more representation groups, e.g., joint scheduling across semantic, texture, depth, and segmentation latents.
• Richer schedule parameterizations (e.g., input-convex neural networks, splines) for finer trajectory control.
• Application to text-to-image, video, and multimodal diffusion models, exploring cross-modal denoising orderings.
• Integration with dynamic or adaptive guidance weights, and systematic analysis of the impact of asynchronously learned schedules on various guidance methods (i.e., classifier-free, classifier-based).

Conclusion

This work delivers a theoretically grounded, empirically validated method for jointly learning asynchronous denoising schedules in latent diffusion. The approach yields substantial improvements in both convergence speed and final sample quality over established hand-tuned designs, even at reduced parameter counts and compute budgets. The analytic schedule-corrected objective and kinetic regularization ensure robust, discretization-friendly schedule learning, and empirical results demonstrate transferability across semantic encoder choices. The implications are substantial for large-scale diffusion systems—optimal scheduling is now a learnable, transferable modeling artifact with clear impact on both training efficiency and generative fidelity.