Latent Forcing: Reordering the Diffusion Trajectory for Pixel-Space Image Generation

Abstract: Latent diffusion models excel at generating high-quality images but lose the benefits of end-to-end modeling. They discard information during image encoding, require a separately trained decoder, and model an auxiliary distribution to the raw data. In this paper, we propose Latent Forcing, a simple modification to existing architectures that achieves the efficiency of latent diffusion while operating on raw natural images. Our approach orders the denoising trajectory by jointly processing latents and pixels with separately tuned noise schedules. This allows the latents to act as a scratchpad for intermediate computation before high-frequency pixel features are generated. We find that the order of conditioning signals is critical, and we analyze this to explain differences between REPA distillation in the tokenizer and the diffusion model, conditional versus unconditional generation, and how tokenizer reconstruction quality relates to diffusability. Applied to ImageNet, Latent Forcing achieves a new state-of-the-art for diffusion transformer-based pixel generation at our compute scale.

• The paper introduces Latent Forcing, reordering latent and pixel denoising to reduce FID scores and improve generation fidelity.
• It employs joint diffusion in latent and pixel spaces with independent noise schedules, enabling lossless reconstruction and simplified training.
• Empirical results on ImageNet-256 demonstrate significant improvements over previous methods, validating the approach’s efficiency and stability.

Latent Forcing: Reordering the Diffusion Trajectory in Pixel-Space Image Generation

Introduction and Motivation

Latent diffusion models (LDMs) have established themselves as a dominant methodology for high-quality image generation, leveraging compact latent spaces to mitigate the intrinsic complexity of pixel-space learning and sampling. However, LDMs incur significant costs: non-end-to-end pipelines, information loss due to aggressive latent compression, and the necessity of carefully engineered decoders. The corresponding tradeoff—the so-called reconstruction-generation dilemma—limits the ability to scale generative performance while preserving semantic fidelity in details such as faces or text. Conversely, recent progress in pixel-space diffusion revisits the direct modeling of raw images, seeking both simplicity and superior information retention, but often at the cost of slower or less stable convergence.

This work introduces the Latent Forcing paradigm, which recasts the generative order—not merely the generative space—as central to diffusion performance. Rather than compressing inputs into a single latent and then generating images from it, Latent Forcing schedules the denoising of latent patches prior to high-frequency pixel features, using joint diffusion in both latent and pixel space with independent noise schedules. The latent space in this context serves as an intermediate computational scratchpad rather than the generative endpoint, and is discarded after synthesis. This reordering enables pixel-space generation with the efficiency and convergence benefits of latent-based architectures, but without their information bottlenecks.

Technical Approach

The Latent Forcing framework extends flow-based diffusion to the joint modeling of $k$ modalities (with $k=2$ in this work: latent and pixel), each governed by an independent time variable. At both training and inference, trajectories through $(t_\text{latent}, t_\text{pixel})$ space are defined by scheduling functions, typically cascading the denoising of latents to precede that of pixels. Crucially, the model uses standard diffusion transformers with minimal additions: a second time conditioning MLP for the latent branch and a shared or expert-split output head.

Key technical innovations and settings include:

• Tokenization: Latents are derived from pre-trained, self-supervised models such as DINOv2 or Data2Vec2, patch-aligned to the pixel space, with per-token normalization to match pixel variance.
• Loss and Prediction Targets: Models predict $x$ directly (x-prediction) with v-loss weighting for stable training in high-dimensional spaces.
• Trajectory Scheduling: Both "Multi-Schedule" (arbitrary ordering, uniform time sampling) and "Single-Schedule" (fixed cascaded or shifted time schedules) models are studied, with ablations on the effect of latent-pixel denoising order.
• Architectural Simplicity: Additive token embeddings, minor parameter increases (e.g., +0.5% for double time MLP), and optional transformer layer specialization (expert heads for pixels/latents) for peak performance.

Formally, given image $x$ and (deterministic) latent $y = f(x)$, diffusion proceeds as:

$$\min_{v_\theta} \mathbb{E}_{t, x, \epsilon} \left[ \sum_{i=1}^k \lambda_i \left\| v_{\theta, i}(z_{1,t_1}, ..., z_{k,t_k}, t_1, ..., t_k) - (x_i - \epsilon_i) \right\|_2^2 \right]$$

with time scheduling functions $t_i = f_i(t_\mathrm{global})$ and appropriate SNR-based weighting.

Empirical Findings

A comprehensive empirical campaign on ImageNet-256 establishes several critical insights:

• Optimal Ordering: Empirically, denoising latent representations before pixel representations yields substantially lower FID (Fréchet Inception Distance) scores than the reverse or joint schedules, both with and without class-conditioning.
• Numerical Results: On conditional ImageNet generation, Latent Forcing achieves new state-of-the-art FID-50K for pixel-space diffusion transformers: 9.76 (unguided), 4.18 (guided) with DINOv2 latents. On unconditional generation, drastic improvements are seen compared to JiT and REPA baselines. The strong positive effect of ordering holds across multiple latent tokenization models.
• Lossless Reconstruction: The architecture maintains lossless reconstruction of the input, removing the need for compression tradeoffs characteristic of traditional LDMs.
• Ablation Analysis: The performance gap between Latent Forcing and prior distillation-based models (e.g., REPA) is shown to arise from ordering rather than solely from the use of additional pretrained features or auxiliary loss terms.
• Guidance: Adapting guidance mechanisms (e.g., Classifier-Free Guidance, AutoGuidance) to multi-modal/ordered schedules further boosts sample quality, with optimal settings depending on the nature of time allocation between modalities.

Theoretical Implications

Recasting diffusion as a multi-time, multi-space process refines the traditional compression dilemma within generative modeling. The analysis shows that the information capacity and SNR trajectory throughout diffusion are not monolithic properties but can be dynamically controlled via scheduling, turning what was a global tradeoff (compression vs. fidelity) into a localized, order-dependent optimization. Notably, the mutual information between noisy and clean variables is strictly monotonic with respect to the noise schedule, and scheduling can be used to optimally allocate representational capacity where it is most needed.

Additionally, Latent Forcing clarifies the effectiveness of representation distillation by highlighting that the temporal point at which conditioning occurs is crucial. Early exposure to high-level features through latent denoising conditions pixel trajectory more efficiently than static or late-stage supervision, a property that is further validated by empirical study.

Practical Implications and Future Perspectives

Latent Forcing delivers an end-to-end, high-performance generative pipeline that is as easy to train and deploy as modern latent or pixel-space methods but side-steps their primary limitations. Practically, this paradigm:

• Eliminates reconstruction bottlenecks by discarding the necessity for heavy compression in the tokenizer.
• Reduces the need for separately trained decoders, simplifying both training and inference.
• Enables new multi-modal and conditioning strategies by extending to any number of tokenizers/modalities with independent time variables.

The approach is straightforward for integration with advances in diffusion transformer architectures, optimization, and large-scale training regimes. Its principle—controlling and reordering the revelation of information during sampling—has potential parallel applications in text, audio, and other structured generation tasks, wherever the sequence of coarse-to-fine reconstruction can be learned or prescribed.

Looking forward, this research suggests that further fine-grained control and dynamic scheduling of latent and pixel denoising could yield additional gains, particularly when aligned with dataset-specific statistics or for conditional generation involving rich side information (e.g., multimodal or temporal signals). Additionally, the theoretical framework provides a foundation for future work on information-theoretic capacity allocation and adaptive diffusion in high-dimensional generative modeling.

Conclusion

Latent Forcing reframes generative diffusion as an explicitly ordered process, achieving superior sample quality, lossless information retention, and practical simplicity by combining the strengths of both latent- and pixel-space approaches. The empirical and theoretical findings establish generation order as a critical, previously underappreciated axis in diffusion generative modeling. This represents a significant step toward general, efficient, and robust end-to-end image synthesis and reframes the role of tokenization and conditional representation in the next generation of generative models.