Generative Modeling via Drifting

Abstract: Generative modeling can be formulated as learning a mapping f such that its pushforward distribution matches the data distribution. The pushforward behavior can be carried out iteratively at inference time, for example in diffusion and flow-based models. In this paper, we propose a new paradigm called Drifting Models, which evolve the pushforward distribution during training and naturally admit one-step inference. We introduce a drifting field that governs the sample movement and achieves equilibrium when the distributions match. This leads to a training objective that allows the neural network optimizer to evolve the distribution. In experiments, our one-step generator achieves state-of-the-art results on ImageNet at 256 x 256 resolution, with an FID of 1.54 in latent space and 1.61 in pixel space. We hope that our work opens up new opportunities for high-quality one-step generation.

• The paper introduces a one-step generative model that maps a prior to data using a dynamically computed, anti-symmetric drifting field.
• It employs kernel-based mean-shift vectors in a learned feature space, achieving superior FID scores on ImageNet with a single inference pass.
• The methodology generalizes to robotics control, outperforming iterative diffusion and adversarial techniques in both efficiency and fidelity.

Generative Modeling via Drifting: A Technical Overview

Formulation and Theoretical Foundations

The paper "Generative Modeling via Drifting" (2602.04770) introduces a non-iterative, training-centric paradigm for generative modeling, departing from the standard inference-time iteration of diffusion and flow-based techniques. The core insight is to frame generative learning as the evolution of a pushforward distribution under a neural network, regulated by a dynamically computed "drifting field." This field enforces a vector field over generated samples, with the property that it vanishes when the generated distribution matches the data distribution, establishing an equilibrium point that is the desired generative match.

Training optimizes a neural network $f$ such that its pushforward $q = f_\# p_\text{prior}$ approximates $p_\text{data}$, with sample-wise updates $x_{i+1}=x_i + V_{p, q_i}(x_i)$. Unlike diffusion models, the entire evolution is subsumed into training, yielding a generator capable of one-step (1-NFE) inference.

Figure 1: Illustration of drifting a sample: a generated sample $\mathbf{x}$ (black) drifts via the mean vector difference between nearby data distribution samples (blue) and generated samples (orange), as specified by the drifting field.

The drifting field formulation is anti-symmetric: $V_{p,q}(x) = -V_{q,p}(x)$. When $p=q$, the field cancels out for every $x$, achieving equilibrium. The training objective reduces directly to minimizing the expected squared norm of the field, $\mathbb{E}[\|V_{p,q}(x)\|^2]$, sidestepping backpropagation through the distribution by leveraging a stop-gradient target.

The field is constructed using normalized kernels and mean-shift vectors between the generated and data samples, typically in a learned feature space:

$$V_{p,q}(x) = \frac{1}{Z_p Z_q} \mathbb{E}_{y^{+} \sim p, y^{-} \sim q}\left[k(x, y^+)k(x, y^-)(y^+ - y^-)\right],$$

with $k(x, y)$ implemented as temperature-controlled, softmax-normalized similarity.

Feature-Space Drifting and the Role of Representation

Training in high-dimensional pixel space is infeasible; thus, the loss is evaluated in a semantically meaningful feature space derived from self-supervised representations (e.g., ResNet-based MAE, SimCLR, MoCo). The loss aligns pushforward feature distributions by minimizing drift in the multi-scale, multi-location feature embeddings of the generated and real data. This is crucial, as the generative mapping—trained using kernelized similarity—is only as effective as the feature space's semantic alignment.

Empirical ablations confirm that feature encoder quality directly impacts FID. Latent-space feature extractors pre-trained with MAE objectives (and subsequent classifier fine-tuning) yield the strongest results, with significant improvement in FID as encoder capacity increases.

Implementation: Architecture and Classifier-Free Guidance

The generator adopts DiT-like transformer architectures for both pixel and latent spaces, with input noise mapped either to VAE latent codes or directly to pixel images. Class-conditional generation utilizes explicit classifier-free guidance (CFG) during training, with negative samples drawn both from other classes and unconditional data, paralleling recent advances in conditional diffusion.

CFG is handled at training by mixing the negative samples and at inference by adjusting the guidance scale $\alpha$, preserving the one-step generation property.

Figure 2: Evolution of the generated distribution; the pushforward $q$ (orange) approaches a bimodal data target $p$ (blue) under the field, robust to initialization and avoiding mode collapse.

Empirical Results

ImageNet Generation

On ImageNet 256$\times$256, Drifting Models achieve state-of-the-art results for one-step (1-NFE) generation. Latent-space models reach FID 1.54 (DiT-L/2), surpassing previous one-step baselines, including MeanFlow (FID 3.43) and iMF (FID 1.72) [geng2025mean, geng2025improved]. Pixel-space models attain FID 1.61, outperforming StyleGAN-XL (FID 2.30) and matching or exceeding diffusion models despite requiring only a single forward pass.

Figure 4: The effect of CFG scale $\alpha$ demonstrates the FID/IS tradeoff, with optimal FID for $\alpha=1.0$ (no CFG) and increasing IS at higher guidance strengths.

Qualitative analysis further substantiates high diversity and fidelity. Nearest-neighbor (NN) analysis using CLIP embeddings indicates generated samples do not trivialize training set memorization.

Figure 3: Nearest neighbor analysis, associating each generated image (orange) with its top 10 real image neighbors (blue) in CLIP space; samples are visually novel.

Uncurated generations illustrate high sample quality and semantic accuracy in both latent and pixel space settings.

Figure 5: Uncurated samples from latent-L/2 model with CFG=1.0, demonstrating high fidelity (FID = 1.54, IS = 258.9).

Robotics Control

The approach generalizes beyond vision, replacing the diffusion process in Diffusion Policy for robotic control with a one-step Drifting Policy. On various tasks, it matches or exceeds the performance of the multi-step baseline, empirically validating the paradigm's domain-agnosticity.

Comparison to Related Paradigms

Diffusion/Flow-based Models: Unlike these methods, Drifting Models do not require SDE/ODE simulation at inference. Training absorbs all distributional evolution, allowing for single-pass generation without multi-step distillation or trajectory approximation.

GANs/VAEs/Normalizing Flows: The approach is adversarial-free and strictly pushforward (not invertible), with distribution alignment encoded in the analytic properties of the drifting field, rather than a discriminator, ELBO, or likelihood loss.

Moment Matching/MMD: There is a formal connection to moment-matching objectives, but the paper's normalized kernel and anti-symmetric drift field generalize prior methods, offering stronger guarantees and more robust optimization for high-dimensional generative mapping.

Ablations and Methodological Insights

• Anti-symmetry in the field is strictly necessary; ablation with asymmetric fields leads to catastrophic mode collapse.
• Positive/Negative sample scaling: Larger batches of both improve kernel estimation precision and feature distribution matching.
• Kernel normalization and temperature: The design of the kernel and normalization axis affects training stability and sample quality.
• Feature-space drift: The limiting factor is the semantic alignment and discriminative capacity of the feature space, explaining the dependence on the MAE encoder's scale and training duration.

Implications and Future Directions

This work reframes generative modeling as a training-time, network-driven distributional evolution. The convergence properties are governed not by ODEs/SDEs at generation but by equilibrium conditions at training, with identifiability discussed under nondegeneracy of the drift kernel and sample coverage.

Practically, Drifting Models bring high-quality, extremely efficient one-step generative inference to domains previously dominated by iterative methods, while preserving or improving on their sample quality benchmarks.

Potential avenues for further advancement include optimizing the field design, feature encoders, and sampling strategies to close residual theoretical gaps (e.g., sufficiency of $V \to 0$ for $q \to p$ in all regimes) and exploring the framework's extensibility to additional data modalities and conditioning tasks. The non-reliance on adversarial loss or invertible architectures increases robustness and scalability, suggesting broad applicability.

Conclusion

"Generative Modeling via Drifting" provides a rigorous framework for training non-iterative generative models, matching or exceeding the empirical quality of multi-step diffusion/flow models with a drastically simplified inference pathway. By leveraging anti-symmetric mean-shift fields in expressive feature spaces, the approach yields new possibilities for fast, high-fidelity generation across vision and control, with significant implications for the efficiency and flexibility of future generative modeling methods.