DiP Pixel-Space Diffusion Framework

• DiP Pixel-Space Diffusion Framework comprises generative techniques that directly model pixel intensities without relying on latent representations.
• It integrates transformer-based, Laplacian, and discrete-state methods to balance sample fidelity, computational efficiency, and controllability.
• The framework achieves state-of-the-art results in image synthesis, restoration, and scientific simulation through innovative diffusion processes.

The DiP Pixel-Space Diffusion Framework encompasses a set of generative modeling methodologies in which the diffusion process is formulated and learned directly in pixel space, bypassing lower-dimensional latent representations such as those used in autoencoder-based latent diffusion models. This family incorporates both optimization-driven image prior exploitation (as in DIIP), highly efficient transformer-based generative frameworks (DiP, PixelDiT), Laplacian and multi-scale hierarchical approaches (Edify Image), and enhancements for editability or physical fidelity (Differential Diffusion, Discrete Spatial Diffusion). The framework addresses the trade-off between sample fidelity, computational tractability, and controllability, showing state-of-the-art results in image synthesis, restoration, editing, and scientific simulation.

1. Mathematical Formulation of Pixel-Space Diffusion

Pixel-space diffusion models operate by defining a discrete-time or continuous-time noising process directly on the pixels $x\in\mathbb{R}^{H\times W\times C}$ or in discrete intensity space $I\in\mathbb{Z}_{\geq0}^{H\times W\times C}$. The classical forward process adopts a Markov chain:

$$q(x_t|x_{t-1}) = \mathcal{N}(x_t;\sqrt{\alpha_t} x_{t-1}, (1-\alpha_t)I), \qquad \bar{\alpha}_t = \prod_{s=1}^t \alpha_s,$$

with the marginal

$$q(x_t|x_0) = \mathcal{N}(x_t;\sqrt{\bar{\alpha}_t} x_0, (1-\bar{\alpha}_t)I)$$

as in DDPM/ADM/DiT-based models (Chen et al., 24 Nov 2025, Yu et al., 25 Nov 2025).

The reverse process is learned to approximate

$$p_\theta(x_{t-1}|x_t) = \mathcal{N}(x_{t-1};\mu_\theta(x_t, t), \Sigma_\theta(t)),$$

where $\mu_\theta$ and possibly $\Sigma_\theta$ are estimated by a neural network (U-Net, Transformer, or ViT-based).

For discrete pixel intensities (necessary for mass conservation or scientific applications), the forward process is a continuous-time Markov jump process, often formalized as

$$\frac{d}{dt}p_t(i_x, i_y, c) = \sum_{\nu\in\{\pm1,0\}^2:\|\nu\|_1=1} r\,p_t(i_x-\nu_x, i_y-\nu_y, c)-4r\,p_t(i_x, i_y, c),$$

ensuring conservation of total intensity in each channel (Santos et al., 3 May 2025).

Pixel-space diffusion can also be decomposed spectrally via Laplacian pyramids, allowing frequency bands to be attenuated at different rates for multi-scale modeling (NVIDIA et al., 2024), or embedded with per-pixel “strength” maps controlling localized change at inference (Levin et al., 2023).

2. Key Framework Variants and Architectures

The umbrella of DiP Pixel-Space Diffusion Framework includes several key instantiations:

• DIIP (Diffusion Image Prior): Restoration is cast as latent optimization over the generator $g(z)$ of a frozen pretrained diffusion model in pixel space, i.e., $z^* = \arg\min_z \|g(z) - y\|^2$, where $y$ is the degraded observation. Unlike Deep Image Prior (DIP), DIIP leverages a highly expressive diffusion prior and avoids explicit knowledge of the degradation operator $H$ (Chihaoui et al., 27 Mar 2025).
• DiP (Taming Diffusion Models in Pixel Space): A global–local two-stage architecture where a large-patch Diffusion Transformer (DiT) backbone models semantic structure and a lightweight Patch Detailer Head (shallow U-Net per patch) restores high-frequency detail. The architecture achieves state-of-the-art FID for pixel diffusion ($\mathrm{FID}=1.90$ on ImageNet-256) at orders-of-magnitude reduced computation relative to prior pixel-space transformers (Chen et al., 24 Nov 2025).
• PixelDiT: A single-stage, dual-level transformer which entirely obviates VAEs. A patch-level DiT captures global context, while a pixel-level DiT refines local details using per-pixel tokens, novel conditioning (pixel-wise AdaLN), and token compaction/expansion. This model achieves $\mathrm{FID}=1.61$ on ImageNet-256 and strong text-to-image results at 1K–pixel resolution (Yu et al., 25 Nov 2025).
• EPG (End-to-End Pixel-Space Generative Modeling): A two-stage, self-supervised pre-training and fine-tuning framework for efficient and stable training of ViT-based pixel diffusion and consistency models, closing the fidelity gap to latent-space methods (ImageNet-256 FID = 2.04) and for the first time successfully training a one-step pixel consistency model (Lei et al., 14 Oct 2025).
• Laplacian Pixel Diffusion (Edify Image): Cascaded diffusion at multiple resolutions with a Laplacian frequency decomposition. Each pyramid band receives statistically distinct noise schedules; higher frequencies are attenuated sooner, and mixture-of-experts U-Nets address different band/time intervals for scalable, high-fidelity generation up to 4K (NVIDIA et al., 2024).
• Differential-Diffusion: Inference-time method injecting a per-pixel strength map, allowing fine-grained spatial control over diffusion step extent. Model-agnostic and applicable across any pretrained DDPM model (Levin et al., 2023).
• Discrete Spatial Diffusion: Continuous-time, discrete-state Markov jump process in pixel space enforcing strict mass conservation, enabling image modeling tasks under global conservation constraints, with applications in scientific domains (Santos et al., 3 May 2025).

3. Training Procedures and Losses

Pixel-space diffusion models are trained with denoising score-matching losses, typically $$\mathcal{L} = \mathbb{E} \lVert \epsilon - \epsilon_\theta(x_t, t) \rVert^2$$, or (for velocity-based formulations) with rectified flow (RF) losses $$\mathcal{L} = \mathbb{E} \lVert f_\theta(x_t, t) - v_t \rVert^2$$, where $v_t$ encodes the flow from $x_t$ to noiseless $x_0$ (Yu et al., 25 Nov 2025). Representation alignment regularizers may be added for improved sample quality.

Architectural innovations (e.g., patch and pixel tokenization in PixelDiT; lateral and residual connections in EPG) are co-trained end-to-end, with batch sizes, epoch counts, and learning rates empirically optimized for sample quality and efficiency (Yu et al., 25 Nov 2025, Lei et al., 14 Oct 2025). Hyperparameters and training schedules are generally consistent with the state-of-the-art class-conditional or unconditional ImageNet pipelines.

Discrete-state models train neural rate predictors via rate-matching or negative log-likelihood on particle transition statistics (Santos et al., 3 May 2025).

4. Sampling Algorithms, Inference, and Early Stopping

Sampling involves ancestral reverse diffusion (standard or with advanced ODE solvers) for Gaussian models, and τ-leaping binomial update schemes for discrete models (Santos et al., 3 May 2025).

For restoration, DIIP employs gradient descent in the generator's latent space—early stopping is critical to avoid overfitting to the corrupted structure. For high-frequency degradations, stopping is triggered by the normalized loss slope minimum; for low-frequency artifacts, by the Laplacian variance maximum (as this monotonicity correlates with onset of artifact overfitting) (Chihaoui et al., 27 Mar 2025).

Differential-Diffusion controls local pixelwise diffusion schedules at inference by masking the path between a fixed original and the sampled model prediction using time-aligned spatial masks, enabling spatially non-uniform edit propagation (Levin et al., 2023).

Cascaded Laplacian frameworks use multistage sampling, switching networks ("mixture-of-experts") at critical time points when frequency bands vanish (NVIDIA et al., 2024).

5. Empirical Performance and Benchmarks

DiP-style pixel-space diffusion models set the current state of the art on unconditional image synthesis:

Model	FID (256x256) ↓	#Params	Latency (A100, steps)	Key Features
PixelDiT-XL (Yu et al., 25 Nov 2025)	1.61	797M	1.07s (100)	Dual-level transformer; no autoencoder
DiP-XL/16 (Chen et al., 24 Nov 2025)	1.90	631M	0.92s (100)	Patch DiT + Patch Detailer
EPG/16 (Lei et al., 14 Oct 2025)	2.04	583M	75 NFE; 128 GFLOPs	2-stage ViT, self-supervised pretrain
PixelFlow-XL	1.98	677M	7.50s (120)	Pixel Transformer only

Performance on restoration tasks using DIIP also surpasses prior training-free or blind approaches (see section 6 below) (Chihaoui et al., 27 Mar 2025).

6. Applications: Image Synthesis, Restoration, Editing, and Science

• Image Restoration (DIIP): Blind denoising, super-resolution ($\times$4, $\times$8), JPEG artifact, deformation, and water-drop removal on CelebA/ImageNet, achieving PSNR up to $28.37$ dB, SSIM = $0.842$, LPIPS = $0.224$ in denoising, and consistently outperforming DIP, DreamClean, BlindDPS, and other zero-shot baselines (Chihaoui et al., 27 Mar 2025).
• Unconditional and Conditional Synthesis: PixelDiT, DiP, and EPG demonstrate class-conditional and text-conditional synthesis at $256^2$–$1024^2$ resolution, matching or surpassing latent-space models in FID while eschewing lossy autoencoders (Yu et al., 25 Nov 2025, Chen et al., 24 Nov 2025, Lei et al., 14 Oct 2025). Edify Image achieves pixel-perfect 4K upsampling via multi-band Laplacian cascades (NVIDIA et al., 2024).
• Editing and Controllability: Differential-Diffusion enables per-pixel user control of the degree of generative alteration, supporting smooth “soft inpainting,” region-selective prompting, and intensity blending, confirmed by high CAM/DAM adherence and user study (Levin et al., 2023).
• Physical and Scientific Modeling: Discrete Spatial Diffusion guarantees strict total-mass conservation and is validated on tasks requiring discrete-valued, conservation-law-respecting outputs, such as microstructure and battery electrode generation, outperforming continuous-space DDPMs on structural and physical realism (Santos et al., 3 May 2025).

7. Comparative Analysis and Evolution

A principal motivation for DiP pixel-space frameworks is to reconcile the high fidelity of pixel-space DDPMs with the computational scalability of latent-space methods. Early pixel-space transformers suffered $O((HW/p^2)^2)$ compute, becoming intractable at moderate resolutions; DiP and PixelDiT circumvent this with patch tokenization, dual-level architectures, and detail refinement heads—yielding inference cost and performance competitive with or superior to latent-space LDMs (Yu et al., 25 Nov 2025, Chen et al., 24 Nov 2025).

Laplacian and multi-scale methods (Edify Image) provide further advances in scalability and quality especially for very high-resolution synthesis (NVIDIA et al., 2024). Optimization and edit frameworks (DIIP, Differential-Diffusion) demonstrate that pixel-space diffusion can be adapted seamlessly to unsupervised restoration and interactive controllable editing, respectively (Chihaoui et al., 27 Mar 2025, Levin et al., 2023).

The expansion of pixel-space frameworks to high-fidelity, high-resolution, training-free restoration, conditional control, and strict physical regularization suggests a convergence of methodological strengths previously exclusive to either latent or pixel domain.

---

References:

• "Diffusion Image Prior" (Chihaoui et al., 27 Mar 2025)
• "DiP: Taming Diffusion Models in Pixel Space" (Chen et al., 24 Nov 2025)
• "PixelDiT: Pixel Diffusion Transformers for Image Generation" (Yu et al., 25 Nov 2025)
• "Advancing End-to-End Pixel Space Generative Modeling via Self-supervised Pre-training" (Lei et al., 14 Oct 2025)
• "Edify Image: High-Quality Image Generation with Pixel Space Laplacian Diffusion Models" (NVIDIA et al., 2024)
• "Differential Diffusion: Giving Each Pixel Its Strength" (Levin et al., 2023)
• "Discrete Spatial Diffusion: Intensity-Preserving Diffusion Modeling" (Santos et al., 3 May 2025)