Generative Image-Based Light Decomposition

• Generative Image-Based Light Decomposition is a method that factors images into components like albedo, shading, and specular effects using deep generative networks.
• It leverages latent diffusion, GANs, and VAEs to achieve high realism and interactive control in relighting and layer-wise editing.
• The approach enhances traditional models with data-driven priors for fine-grained per-light source control and consistent, photorealistic reconstructions.

A generative image-based light decomposition model is a data-driven approach that factors observed images into physically or semantically interpretable lighting components—such as albedo, shading, illumination, specular effects, or per-source contributions—using a generative network that learns from large-scale data. Modern instances leverage latent diffusion models, GANs, or VAEs to regularize the decomposition process, often achieving higher realism, editability, and generalization than purely analytical or regression-based techniques. These models address classic intrinsic image decomposition, real-world relighting, photorealistic editing, and per-light source control, and can be realized for 2D or 3D scene representations.

1. Mathematical Foundations of Image-Based Light Decomposition

Let $I(x)$ denote the observed RGB image at pixel $x$. The core physical premise for decomposition is the rendering equation, typically simplified in the image domain as:

$I(x) = R(x) \odot L(x) + S(x)$

where:

• $R(x)$: intrinsic reflectance or albedo (material property)
• $L(x)$: illumination or shading (lighting and geometry)
• $S(x)$: additive term capturing specular or other non-Lambertian effects

This decomposition is inherently ill-posed; the mapping from $I(x)$ to $(R(x),L(x),S(x))$ is (typically) not unique. Generative models constrain the solution space, leveraging powerful priors learned from data to yield plausible, disentangled factors. For advanced lighting control tasks, various works further factor $L(x)$ into per-light or per-effect components:

$$I_{\mathrm{obs}}(x) = L_0(x) + \sum_{i=1}^N s_i\,\alpha_i\,\mathbf{c}_i\,\odot\,L_i(x)$$

with $L_0(x)$ representing ambient illumination and $L_i(x)$ the contribution from a discrete light source $i$ (e.g., in indoor scene relighting) (Liang et al., 21 Jan 2026).

2. Generative Model Architectures and Training Paradigms

2.1 Latent Diffusion and Transformer-Based Models

Latent diffusion models perform decomposition in a learned low-dimensional space, often using a VAE encoder to map RGB images to compact features, and a U-Net/Transformer diffusion backbone for denoising and inference (Zeng et al., 2024, Jiang et al., 2024, Yang et al., 2024, Liang et al., 21 Jan 2026). Conditioning mechanisms may include concatenated intrinsic channels, cross-attention on text, or mask-based spatial guidance. Examples:

• RGB$\leftrightarrow$X employs a paired encoder/decoder to map from RGB to per-pixel normals, albedo, roughness, metallicity, and irradiance in the latent space (Zeng et al., 2024).
• LightenDiffusion splits the latent space via a Retinex-in-the-latent-domain operation, then uses a conditional diffusion model to reconstruct enhanced content (Jiang et al., 2024).

2.2 GAN and VAE Bank Models

Some approaches train independent VAEs or GANs on "Platonic" examples of each layer (e.g., Mondrian images for albedo, 3D renderings for shape, etc.), and decompose new images by jointly inverting the bank (Rock et al., 2016, Shah et al., 2023). The JoIN framework, for instance, optimizes the latent codes of StyleGAN2 generators for each intrinsic type such that their rendered combination matches the observed image, regularized to stay on the generator manifold (Shah et al., 2023).

2.3 Mask-Guided and Per-Light Conditioning

Per-light decomposition and interactive relighting leverage additional mask inputs, spatial attention, and local adaptation layers (e.g., LoRA adapters in diffusion blocks). LuxRemix uses masked image inputs to produce one-light-at-a-time (OLAT) decompositions, enabling independent on/off, chromaticity, and intensity control per source (Liang et al., 21 Jan 2026).

2.4 Hybrid and Layered Representations

Layered methods, such as LayerDecomp, use parallel denoising/diffusion branches for distinct image layers (background and RGBA foreground with associated visual effects), with a consistency loss enforcing that their blend reconstructs the original image (Yang et al., 2024). This supports transparent effects (shadows, reflections) and spatial editing.

3. Loss Functions, Priors, and Training Strategies

Losses in these models balance physical fidelity, perceptual realism, priors on each factor, and regularization for disentanglement. Core categories include:

• Reconstruction loss: Enforces $I \approx R \odot L$ (Retinex), or more complex recomposition for multi-layer/multi-light scenarios.
• Adversarial loss: PatchGAN or similar discriminators ensure the realism of outputs or individual decomposed layers (Shi et al., 2019, Yang et al., 2024).
• Perceptual loss: VGG or LPIPS-based terms encourage perceptual similarity at high level features (Shah et al., 2023).
• Consistency losses: Maintain correspondence between decomposed layers when recomposed, and enforce invariance/consistency for shared content under different illuminations (Yang et al., 2024, Jiang et al., 2024).
• Physics-inspired regularizers: Enforce smoothness of illumination, reflectance constancy, or chromaticity (Weligampola et al., 2021, Yi et al., 2023).
• Latent priors or k-nearest-neighbor constraints: Penalize deviation from the latent manifold of each generator (Shah et al., 2023, Rock et al., 2016).
• Custom ablations and multi-path diffusion objectives: Separate restoration tasks for reflectance and illumination with tailored noise/correlation penalties (Yi et al., 2023).

Dataset construction combines simulated and real data, with large-scale triplet sets (content, light, composite) for training and evaluation of decoupling capabilities (Li et al., 20 Aug 2025, Yang et al., 2024).

4. Applications: Relighting, Editing, and Control

Generative decomposition enables a diverse set of applications:

• Photorealistic relighting: Models such as HeadLighter and LuxRemix decompose 3D scenes or heads into material and lighting, supporting arbitrary viewpoint and environment edits with interactive control (Wang et al., 5 Jan 2026, Liang et al., 21 Jan 2026).
• Low-light image enhancement: Diffusion-based Retinex models enhance underexposed images by separately restoring illumination and reflectance, often outperforming CNN-based or single-path GAN baselines on PSNR/SSIM/LPIPS and perceptual scores (Yi et al., 2023, Jiang et al., 2024).
• Layer-wise editing: LayerDecomp supports object removal, moving, or transparency editing, maintaining natural shadows and reflections without explicit human annotation (Yang et al., 2024).
• Per-light source decomposition: LuxRemix and GS-ID enable toggling, recoloring, and intensity scaling for each discrete source in indoor or 3D Gaussian Splatting scenes, with real-time rendering and high photometric accuracy (Liang et al., 21 Jan 2026, Du et al., 2024).
• Visual effect transfer: TransLight decouples and injects complex light effects (e.g., lens flares, volumetric beams) from reference to target images, offering high-fidelity, spatially controlled composite results (Li et al., 20 Aug 2025).

5. Quantitative Evaluation and Comparative Benchmarks

Performance evaluation in these systems involves photometric and perceptual measures:

Model / Method	PSNR↑	SSIM↑	LPIPS↓	FID↓	Special Notes
Diff-Retinex	21.98	0.863	0.048	47.85	LOL test, best among unsup.
LightenDiffusion	–	–	–	–	Top NIQE/PI (unpaired data)
LayerDecomp (recomp)	30.53	–	0.0494	12.75	Camera test set
GS-ID (synthetic)	36.72	0.977	0.027	–	Best on novel-view tasks
LuxRemix-SV	27.68	0.898	0.082	–	OLAT decomposition
TransLight	19.58	0.7931	0.1982	6.02	Light-FID metric

These models consistently outperform prior work on realism, editability, and decomposition accuracy, especially on benchmarks involving photorealistic relighting, user-driven layer manipulation, and complex illumination transfer.

6. Limitations and Proposed Directions

While state-of-the-art generative decomposition achieves high-fidelity results, several challenges remain:

• Data bias and domain generalization: Models overfit to synthetic or stylized priors; real-world generalization is improved but not fully solved by hybrid datasets (Zeng et al., 2024).
• Physical completeness: Most models decompose only first-order effects (albedo, shading, specular); complex phenomena like translucency, high-order interreflections, and caustics remain open.
• Resolution scaling: Large images suffer from artifacts; multi-scale latent diffusion or patch-based inference are suggested extensions.
• Disentanglement robustness: Purely data-driven models can leak content between channels; dedicated regularizers and stronger supervision via physics or multi-view data are emerging trends (Li et al., 20 Aug 2025, Wang et al., 5 Jan 2026).
• Real-time 3D relighting: While frameworks like GS-ID and LuxRemix achieve interactive rates, deferred shading under thousands of light sources is computationally expensive; optimization of parametric light representations and screen-space shading is an active area (Du et al., 2024).
• Content-light decoupling: Extracting subtle or sparse light effects for user-guided editing remains challenging, as does the faithful transfer of volumetric or localized illumination features (Li et al., 20 Aug 2025).

7. Extensions Across Domains and Representation Types

The generative image-based light decomposition paradigm has been adapted across a spectrum of tasks and representations:

• 2D image editing: Early VAE/GAN frameworks enabled the decomposition of photographs into albedo, shading, and mesostructure without ground-truth supervision (Rock et al., 2016, Shah et al., 2023).
• Latent Retinex/diffusion: Recent models operate primarily in latent feature space for enhanced generalizability and scene independence (Jiang et al., 2024, Yi et al., 2023).
• 3D and multi-view: HeadLighter and LuxRemix push decomposition into fully 3D or multi-view-reconstructed scenes, integrating lightstage-captured or learned relighting priors for real-time, physically plausible control (Wang et al., 5 Jan 2026, Liang et al., 21 Jan 2026).
• Decomposition for creative control: LayerDecomp, TransLight, and similar systems bring layer-wise decomposition to the forefront of generative image editing, enabling precise, artist-guided manipulation of light, transparency, and effect layers (Yang et al., 2024, Li et al., 20 Aug 2025).

Generative image-based light decomposition thus forms a foundational component for next-generation, data-driven relighting, editing, and scene understanding systems, closing the gap between inverse rendering, photorealistic synthesis, and creative media applications.