V-RGBX: Video Editing with Accurate Controls over Intrinsic Properties

Abstract: Large-scale video generation models have shown remarkable potential in modeling photorealistic appearance and lighting interactions in real-world scenes. However, a closed-loop framework that jointly understands intrinsic scene properties (e.g., albedo, normal, material, and irradiance), leverages them for video synthesis, and supports editable intrinsic representations remains unexplored. We present V-RGBX, the first end-to-end framework for intrinsic-aware video editing. V-RGBX unifies three key capabilities: (1) video inverse rendering into intrinsic channels, (2) photorealistic video synthesis from these intrinsic representations, and (3) keyframe-based video editing conditioned on intrinsic channels. At the core of V-RGBX is an interleaved conditioning mechanism that enables intuitive, physically grounded video editing through user-selected keyframes, supporting flexible manipulation of any intrinsic modality. Extensive qualitative and quantitative results show that V-RGBX produces temporally consistent, photorealistic videos while propagating keyframe edits across sequences in a physically plausible manner. We demonstrate its effectiveness in diverse applications, including object appearance editing and scene-level relighting, surpassing the performance of prior methods.

• The paper introduces an end-to-end framework that decouples video intrinsic properties, enabling physically coherent and temporally stable editing.
• It leverages a novel interleaved conditioning scheme with diffusion transformer backbones for faithful inverse and forward rendering, as demonstrated by improved PSNR, SSIM, and LPIPS metrics.
• Keyframe-based intrinsic-aware editing allows precise relighting, texture, and material modifications, reducing edit drift and enhancing consistency across videos.

V-RGBX: Video Editing with Accurate Controls over Intrinsic Properties

Introduction and Motivation

The paper "V-RGBX: Video Editing with Accurate Controls over Intrinsic Properties" (2512.11799) introduces an end-to-end video editing framework that operates in the intrinsic property space—explicitly disentangling and conditioning on scene attributes such as albedo, normal, material, and irradiance. Prior controllable video synthesis methods, mostly based on video diffusion, have achieved high-fidelity pixel-space generation but lack mechanisms for independent and physically coherent control over these intrinsic modalities. This limitation restricts applications requiring semantically meaningful manipulations, such as relighting or object retexturing, where isolating intrinsic factors is paramount for edit propagation and temporal stability. V-RGBX closes this gap via a unified framework that performs video inverse rendering (RGB$\rightarrow$X), photorealistic synthesis from structured intrinsic representations (X$\rightarrow$RGB), and keyframe-based intrinsic-aware editing with precise temporal propagation.

Methodology

V-RGBX consists of three principal components: an Inverse Renderer for extracting per-frame intrinsic channels, an Intrinsic Conditioning Sampler for forming temporally interleaved and edited intrinsic conditioning sequences, and a Forward Renderer for synthesizing the final edited RGB video conditioned on both intrinsic channels and appearance reference keyframes.

The framework relies on diffusion transformer (DiT) backbones and employs a novel interleaved intrinsic conditioning scheme coupled with a learnable type-aware embedding, ensuring the preservation of modality identity within compressed temporal chunks. When propagating keyframe edits, the system identifies edited modalities and constructs temporal conditioning by interleaving edited and untouched intrinsic sequences, maximizing temporal consistency and disentanglement. The Forward Renderer incorporates keyframe-based reference encoding as a complementary source of global scene cues, enhancing reconstruction fidelity and appearance style consistency. Training utilizes v-prediction objectives; at inference, classifier-free guidance balances edit consistency and fidelity.

The overall system architecture is summarized below.

Figure 1: The architecture of V-RGBX, highlighting the inverse renderer, intrinsic conditioning sampler, and forward renderer, which together enable intrinsic-aware video editing.

Inverse and Forward Rendering Performance

V-RGBX achieves strong performance in both inverse rendering (RGB$\rightarrow$X) and forward rendering (X$\rightarrow$RGB) tasks. On diverse benchmarks, including synthetic Evermotion and challenging real-world RealEstate10K videos, the model surpasses intrinsic-aware and appearance-only baselines in pixel-level metrics (PSNR, SSIM, LPIPS) and video-level metrics (FVD, smoothness). Quantitative evaluation demonstrates improved albedo, normal, and irradiance estimation relative to RGBX and DiffusionRenderer, with robust temporal coherence across all intrinsic channels.

Qualitative ablation studies confirm that the explicit intrinsic type embedding and reference conditioning are both critical for the reduction of temporal flicker, modality confusion, and color instability in composited sequences. The RGB$\rightarrow$X$\rightarrow$RGB cycle consistency evaluation further establishes that the model maintains sufficient information in the intrinsic domain for reconstructing faithful appearance and structure across long sequences.

Figure 2: V-RGBX decomposes RGB videos into stable and spatially coherent intrinsic channels on synthetic Evermotion scenes.

Figure 3: V-RGBX achieves higher fidelity and more temporally consistent albedo/normal decomposition compared to baselines.

Figure 4: Scene generation from intrinsic channels (X$\rightarrow$RGB) reveals the advantages of temporal coherence and shadow modeling.

Keyframe-based Intrinsic-aware Video Editing

The core contribution of V-RGBX lies in enabling explicit, keyframe-driven edits across intrinsic modalities and reliable propagation throughout the video. The workflow begins with user edits provided on one or several keyframes using external tools; the edited frames are decomposed into their respective intrinsic maps. The Intrinsic Conditioning Sampler then produces a temporally aligned sequence that splices the edited intrinsics with randomly chosen, unedited ones for frames not modified, avoiding conflicts and enhancing consistency.

This enables physically grounded propagation of edits such as relighting, texture changes (albedo), material property adjustment, or geometry refinement (normal modifications). Unlike global or latent conditioning used in previous diffusion models, V-RGBX supports multi-modality, localized edits, and complex edit combinations—propagating only the desired intrinsic changes while preserving untouched content spatiotemporally.

Representative results underline the system’s superiority in controlling light and shadow propagation, accurate texture transfer, and faithful material editing with minimal drift or cross-modal contamination.

Figure 5: V-RGBX demonstrates consistent keyframe-edit video propagation: property drifting and entanglement seen in baselines are mitigated.

Figure 6: Editing light color and shadow (relighting) is naturally supported and produces plausible novel illumination.

Figure 7: Intermediate results across various edit types show intrinsic-space modification and reliable propagation in V-RGBX.

Architectural Components and Ablations

Ablation studies demonstrate the significance of both the Intrinsic Type Embedding module and the Reference Condition. Removing type embedding yields cross-modal confusion and temporal artifacts; adding the reference condition aligns appearance and enables finer detail preservation. The design leverages DiT backbones (WAN-2.1) for scalable video-level modeling, with intrinsic-aware conditioning facilitating generalization to incomplete or sparse edit specifications. The method is extensible and amenable to additional modalities or new edit types.

Figure 8: Ablating ITE and reference condition degrades temporal stability, modality disambiguation, and color fidelity.

Discussion, Implications, and Future Directions

V-RGBX advances video generation by bridging physically-motivated inverse/forward rendering and generative diffusion, paving the way for semantically meaningful and temporally coherent intrinsic-aware editing workflows. This intrinsic decoupling enables finer control for professional postproduction tasks, VFX, content creation, and scientific applications (e.g., scene relighting, asset insertion). The disentanglement of physical properties also holds promise for data augmentation in vision tasks, sim2real transfer, or inversion/editing in robotics and simulation.

Current constraints include limited OOD generalization (due to synthetic indoor-only pretraining) and single-modality-per-frame conditioning, which may restrict complex, layered edits. Scaling the system to real-time scenarios and adapting to outdoor or long-form content will require further architectural and dataset innovations. Integrating long-range memory models and multi-modal fusion could enhance persistence and edit flexibility. The framework’s core disentanglement principles may further inform video-based world modeling, real-time simulation, and robust scene understanding.

Conclusion

V-RGBX establishes a unified, edit- and modality-aware diffusion-based video generation pipeline that delivers robust, temporally stable, and physically consistent propagation of fine-grained intrinsic edits. By integrating video inverse rendering, multi-modality conditioning, and keyframe-based edit propagation, it sets a new direction for controllable video synthesis and editing beyond pixel-space manipulation, with high impact for practical and theoretical research in video generation.