Over++: Generative Video Compositing for Layer Interaction Effects

Abstract: In professional video compositing workflows, artists must manually create environmental interactions-such as shadows, reflections, dust, and splashes-between foreground subjects and background layers. Existing video generative models struggle to preserve the input video while adding such effects, and current video inpainting methods either require costly per-frame masks or yield implausible results. We introduce augmented compositing, a new task that synthesizes realistic, semi-transparent environmental effects conditioned on text prompts and input video layers, while preserving the original scene. To address this task, we present Over++, a video effect generation framework that makes no assumptions about camera pose, scene stationarity, or depth supervision. We construct a paired effect dataset tailored for this task and introduce an unpaired augmentation strategy that preserves text-driven editability. Our method also supports optional mask control and keyframe guidance without requiring dense annotations. Despite training on limited data, Over++ produces diverse and realistic environmental effects and outperforms existing baselines in both effect generation and scene preservation.

• The paper introduces a diffusion-based framework that synthesizes realistic environmental effects for video compositing.
• It leverages tri-mask supervision and hybrid paired/unpaired training to preserve foreground fidelity and enable robust, prompt-driven control.
• Experimental results show improved metrics and user preference over competing VFX methods, emphasizing both quality and control.

Over++: A Generative Framework for Video Compositing of Layer Interaction Effects

Problem Formulation and Motivation

The Over++ framework addresses the challenge of synthesizing physically grounded, prompt-driven environmental interaction effects—such as shadows, reflections, dust, and splashes—at the interface of foreground and background layers in video compositing. Traditional workflows in professional VFX rely on extensive manual effort to generate these effects, which are critical for scene realism, yet current generative video models lack the controllability and fidelity required for seamless integration into pipeline-driven compositing environments. Existing solutions are limited by inadequate support for preserving the original video content, inflexibility in user control (spatial or semantic), and reliance on labor-intensive, per-frame mask annotations.

Over++ formalizes an "augmented compositing" task: Given input video layers (foreground, background), a composite lacking effects, a text prompt describing the desired environmental interaction, and (optionally) a user-supplied spatio-temporal mask, the system should synthesize new video frames that add plausible, semi-transparent effects in a manner consistent with the original content and the prompt, while robustly supporting both mask-guided and mask-free regimes.

Methodology and Architectural Innovations

Over++ is instantiated as a video diffusion model $\mathcal{G}$ trained to synthesize environmental effects conditioned on composite input video, spatial effect masks, and textual prompts. Unlike prior inpainting-based systems, which erase the masked regions in the latent space, Over++ preserves the full latent context during denoising to maintain scene continuity and suppress hallucinated artifacts. The model architecture supports mask guidance as well as flexible prompt-driven effect synthesis, with a design that is compatible with standard video diffusion model backbones (e.g., CogVideoX-5B).

Data scarcity is mitigated by an advanced dataset construction pipeline: leveraging methods such as Omnimatte decomposition, the authors distill paired data (with/without effects) from real and synthetic videos. They also introduce a form of "tri-mask" supervision and augment training with unpaired, text-driven effect videos synthesized from pretrained models, preserving the model's instruction-following and prompt-editing capabilities.

For effect localization, Over++ uses a robust difference-based masking technique, applying grayscale thresholding and morphological filtering to compute binary masks from paired videos, which balances pixel-level precision with practical robustness to loose or imperfect annotations typical of real-world user input.

To prevent prompt drift and maintain language-conditioned generation, Over++ blends paired (supervised) and unpaired (text-only) examples during training, relying on classifier-free guidance during inference. This hybrid protocol is critical to supporting prompt-controlled generation with a high degree of semantic fidelity, as confirmed by ablation experiments reported in the work.

Experimental Validation

Empirical evaluation demonstrates that Over++ achieves numerically superior or comparable results to baselines across image- and video-level metrics, including CLIP-based directional scores (quantifying prompt alignment), SSIM, PSNR, LPIPS, FVD, VMAF, and VBench. Notably, Over++ exhibits a strong balance: it produces prompt-consistent, spatially localized, physically plausible effects without degrading foreground/background fidelity, a failure mode observed in several competing approaches.

Qualitative comparisons with AnyV2V, LoRA-Edit, VACE, and Runway Aleph reveal that alternative models frequently alter input video content undesirably, whereas Over++ preserves composition while synthesizing realistic interaction effects. A user study involving both professional VFX artists and general users found a clear preference (especially among professionals) for the fidelity to prompt, mask, and preservation of original content offered by Over++.

Extensions, Usability, and Robustness

Over++ offers significant workflow flexibility:

• Spatial Control: Mask-guided generation enables users to constrain effects spatially or temporally, with demonstrated robustness to coarse or erroneous masks.
• Semantic Control: Prompt-based modulation supports nuanced changes to effect style, color, and magnitude, enhanced by classifier-free guidance scaling.
• Keyframe Annotation: The hybrid tri-mask supervision protocol allows for economical user input, interpolating effects between sparsely annotated keyframes while maintaining temporal consistency.

Ablation studies indicate that the inclusion of unpaired, prompt-diverse data is essential for strong prompt-following capability; models trained with paired data alone fail to reliably modulate effect type or appearance in response to text prompts.

Theoretical and Practical Implications

The Over++ approach exemplifies a shift in generative compositing: from unconditional or weakly controlled effect generation to a model that offers explicit, compositional integration of physical interactions, semantic controls, and robust preservation of primary content. The methodology eliminates the need for motion/geometry priors or dense annotations, removing a bottleneck for scalable integration into professional video effects pipelines. The unified treatment of mask and prompt as jointly composable control signals is particularly noteworthy, as it reconciles semantic and spatial constraints in a single generative framework.

The capacity of Over++ to generalize from relatively limited paired data to diverse and in-the-wild effect settings demonstrates the efficacy of its hybrid training strategy. The reliance on strong pre-trained priors, judicious use of synthetic and real decompositions, and explicit mitigation of prompt drift inform future best practices for controllable generative video editing.

Limitations and Future Directions

While Over++ achieves high-quality, temporally consistent effect synthesis, there are recognized limitations. Absolute pixel-level reconstruction of inputs is impeded by VAE bottlenecks; test-time optimization and explicit high-fidelity modules may be required for applications mandating pixel-perfect output. Some failure cases are observed for challenging masks or backgrounds with complex non-rigid phenomena. The approach does not directly address color harmonization or relighting; integration with harmonization modules is a logical extension.

Broader directions include alignment with state-of-the-art video generation backbones (e.g., Lumiere, Veo3), generalization to more diverse effect spaces (such as volumetric or multi-modal effects), and incorporation of learned physics priors for further realism and control.

Conclusion

Over++ introduces a technically rigorous, highly controllable framework for synthesizing semi-transparent environmental effects that bridge foreground-background interactions in video compositing (2512.19661). Its architectural innovations and tailored data curation pipeline enable prompt- and mask-driven effect generation without compromising source fidelity, supporting both professional and non-expert workflows. The research provides a foundation for future work on unified, physically grounded generative video editing and compositing within both academic and production settings.