Physically-Based Editing of Indoor Scene Lighting from a Single Image

Abstract: We present a method to edit complex indoor lighting from a single image with its predicted depth and light source segmentation masks. This is an extremely challenging problem that requires modeling complex light transport, and disentangling HDR lighting from material and geometry with only a partial LDR observation of the scene. We tackle this problem using two novel components: 1) a holistic scene reconstruction method that estimates scene reflectance and parametric 3D lighting, and 2) a neural rendering framework that re-renders the scene from our predictions. We use physically-based indoor light representations that allow for intuitive editing, and infer both visible and invisible light sources. Our neural rendering framework combines physically-based direct illumination and shadow rendering with deep networks to approximate global illumination. It can capture challenging lighting effects, such as soft shadows, directional lighting, specular materials, and interreflections. Previous single image inverse rendering methods usually entangle scene lighting and geometry and only support applications like object insertion. Instead, by combining parametric 3D lighting estimation with neural scene rendering, we demonstrate the first automatic method to achieve full scene relighting, including light source insertion, removal, and replacement, from a single image. All source code and data will be publicly released.

• The paper proposes a novel physically-based method that combines scene reconstruction and neural rendering to enable accurate indoor lighting editing.
• It introduces editable representations for both visible and invisible light sources by using specialized networks to predict parameters such as position, intensity, and orientation.
• Quantitative evaluations demonstrate high accuracy and flexibility in relighting tasks, supporting interactive scene editing and augmented reality applications.

Physically-Based Editing of Indoor Scene Lighting from a Single Image

The paper proposes a novel method for realistic editing of indoor scene lighting using only a single image input. This is accomplished through sophisticated physically-based model reconstructions and neural rendering techniques that facilitate accurate scene relighting, including editing of light sources, materials, and geometric configurations.

Methodology

The method is built on two primary components: a comprehensive scene reconstruction and a neural rendering framework.

1. Scene Reconstruction: The reconstruction process estimates scene reflectance and 3D parametric lighting. It predicts material parameters like albedo, normal, and roughness and separates visible from invisible light sources for intuitive editing. The light source modeling includes both ambient and directional components to capture complex light transport phenomena.
2. Neural Rendering Framework: This framework combines classical rendering techniques with neural networks to handle direct and indirect illumination, including soft shadows and interreflections. The approach integrates Monte Carlo ray tracing with learned components to approximate global illumination efficiently.

   Figure 1: Overview of the method from RGB input to edited scene rendering, utilizing neural rendering modules for direct and indirect shading.

Light Source Representation and Prediction

The study introduces editable representations for indoor light sources, which are essential for effective relighting:

• Visible and Invisible Light Sources: Prediction involves using neural networks to differentiate and model light sources with respect to their geometry and radiance properties. For windows, a multi-lobe spherical Gaussian model represents directional radiance influenced by external sun lighting (Figure 2).
• Prediction Networks: There are specific networks for lamps and windows to estimate parameters like position, intensity, and orientation. These predictions are refined through rendering loss, ensuring that the predicted lighting aligns with the observed shading patterns in the image.

Neural Rendering and Differentiable Modules

The neural renderer is designed for efficiency and quality:

• Direct and Indirect Shading Modules: These components use both geometric sampling and learned networks to simulate complex shadowing and specular reflections (Figure 3).
• Shadow Renderer: A depth-based approach combined with a CNN denoises shadow artifacts while handling occlusion boundaries robustly.
• Reflection and Lighting Prediction: The reflection module predicts local environment maps that allow the rendering of new objects with consistent lighting effects.

  Figure 4: The neural renderer models illumination accurately by overcoming limitations of path tracing with incomplete geometry.

Results and Performances

The method's capabilities are demonstrated with quantitative evaluations against state-of-the-art benchmarks in scene relighting tasks:

• Accuracy: It achieves high fidelity in predicting scene light parameters, allowing realistic object insertion and scene editing.
• Flexibility: The framework supports a variety of editing applications such as turning on fictional light sources and changing object materials with consistent global interaction effects (Figure 5).

The effectiveness is underscored by low errors in light prediction metrics and superior qualitative results compared with existing solutions.

Conclusion

The paper presents a significant advancement in relighting scenes using limited input data. This approach, through physically-based and neural rendering, opens up new possibilities for interactive scene editing and augmented reality applications. Future developments may extend the method to multi-view scenarios, enhancing robustness, and versatility.

代码公开使这些技术在实际应用中具有普遍性和拓展性。