Flash-Splat: 3D Reflection Removal with Flash Cues and Gaussian Splats

Abstract: We introduce a simple yet effective approach for separating transmitted and reflected light. Our key insight is that the powerful novel view synthesis capabilities provided by modern inverse rendering methods (e.g.,~3D Gaussian splatting) allow one to perform flash/no-flash reflection separation using unpaired measurements -- this relaxation dramatically simplifies image acquisition over conventional paired flash/no-flash reflection separation methods. Through extensive real-world experiments, we demonstrate our method, Flash-Splat, accurately reconstructs both transmitted and reflected scenes in 3D. Our method outperforms existing 3D reflection separation methods, which do not leverage illumination control, by a large margin. Our project webpage is at https://flash-splat.github.io/.

• The paper presents a novel unsupervised method that leverages unpaired flash and no-flash images to separate transmitted and reflected light in 3D scenes.
• It employs 3D Gaussian splatting to synthesize pseudo-paired images and enhance reconstruction accuracy with flash cues.
• Experimental results demonstrate that Flash-Splat outperforms existing approaches in both image quality and practical applicability.

Flash-Splat: 3D Reflection Removal with Flash Cues and Gaussian Splats

The paper "Flash-Splat: 3D Reflection Removal with Flash Cues and Gaussian Splats" presents a novel approach to the challenging problem of separating transmitted and reflected light in 3D scenes. This work introduces a framework, termed Flash-Splat, which leverages flash cues without necessitating paired flash/no-flash captures, making it practical for in-the-wild scenarios.

Introduction and Motivation

The separation of transmitted and reflected scenes from images is a well-studied problem in computational photography. Traditional methods often require assumptions or high-quality paired images, limiting their applicability. The authors propose a robust, unsupervised method that addresses these limitations by employing modern inverse rendering techniques and Gaussian splatting, enabling accurate 3D reconstruction and reflection separation.

Methodological Innovations

The key insight of Flash-Splat lies in its ability to utilize unpaired flash and no-flash images. The method sidesteps the stringent requirement for perfectly aligned image pairs by employing:

1. 3D Gaussian Splatting: This facilitates the creation of pseudo-paired flash/no-flash images and 3D representations. By capturing separate sequences of flash and no-flash views, the system reconstructs images and scenes by effectively synthesizing the missing counterpart using powerful inverse rendering techniques.
2. Flash Cues: Flash cues are used as priors to reduce the ill-posedness of the reflection separation task. By synthesizing a flash image for a given view when only a no-flash image is available, and vice versa, Flash-Splat performs linearity regularization to ensure the recovered transmission component is free of reflections.
3. Initialization via Pseudo Pairs: The method initializes 3D representations using sparse point clouds derived from the differences between flash and no-flash reconstructions, enhancing convergence in distinguishing transmission from reflection.

Experimental Analysis

The paper conducts extensive real-world experiments to validate Flash-Splat. The experiments demonstrate the superiority of this method over contemporary approaches, such as NeRFReN and traditional deep learning methods, in both indoor and outdoor settings. Results indicate that the proposed approach achieves superior separation, corroborated by qualitative assessments and quantitative metrics like PSNR and LPIPS.

Practical and Theoretical Implications

Practically, this method significantly eases data acquisition procedures for reflection removal, broadening its applicability in environments where capturing perfectly aligned flash/no-flash images is impractical. Theoretically, Flash-Splat enriches the set of tools available for inverse rendering tasks by illustrating the synergy between illumination control and advanced neural scene representation techniques.

Future Directions

The paper opens up several avenues for future research. Enhancing the method's robustness to dynamic scenes and curved reflective surfaces would be valuable. Additionally, integrating other cues, such as polarization, could further refine separation quality. Continued exploration of advanced differentiable rendering techniques may yield even more efficient solutions for related vision tasks.

In summary, Flash-Splat represents a substantial contribution to the domain of reflection removal in inverse rendering, offering a scalable solution that outperforms existing methods by a considerable margin. Its innovative use of unpaired flash/no-flash cues marks a significant step toward more accessible and reliable 3D scene parsing.