NeRF-MIR: Neural Radiance for Masked & Mirror Scenes

• The paper introduces NeRF-MIR, a framework that extends neural radiance fields to restore masked images and model mirror reflections using advanced techniques like PERE and PIRE.
• It employs entropy-guided patch selection and progressively iterative self-restoration to robustly handle occlusions and improve performance metrics such as PSNR and SSIM.
• By integrating recursive ray tracing and reflection probability fields, NeRF-MIR achieves physically accurate synthesis for real-time VR, AR, and mixed reality applications.

NeRF-MIR encompasses several advances in the intersection of Neural Radiance Fields (NeRF) and challenging 3D scene scenarios, notably including masked image restoration, immersive rendering for mixed/virtual reality, and physically accurate synthesis involving mirrors and reflective surfaces. The common thread across these efforts is the integration and extension of neural volumetric rendering methods to handle occlusions, missing information, or complex light transport phenomena such as ideal specular reflection, all in a multi-view-consistent and self-supervised manner.

1. Core Concepts and Motivations

NeRF-MIR refers to frameworks which harness Neural Radiance Fields for Masked Image Restoration, Mixed or Mirror-aware Rendering, and other challenging compositional scene cases. In the context of masked image restoration, NeRF-MIR is designed to address the ubiquitous scenario in which captured images contain significant occlusions, noise, or artificially masked regions, impairing the volumetric reconstruction capability of standard NeRF pipelines (Huang et al., 24 Jan 2026). Alternative usages of the NeRF-MIR acronym encompass immersive neural graphics primitives for real-time VR (Li et al., 2022) and methods that incorporate explicit mirror or specular surface modeling for physically grounded synthesis of scenes containing reflective surfaces (&&&2&&&).

Motivations for developing NeRF-MIR approaches include:

• Resilience to corrupted or partially missing input data (e.g., due to occlusions, masking, distractor artifacts).
• Physically accurate modeling of phenomena not captured by classical radiance field volumetric rendering, such as perfect (planar) mirrors and recursive reflectivity.
• Enabling high-fidelity and coherent scene restoration for applications in VR, AR, robotics, and autonomous perception.

2. Patch-based Entropy Ray Emitting and Progressive Self-Restoration

The NeRF-MIR approach to masked image restoration (Huang et al., 24 Jan 2026) introduces two principal methodological innovations: Patch-based Entropy for Ray Emitting (PERE) and Progressively Iterative Restoration (PIRE).

Patch-based Entropy for Ray Emitting (PERE):

• Images are partitioned into non-overlapping patches. For each patch, local entropy across RGB channels is computed as $$H(P_k) = -\sum_{j=1}^3 \sum_{m=1}^M p_k^{(j)}(m)\log p_k^{(j)}(m)$$, where $p_k^{(j)}(m)$ is the empirical intensity distribution.
• Patches with higher entropy are allocated more rays for network optimization, as these contribute more scene information, while heavily masked or textureless regions are deprioritized.
• This weighting mechanism mitigates overfitting to non-informative or masked areas.

Progressively Iterative Restoration (PIRE):

• Training is organized into $T$ stages. At each stage $t$, previously masked pixels are replaced by the most recent NeRF rendering from iteration $t-1$.
• Each successive restoration stage refines the output for masked areas, leveraging predictions from multi-view-consistent NeRF in a self-training regime.
• This iterated completion process avoids collapse to trivial solutions (e.g., black dots), enhancing both local and global scene plausibility.

Dynamically-weighted Loss:

• Masked and unmasked rays are combined in a loss function weighted by $\alpha(t)$, which schedules the importance of learning from masked regions as training proceeds.

3. Mirror-aware NeRF Extensions and Physical Ray Tracing

Accurately rendering mirrors and specular surfaces in NeRF frameworks necessitates volume rendering pipelines capable of handling recursive light transport and planarity constraints. Mirror-NeRF (Zeng et al., 2023), an instantiation of NeRF-MIR, advances the field as follows:

• A unified neural radiance field is augmented with a reflection probability field $m(\mathbf{x}) \in [0,1]$, predicting the likelihood that a point $\mathbf{x}$ is perfectly specular.
• Surface normals are predicted via an auxiliary network head trained to match the analytic gradient of volume density, stabilizing the initialization of specular interactions.
• Whitted-style recursive ray tracing is deployed, integrating reflection via law-of-reflection direction swizzling for secondary rays: for a hit point $x$ with predicted normal $\mathbf{n}$ and incident direction $\mathbf{d}$, the reflected direction is $$\mathbf{d}_\mathrm{r} = \mathbf{d} - 2 (\mathbf{n} \cdot \mathbf{d})\mathbf{n}$$.
• Losses include photometric reconstruction, binary reflection mask supervision, normal regularization, and geometric planar constraints via quadruplet sampling.

This pipeline enables physically correct, multi-view-consistent reflection synthesis, avoids phantom geometry behind mirrors, and supports editing tasks such as mirror insertion, controlled mirror roughness via microfacet noise, and reflection substitution (Zeng et al., 2023).

4. Masked Dataset Construction, Benchmarking, and Quantitative Results

To permit rigorous benchmarking of NeRF-based masked image restoration, NeRF-MIR introduced three synthetic and real multi-view masked datasets: LLFF-M, Spaces-M, and Blender-M. These datasets are constructed by inserting random square or round masks (25%–90% coverage, fixed or random per view) into canonical NeRF datasets, with binary masks recording masked-ray locations (Huang et al., 24 Jan 2026).

Quantitatively, NeRF-MIR achieves significant performance gains over both 2D inpainting (HiFill, LaMa, TFill, ZITS++) and NeRF-based baselines (SPIn-NeRF, NeRF-In). Across LLFF-M and Spaces-M at 25%–50% mask levels, NeRF-MIR outperforms competitors by +1–3 dB in PSNR and +0.05 in SSIM. Incorporating PERE, dynamic loss, and PIRE yields cumulative improvements, with PIRE contributing +1.0 dB and eliminating persistent artifacts in masked regions. The methodology degrades gracefully as mask fraction increases to 90%; patch size choice has minimal performance impact. On real-world scenes (iPhone-captured, YOLO-detected occlusions), NeRF-MIR removes distractors and reconstructs geometry without requiring direct inpainting supervision (Huang et al., 24 Jan 2026).

5. Immersive and Real-Time Rendering for VR: System Architecture

NeRF-MIR as an immersive neural graphics system utilizes instant-ngp (Instant Neural Graphics Primitives) for accelerating volumetric rendering in VR applications (Li et al., 2022). The pipeline comprises:

• Preprocessing: Multiview RGB capture, structure-from-motion pose estimation (e.g., through COLMAP), and network training over multi-resolution hash-coded volumetric MLP.
• Real-time rendering loop: Stereoscopic camera pose acquisition, OpenGL-shared context between Unity and the rendering backend, volume marching through the radiance field, and NVIDIA DLSS-based super-resolution to scalable display resolutions.
• Latency mitigations: Asynchronous frame assembly, late-latching for pose prediction, zero-copy GPU buffer sharing.

Performance benchmarking demonstrates frame rates of 30–37 fps at $1280 \times 720$ per eye for small to moderate scenes, with higher complexity scenes requiring DLSS for interactive rates. The system does not support dynamic NeRFs natively, but suggests future directions involving foveated sampling, support for dynamic scenes, and mesh integration for physical interactions (Li et al., 2022).

6. Limitations and Future Directions

Identified limitations across NeRF-MIR variants include:

• Reliance on known camera poses and, in some real-world deployments, external object/mask detectors (e.g., YOLO, Segment-Anything).
• Computational cost due to staged self-training (PIRE) and recursive volumetric rendering (mirror rays) (Huang et al., 24 Jan 2026, Zeng et al., 2023).
• Inability to recover from all-views-masked regions (insufficient multi-view signal).
• Lack of explicit handling for refraction and non-planar (general curved) specular surfaces, though the Whitted-style ray-tracing architecture readily extends to further bounces and types (Zeng et al., 2023).
• Absence of built-in geometry for collisions in immersive systems and limited support for fully dynamic environments (Li et al., 2022).

Proposed research avenues include integrating pose refinement (BARF-style), leveraging hardware-accelerated NeRFs (e.g., 3D Gaussian splatting, instant-ngp), end-to-end occlusion/distraction discovery via uncertainty models, and generalizations to dynamic scenes and unbounded 360° capture domains (Huang et al., 24 Jan 2026, Li et al., 2022, Zeng et al., 2023).

7. Comparative Summary of NeRF-MIR Contributions

Approach / Variant	Principle Contribution	Application Domain
NeRF-MIR (Huang et al., 24 Jan 2026)	PERE entropy-guided ray emitting, PIRE self-training, dynamic loss for masked image restoration	Corrupted/masked multi-view NeRF inputs
Mirror-NeRF (Zeng et al., 2023)	Reflection probability field, Whitted-style ray tracing, recursive physical mirror rendering	Mirror scene synthesis, editing
NeRF-MIR Immersive (Li et al., 2022)	Instant-NGP for VR, DLSS super-resolution, Unity integration	Real-time immersive VR / AR

These advances collectively extend the NeRF framework’s applicability to scenarios with missing data, complex reflectance, and high-performance immersive requirements, illustrating the continued expansion of neural volumetric graphics primitives.