Multi-View Compensation Loss for 3D Reconstruction

• MVCL is a loss function that leverages redundant, clean observations across views to improve photometric and geometric consistency in neural 3D reconstructions.
• It aggregates multi-view evidence to mitigate occlusions and distractors, thereby enhancing radiance field learning with improved PSNR and SSIM metrics.
• MVCL employs a view-dependent compensation network to align noisy monocular normal priors, resulting in smoother surfaces and sharper structural details.

Multi-View Compensation Loss (MVCL) is a class of losses designed to address supervision gaps and inconsistencies among multiple views in neural 3D scene reconstruction. MVCL is formulated to leverage redundant, clean observations of scene content across multiple input images—propagating reliable supervision information to views partially corrupted by occlusions, distractors, or measurement bias. Notably, its two principal application domains are (i) photometric aggregation for robust radiance field learning under distractor removal (Huang et al., 16 Jan 2026), and (ii) view-dependent compensation for neural implicit surface alignment with noisy monocular geometric priors (Chen et al., 2024).

1. Formal Definitions and Variants

Radiance Field MVCL:

Let $\mathcal{R}$ be the set of rays intersecting unmasked regions in a multi-camera setup, and $K$ be the number of views. For each rendered color $\tilde{\mathbf C}(\mathbf r)$ and ground-truth color $\mathbf C(\mathbf r)$ at ray $\mathbf r$, define $$n_r = \sum_{k=1}^K \mathbb{I}(\mathbf r \notin M_k)$$ as the count of views where that ray is unmasked ($\mathcal{M} = \{M_1,\ldots, M_K\}$ are the detector bounding box regions per view). The MVCL is

$$\mathcal L_{\mathrm{MVCL}} = \frac{s\,n_r}{|\mathcal{R}|} \sum_{\mathbf r \in \mathcal{R}} \|\tilde{\mathbf C}(\mathbf r)-\mathbf C(\mathbf r)\|_2$$

where $s$ is a scale factor for balancing with other loss terms (Huang et al., 16 Jan 2026).

Normal Alignment MVCL:

Given input normal priors $\mathbf{N}(\bm r)$ and rendered, compensated normals $\mathbf{N}^{\mathrm{comp}}(\bm r)$ at ray $\bm r$:

$$\mathcal{L}_n = \sum_{\bm r\in\mathcal R} \|\mathbf{N}^{\mathrm{comp}}(\bm r) - \mathbf{N}(\bm r)\|_1 + \|1 - (\mathbf{N}^{\mathrm{comp}}(\bm r)^\top \mathbf{N}(\bm r))\|_1$$

This formulation enforces both L1 norm alignment and cosine (angular) consistency (Chen et al., 2024).

2. Motivation and Theoretical Basis

MVCL is motivated by the incomplete or inconsistent supervision that commonly arises in multi-view 3D learning:

• Distractor Removal (Photometric Aggregation): Detected distractors occlude parts of the scene; solely per-view losses leave masked regions unconstrained and prone to artifact. MVCL aggregates the “clean” supervision available in other views for the same 3D locus, effectively distributing photometric evidence to fill in masked or uncertain regions (Huang et al., 16 Jan 2026).
• Normal Compensation (SDF Alignment): Monocular normal priors, while available in abundance, exhibit view-dependent biases; naïve supervision leads to reconstruction artifacts due to inconsistent gradients. MVCL introduces a learned view-dependent transformation (via a small MLP) that adapts per-view normal priors to a globally consistent geometry (Chen et al., 2024).

3. Multi-View Aggregation Mechanisms

MVCL operationalizes cross-view information flow:

• In radiance fields: For any $(u, v)$ pixel coordinate, rays from $K$ cameras may or may not intersect masked areas. The loss term is upweighted for ray locations visible in a greater number of views ($n_{r_k}$), propelling the model to “hallucinate” masked content using consensus cues from clean images.
• In SDF-based reconstruction: View-dependent normal biases are captured and corrected by an auxiliary network $f_n$ outputting Euler angles $(\gamma, \beta, \theta)$, which rotate the SDF normal into alignment with the prior. The compensated norm is volume-rendered and compared across views.

These approaches ensure that multi-view consistency is fostered not by hard constraints but by soft, aggregated alignment.

4. Loss Weighting, Schedules, and Integration

Weighting Strategies

• Radiance Field Use Case: The scale $s$ is typically set to 1, and the outer weight $\lambda_1$ is used to modulate the MVCL influence, with experimental values $\lambda_1=0.01$ for initialization and higher values later for detail refinement. The loss integrates as

$$\mathcal L_\text{total} = \mathcal L_\text{rgb} + \lambda_1\,\mathcal L_\text{MVCL} + \lambda_2\,\mathcal L_\text{LPIPS}$$

where $\mathcal L_\text{LPIPS}$ encourages perceptual fidelity (Huang et al., 16 Jan 2026).

• SDF Use Case: The MVCL term $\mathcal{L}_n$ is weighted by $\lambda_n=0.1$. Training proceeds in two stages: the compensation MLP is omitted for the first $\sim$20k iterations to avoid lock-in to noisy normals, then introduced and jointly optimized with other losses.

Integration with Other Objectives

MVCL is complementary to photometric and perceptual losses (e.g., $\mathcal L_\text{rgb}$, $\mathcal L_\text{LPIPS}$, eikonal losses), jointly fostering geometric and textural fidelity as well as multi-view consistency.

5. Architectural and Implementation Details

• Distractor Removal (IDDR-NGP): MVCL operates on Instant-NGP’s implicit representations; only rays outside detected distractor regions are considered. Error terms are upweighted based on view visibility counts (Huang et al., 16 Jan 2026).
• Normal Compensation (NC-SDF):
  • Compensation is handled by $f_n$, a 4-layer MLP generating rotation parameters for each sample point.
  • Compensated normals are rotation-transformed SDF normals, then volume-rendered per-ray with sample weights $T_i \alpha_i$ ($N=64$ samples per ray).
  • The loss is calculated every iteration for all sampled rays; pixel sampling for normals uses the same schedule as for color.
  • The architecture and training regime mirror NeuS, with only the compensation MLP as an addition.

6. Empirical Evaluation and Quantitative Impact

Experimental ablations in both fields highlight the standalone and combined contributions of MVCL:

Configuration	PSNR (outdoor)	SSIM (outdoor)	F-score (indoor)
Baseline (no MVCL)	30.66	0.94	0.749 (Chen et al., 2024)
+ LPIPS	30.76	0.95	—
+ MVCL	31.04	0.95	0.781
+ LPIPS+MVCL	32.58	0.96	—

Key observations:

• In distractor removal (Huang et al., 16 Jan 2026), MVCL alone increases PSNR by 0.38 dB (outdoor) and 1.35 dB (indoor) over the base model, with higher SSIM and lower LPIPS when combined with perceptual loss.
• In NC-SDF (Chen et al., 2024), MVCL yields a +0.032 to +0.033 absolute F-score improvement (4.3% relative). Qualitatively, surfaces become globally smoother and exhibit enhanced sharpness on edges and fine details.

7. Research Significance and Broader Implications

MVCL establishes a robust, generalizable mechanism for propagating reliable supervision across multi-view inputs—addressing fundamental limitations in both radiance field and implicit surface learning under incomplete or inconsistent data. It allows the integration of noisy, view-dependent priors with learned compensation, supports more faithful reconstructions in the presence of real-world distractors, and delivers measurable improvements in both geometric and appearance fidelity.

A plausible implication is that the MVCL paradigm can extend to other modalities where multi-view redundancy and occlusion are prevalent, and where supervision from consensus can overcome view-specific artifacts or noise. Its integration with learned perceptual metrics (e.g., LPIPS) and adaptive schedules further underscores its flexibility for diverse scene reconstruction objectives.

References:

• "IDDR-NGP: Incorporating Detectors for Distractor Removal with Instant Neural Radiance Field" (Huang et al., 16 Jan 2026)
• "NC-SDF: Enhancing Indoor Scene Reconstruction Using Neural SDFs with View-Dependent Normal Compensation" (Chen et al., 2024)