ReLi3D: Relightable Multi-view 3D Reconstruction with Disentangled Illumination

Abstract: Reconstructing 3D assets from images has long required separate pipelines for geometry reconstruction, material estimation, and illumination recovery, each with distinct limitations and computational overhead. We present ReLi3D, the first unified end-to-end pipeline that simultaneously reconstructs complete 3D geometry, spatially-varying physically-based materials, and environment illumination from sparse multi-view images in under one second. Our key insight is that multi-view constraints can dramatically improve material and illumination disentanglement, a problem that remains fundamentally ill-posed for single-image methods. Key to our approach is the fusion of the multi-view input via a transformer cross-conditioning architecture, followed by a novel unified two-path prediction strategy. The first path predicts the object's structure and appearance, while the second path predicts the environment illumination from image background or object reflections. This, combined with a differentiable Monte Carlo multiple importance sampling renderer, creates an optimal illumination disentanglement training pipeline. In addition, with our mixed domain training protocol, which combines synthetic PBR datasets with real-world RGB captures, we establish generalizable results in geometry, material accuracy, and illumination quality. By unifying previously separate reconstruction tasks into a single feed-forward pass, we enable near-instantaneous generation of complete, relightable 3D assets. Project Page: https://reli3d.jdihlmann.com/

• The paper introduces a unified framework that jointly estimates geometry, spatially varying BRDF parameters, and HDR environment maps from sparse multi-view inputs in under 0.4 seconds.
• It employs a transformer-based cross-view fusion network with dual-path decoding and differentiable Monte Carlo rendering to disentangle material and illumination ambiguities.
• Empirical results demonstrate superior material decomposition and reconstruction accuracy, outperforming existing methods across synthetic and real-world datasets.

ReLi3D: Relightable Multi-view 3D Reconstruction with Disentangled Illumination

Introduction

The reconstruction of production-grade, relightable 3D assets from sparse image collections presents a fundamental challenge in computer vision and graphics. Existing approaches typically treat geometry, material, and environment illumination as disjoint objectives, which can lead to sub-optimal reconstructions and ill-posed decompositions, particularly in the context of feed-forward single-view methods. The "ReLi3D" system (2603.19753) presents the first unified end-to-end architecture for joint geometry, spatially-varying physically-based materials, and coherent HDR environment recovery from sparse, posed multi-view images, with inference times on the order of 0.3–0.4 seconds. This essay reviews the technical contributions and empirical findings of ReLi3D, situating its innovations within the state of the art of 3D asset generation.

Figure 1: ReLi3D outputs high-quality 3D meshes with physically based materials from sparse image inputs while disentangling illumination; all in 0.3 seconds, robustly across domain and view count.

Methodology

Unified Multi-View Feed-Forward Reconstruction Architecture

ReLi3D's core insight is the centrality of multi-view constraints to resolving the material–illumination ambiguity that plagues single-view inverse rendering. The architecture leverages a transformer-based cross-view fusion backbone. Multiple input images, together with foreground segmentation masks and camera poses, are encoded using DINOv2-derived tokens which feed into a hierarchical two-stream transformer: a "hero" view token serves as the reconstruction query, attended by latent and cross-view memory tokens for feature fusion. This produces a unified triplane feature bank consistent across arbitrary input views, forming the basis for subsequent geometry, material, and illumination estimation.

Figure 2: Multi-view fusion via cross-conditioning transformer, followed by dual-path prediction for geometry/appearance and illumination; unified through MIS-based differentiable rendering.

Geometry and Material Prediction Path

From the unified triplane features, geometry and spatially varying BRDF (svBRDF) parameters (albedo, roughness, metallic, normals) are regressed via task-specific MLP decoders. Meshes are extracted using Flexicubes for high isosurface fidelity, UV-mapped for texturing, and parameterized according to the Disney principled BRDF model. This pipeline enables accurate spatially-varying PBR materials, which are crucial for faithful relighting.

Illumination Prediction Path

A parallel transformer pathway predicts a compact latent encoding (RENI++), representing the HDR environment. To increase robustness, stochastic masking of backgrounds during training forces the model to alternate between (a) direct lighting estimation from environment pixels and (b) inferring lighting from indirect observations (material reflectance and shading cues).

Differentiable Monte Carlo MIS Rendering and Disentangled Training

The two-path outputs are unified via a differentiable Monte Carlo renderer employing Multiple Importance Sampling, enforcing physical consistency between materials and illumination. This mechanism is critical for disentanglement; the loss surface penalizes predicted material/illumination pairs that do not reconstruct held-out views. Mixed-domain supervision (synthetic with PBR ground truth, synthetic RGB, and real RGB captures) provides further regularization and domain coverage. Progressive training stages (volumetric → spherical Gaussian → MC rendering) stabilize learning and improve final decomposition fidelity.

Experimental Results

Spatially-Varying Material/Illumination Disentanglement

Quantitative and qualitative analysis demonstrates that ReLi3D yields high-fidelity material decomposition, with multi-view constraints driving significant improvements over global or single-parameter baselines (see PSNR, SSIM, and RMSE in tabular results). In the context of relighting in novel HDR environments, ReLi3D surpasses competitors—including generative 3D diffusion models—on both material metrics and visual realism.

Figure 3: Predicted PBR maps closely match ground truth, enabling detailed relighting; multi-view constraints recover true spatial variation unobtainable by single-view or global methods.

Figure 4: Even with a single view, ReLi3D recovers correct sun location and chromaticity; additional views and background cues progressively resolve direct illumination components.

Generalization, Speed, and Robustness

ReLi3D reconstructions saturate performance with only 4–8 input views. Geometry is competitive with SOTA (Chamfer Distance, F-score), though the key advantage is in material and illumination accuracy at a fraction of the compute overhead of diffusion-based approaches. Generalization extends to real-world UCO3D and Stanford ORB datasets, validating the mixed-domain training protocol.

Figure 5: Highly detailed PBR decompositions from single-view and multi-view input; uniquely supports true spatially-varying roughness/metallic parameters.

Figure 6: ReLi3D maintains accurate geometry and SVBRDF across diverse synthetic objects, outperforming alternative feed-forward and generative methods.

Figure 7: Robustness to imperfect segmentation, motion blur, and challenging illumination in real-world scenes; multi-view data improves site occlusion and basecolor recovery.

Real-World Decomposition and Illumination

In-the-wild settings, the architecture demonstrates robustness to noise, blur, and background clutter, with accurate separation of metals, dielectrics, and surface normals, and strong generalization across varied lighting conditions.

Figure 8: Real-world material maps (albedo, roughness, metallic, normal) faithfully separate metallic and non-metallic regions even on cluttered, blurry inputs.

Figure 9: Recovery of complex, mixed-material BRDFs on Blender Shiny objects; relighting under novel environments validates physical correctness of decomposition.

Illumination Prediction Baseline Comparison

ReLi3D outperforms SPAR3D and DiffusionLight baselines in both speed and environment map correctness; competitors either severely underfit environment or hallucinate.

Figure 10: Environment prediction: ReLi3D matches ground truth environment map structure, while alternatives hallucinate or over-smooth results.

Ablations and Limitations

Removal of the Monte Carlo renderer results in catastrophic collapse of disentanglement (Δ PSNR > 2 dB image quality drop). Progressive refinement phases are necessary for full convergence, with the final MC stage key for material sharpness. Triplane resolution is a limiting factor for geometric/texture detail relative to resource-intensive SOTA diffusion models. Ambiguous lighting (multiple collimated sources, strong self-shadowing) and transparent surfaces remain as failure modes. Performance degrades when camera pose accuracy is poor.

Figure 11: Failure cases: Baked-in lighting for strong self-shadowing and unreliable basecolor in severely dark scenes; ReLi3D retains advantage, but decomposition is not always perfect.

Theoretical and Practical Implications

The formalization of disentanglement as a multi-view, physically-constrained learning problem is significant. The architecture's ability to unify geometry, appearance, and environment into a single feed-forward pipeline with MC rendering constraints challenges the historical separation of inverse rendering sub-tasks. Practically, this advances the streamlining of asset digitization pipelines in VFX, AR, robotics, and industrial capture, shifting the operational bottleneck from material-illumination ambiguity to sensor quality and annotation accuracy.

The demonstrated cross-domain, mixed-supervision protocol further suggests that robust, physically plausible inverse rendering from in-the-wild data is attainable without reliance on extremely large-scale datasets or lengthy optimization, provided the scene is sufficiently observed (multi-view) and pose information is accurate.

Prospects for Future Research

ReLi3D's two-path and differentiable rendering architecture can be extended along multiple axes: higher-resolution triplane representations for further detail, improved environment priors (beyond RENI++), and incorporation of generative or diffusion-based super-resolution modules for mesh and material refinement. Transparent and layered material decomposition remain an open challenge. Integrating pose-agnostic or self-supervised pose estimation could remove the last practical constraint, facilitating adoption for unconstrained internet-scale 3D asset collections. Finally, the disentanglement architecture provides a blueprint that could be adapted to dynamic scenes or non-rigid, articulated objects.

Conclusion

ReLi3D establishes a new standard for fast, accurate, and physically disentangled multi-view 3D reconstruction. Its integration of geometry, spatially varying PBR materials, and HDR illumination into a single inference pipeline, enforced and regularized through Monte Carlo rendering, sets a powerful technical precedent. The empirical results offer evidence for the central role of multi-view constraints and differentiable physics-based rendering in resolving the fundamental ambiguities of appearance estimation. This work constitutes a reference point for both future asset digitization systems and theoretical studies of inverse rendering under uncertainty.

---

Reference: "ReLi3D: Relightable Multi-view 3D Reconstruction with Disentangled Illumination" (2603.19753)