Inverse Rendering in Urban Scenes

• Inverse rendering in urban scenes is the computational process of extracting intrinsic properties like 3D geometry, albedo, and lighting from captured urban imagery.
• Hybrid approaches using techniques such as hash-grid neural volumes, signed distance fields, and 3D Gaussian primitives are employed to address challenges from uncontrolled lighting and complex occlusions.
• This method enables applications in novel-view synthesis, relighting, photorealistic scene reconstruction, and autonomous navigation in urban environments.

Inverse rendering in urban scenes refers to the computational recovery of scene-intrinsic properties—including 3D geometry, materials (albedo, BRDF parameters), and illumination—directly from captured imagery, often in highly complex outdoor environments. This process underpins photorealistic scene understanding, novel-view and relit rendering, object insertion, and autonomous navigation. Urban-scale inverse rendering must address wide-baseline, multiview setups with uncontrolled, time-varying lighting, heterogeneous materials, and large, intricate geometry typical of cities. The problem is fundamentally ill-posed: identical pixel values may arise from multiple combinations of shape, reflectance, and lighting, with the solution space further constrained by the physics of image formation, data priors, and domain-specific regularization.

1. The Inverse Rendering Problem and Urban Scene Challenges

Inverse rendering aims to factor observed radiance into intrinsic scene properties by inverting the rendering equation: $$L_o(x,\omega_o) = L_e(x,\omega_o) + \int_{\Omega} f_r(x,\omega_i,\omega_o) \, L_i(x,\omega_i) \, \max(0, \mathbf{n} \cdot \omega_i) \, d\omega_i$$ where $L_o$ is outgoing radiance, $f_r$ is the BRDF, $L_i$ is incident radiance, and $\mathbf{n}$ is the surface normal. In cities, both multi-illumination (rapid variation in outdoor lighting) and complex indirect light/shadow effects from dense building arrangements make this inverse problem drastically underdetermined (Wang et al., 1 Feb 2026). Challenges specific to urban domains include:

• Reconstructing geometry at both building and street scale;
• Separating albedo and global illumination in the presence of interreflections and cast shadows;
• Handling material and lighting variation across hundreds of sky maps or daylight conditions;
• Robustness to occlusions, transient objects, and sensor noise.

A correct inverse solution must yield illumination-invariant albedo and normals, assigning all appearance variation across time-of-day or weather to the shading term.

2. Scene Representations and Physical Foundations

Methods adopt hybrid scene representations to manage the scale and complexity of urban environments:

• Hash-grid neural volumes (as in UrbanIR), where geometry is encoded via a spatial hash-grid coupled with MLPs predicting density, surface normals, albedo, and semantics (Lin et al., 2023).
• Signed distance fields (SDFs) for watertight mesh extraction and smooth surface reconstruction, which allow for frequency-aware (material-adaptive) geometric regularization (Xie et al., 2023).
• 3D Gaussian primitives enabling scene graphs that support both static backgrounds and dynamic moving objects, as well as multi-modal material encoding and LiDAR radiance simulation (Chen et al., 23 Jul 2025).
• Mesh-based volumetric G-buffers and explicit secondary-ray integration for accurate light transport, surface-based BRDF parameterization, and MC evaluation of shadowing (Wang et al., 2023).

Material properties are parameterized as spatially varying fields, supporting both diffuse and specular response, often through neural fields tied to instance or semantic segmentation for intra-class consistency (Xie et al., 2023). Illumination can be represented via analytic sun+sky models, HDR sky domes, or spherical harmonics, depending on the physical accuracy required and data availability (Lin et al., 2023, Wang et al., 1 Feb 2026, Chen et al., 23 Jul 2025).

3. Algorithms and Loss Functions

Urban-scale inverse rendering involves complex multi-task learning protocols and loss formulations. Key components include:

Joint Optimization Objective (see UrbanIR)

$$\min_{\theta,L}\left[ \mathcal{L}_{\text{render}} + \mathcal{L}_{\text{deshadow}} + \mathcal{L}_{\text{visibility}} + \mathcal{L}_{\text{normal}} + \mathcal{L}_{\text{semantics}} \right]$$

Where main losses cover:

• Photometric reconstruction: pixelwise difference between observed and rendered images.
• Deshadow loss: ensures shadow-free albedo by penalizing discrepancy with shadow-removed reference imagery.
• Visibility/occlusion: enforces that the volume-rendered transmittance field matches shadow masks and geometric occluders.
• Normal/semantic consistency: aligns estimated normals and high-level class-based priors.

Advanced Losses:

• Curvature and Eikonal regularization for SDFs to preserve sharp edges in glass but detail in stone (Xie et al., 2023).
• Material-parameter clustering using segmentation-guided Hungarian or KMeans loss to enforce spatial material coherence (Xie et al., 2023).
• Multimodal consistency between RGB and LiDAR-derived albedo and normals, propagating information across spectral domains (Chen et al., 23 Jul 2025).

Optimization follows a multi-stage protocol: geometry and coarse appearance are learned first, followed by finer material and illumination refinement, often with alternating or annealed loss weights to balance competing signals.

4. Illumination, Shading, and Multi-illumination Datasets

Physically-based illumination models are central to relightability and shadow accuracy. These include:

• Analytic sky models (e.g., Hosek-Wilkie, parametric sun + uniform sky) with few learnable parameters, allowing plausible daylight or nighttime scenes (Lin et al., 2023, Xie et al., 2023).
• SH-based global illumination for efficient encoding of environment lighting in Lambertian-dominated scenes (Yu et al., 2021).
• Explicit HDR sky domes or MC integration for full PBR pipelines, enabling specular effects and high dynamic range (Wang et al., 2023, Chen et al., 23 Jul 2025).

Training and benchmarking rely increasingly on multi-illumination synthetic datasets such as LightCity, which offer dense multi-view, multi-illumination imagery, ground-truth albedo/shape/material, and diverse environmental variation (Wang et al., 1 Feb 2026). These datasets reveal that models often bleed shading into albedo or struggle with indirect light; physically-based priors and multi-illumination supervision are crucial for robust decomposition.

5. Representative Methods and Comparative Performance

UrbanIR: Employs a neural volume model with tight coupling between geometry, albedo, normals, and semantics, robust physically-based visibility, and parametric sun-sky illumination. Its built-in shadow volume allows for sharp, geometry-consistent shadows in relighting or object insertion, outperforming NeRF-OSR and mesh-based approaches in view synthesis and shadow de-baking (Lin et al., 2023).

FEGR: Proposes a two-layer strategy—volumetric neural fields for primary ray G-buffer construction, and mesh-based MC shading for secondary rays enabling BRDFs and high dynamic range relighting. Semantic priors prevent shadow “bake-in” and yield strong geometry-material-lighting disentanglement. On benchmarks, FEGR achieves higher PSNR for novel view/lighting conditions and is preferred in human evaluations for object insertion realism (Wang et al., 2023).

InvRGB+L: Integrates a physics-based LiDAR shading model and enforces material consistency between RGB and LiDAR modalities, leveraging 3D Gaussian splatting for robust scene representation. This leads to improvements in both relighting metrics (PSNR, SSIM, LPIPS) and LiDAR simulation accuracy compared to state-of-the-art (e.g., UrbanIR, LiDARSim) (Chen et al., 23 Jul 2025).

The table below summarizes distinctive aspects:

Method	Geometry Rep.	Illumination Model	Relighting Support	Unique Features
UrbanIR	Hash-grid Neural Vol.	Parametric Sun/Sky	Yes	Volumetric shadow field, semantic losses
FEGR	Neural SDF + Mesh	HDR Sky Dome, MC Sampling	Yes	Hybrid neural/mesh, PBR
InvRGB+L	3D Gaussian Primitives	Sun/Sky SH, LiDAR BRDF	Yes	Joint RGB/LiDAR shading, dynamic scene graph
DeRenderNet	2D CNN, single-image	Learned latent code	Partial (intrinsics)	Shape-(in)dependent separation
Holistic Facade	SDF, Diffuse/Specular	Hosek–Wilkie analytic	Yes	Material/semantic adaptivity, façade focus

6. Evaluation, Benchmarks, and Limitations

Metrics include:

• Novel-view synthesis: PSNR, SSIM, LPIPS for photorealism.
• Intrinsic decomposition: scale-invariant PSNR/SSIM, LMSE, and LPIPS for albedo/shading, WHDR on IIW for out-of-domain.
• Geometry: mean and median angular error for normal estimation.
• Material: MSE for roughness, metallic parameters.

On LightCity, mixed multi-illumination training and shaded regularization improve both intrinsic decomposition and 3D inverse rendering (Wang et al., 1 Feb 2026). Noted limitations across methods include:

• Residual shadowing in albedo or incomplete separation of indirect light, especially in single-image approaches (Zhu et al., 2021);
• Leakage of errors from monocular priors (normal/shadow removal networks) into 3D pipelines (Lin et al., 2023);
• Performance degradation in geometry and material estimation under extreme or previously unseen illuminations;
• Lack of robust modeling for transient/dynamic content and complex weather phenomena.

7. Prospects and Future Directions

Research trends point to:

• Integration of generative illumination priors and domain-adaptive pipelines to enhance generalization and albedo consistency across arbitrary sky maps (Wang et al., 1 Feb 2026);
• Incorporation of physically-based multi-bounce and view-dependent BRDFs, and joint estimation of camera pose/illumination for greater realism (Lin et al., 2023, Wang et al., 2023);
• Richer datasets including weather, dynamic objects, and diverse material categories to benchmark new methods (Wang et al., 1 Feb 2026);
• Hybrid models leveraging advantages of neural fields, explicit meshes, and probabilistic (e.g., Gaussian) representations for large-scale city reconstruction (Chen et al., 23 Jul 2025);
• Active sensing and semantic adaptivity to recover thin structures and instance-consistent materials in complex cityscapes (Xie et al., 2023).

The field is moving toward scalable, multimodal, and physically grounded urban inverse rendering that supports high-fidelity relighting, robust AR/AV simulation, and semantically editable digital twins.