Real-time Rendering with a Neural Irradiance Volume

Abstract: Rendering diffuse global illumination in real-time is often approximated by pre-computing and storing irradiance in a 3D grid of probes. As long as most of the scene remains static, probes approximate irradiance for all surfaces immersed in the irradiance volume, including novel dynamic objects. This approach, however, suffers from aliasing artifacts and high memory consumption. We propose Neural Irradiance Volume (NIV), a neural-based technique that allows accurate real-time rendering of diffuse global illumination via a compact pre-computed model, overcoming the limitations of traditional probe-based methods, such as the expensive memory footprint, aliasing artifacts, and scene-specific heuristics. The key insight is that neural compression creates an adaptive and amortized representation of irradiance, circumventing the cubic scaling of grid-based methods. Our superior memory-scaling improves quality by at least 10x at the same memory budget, and enables a straightforward representation of higher-dimensional irradiance fields, allowing rendering of time-varying or dynamic effects without requiring additional computation at runtime. Unlike other neural rendering techniques, our method works within strict real-time constraints, providing fast inference (around 1 ms per frame on consumer GPUs at full HD resolution), reduced memory usage (1-5 MB for medium-sized scenes), and only requires a G-buffer as input, without expensive ray tracing or denoising.

• The paper introduces a Neural Irradiance Volume (NIV) that pre-computes a 5D neural representation for efficient, noise-free indirect illumination.
• It employs multilevel hash grid encodings with a compact MLP to achieve 10x higher quality than traditional probe-based methods at low memory cost.
• The method generalizes to dynamic scenes without runtime ray tracing, enabling real-time interactive graphics and potential for inverse rendering.

Real-time Rendering with a Neural Irradiance Volume: Technical Overview

Introduction and Motivation

Indirect diffuse global illumination (GI) constitutes a substantial portion of perceived radiance in realistic scenes, yet its computation is prohibitively expensive for real-time applications. Traditional solutions employ probe-based irradiance volumes, baking the static scene’s indirect lighting into a regular grid structure that can be queried efficiently at runtime, including for dynamic objects inserted into the scene post-bake. However, these grids suffer from cubic memory scaling, poor adaptability to heterogeneous spatial frequency, and interpolation artifacts. More recent neural approaches cache radiative quantities at surfaces or online during progressive path tracing, but their inference expense and reliance on denoising or expensive runtime ray tracing has limited adoption under strict real-time constraints.

"Real-time Rendering with a Neural Irradiance Volume" (2602.12949) proposes Neural Irradiance Volume (NIV), a novel pre-computed, neural-based 5D representation of the indirect irradiance field that achieves high-quality, memory-efficient indirect illumination for both static and dynamic geometry, at real-time rates (≈1 ms per frame at full HD on consumer GPUs). NIV employs compact coordinate-based neural networks combined with multilevel hash grid encodings to compress the irradiance field without introducing scene-specific heuristics and avoids costly Monte Carlo integration or denoising at runtime.

Figure 1: In many scenes, indirect lighting contributes the majority of visible radiance. By caching indirect lighting using an irradiance volume---such as a neural irradiance volume---the significant cost of Monte Carlo integration at runtime can be avoided.

Model Architecture and Representation

The core of NIV is a four-layer fully connected MLP mapping a 5D input (3D position $x$ and 2D unit-length direction $n$) to a predicted irradiance value. High-frequency functional variations are modeled via a multilevel hash encoding [muller2022instant], which adaptively maps spatial inputs to compressed latent vectors, significantly improving efficiency by amortizing representational capacity over relevant volumes and culling inside-geometry samples during training. A fraction of training budget is targeted to scene surfaces with normals aligned to the corresponding directions, to synthesize the high spatial frequency content needed for accurate shading on both static and dynamic geometry (Figure 2).

Figure 2: Culling training data inside scene geometry and allocating a portion of the training budget to sampling scene surfaces both improve reconstruction quality. Combining these strategies produces the most robust results.

NIV learns pre-integrated, volumetric indirect irradiance, which allows noise-free and bias-free inference—contrasting with neural radiance field approaches that either require runtime sampling/denoising or are limited to static surfaces.

Comparison to Probe-based and Surface Neural Caches

Classical probe-based irradiance volumes require dense grids to capture high-frequency indirect lighting, but the cubic scaling in memory quickly becomes prohibitive. NIV achieves an order of magnitude higher reconstruction quality (as measured by MSE) for the same memory budget—e.g., 10x improvement at ∼5 MB, as proven on multiple complex scenes (Figure 3).

Figure 3: Horizontal slice through the banners of Sponza (5.4 MB budget). NIV captures irradiance bleed and shadows better than probe-based methods. NIV has $\sim$10x higher quality across the scene's volume. Ray tracing the visibility towards probes during evaluation (“+ RT”) reduces the overall error but adds a significant performance overhead.

Surface neural caches store outgoing radiance only for known static geometry and require deferred query/ray tracing to shade unseen dynamics, introducing either bias or variance, and requiring costly denoising. NIV, in contrast, enables direct, noise-free inference for dynamic objects with no runtime ray tracing, outperforming neural surface caches in both quality and efficiency under matched memory and runtime budgets (Figure 4, Figure 5, Figure 6).

Figure 4: Rendering with a neural surface cache requires a single deferred surface lookup, which introduces noise. A neural denoiser reduces noise but produces blotchiness; our method avoids path tracing and denoising while significantly improving quality compared to a reference image.

Figure 5: Runtime comparison between NIV, a denoised neural surface cache and denoised path tracing. The surface cache only traces a single bounce (4 spp) as opposed to path tracing (16 spp). Timing excludes rasterization and shows NIV as competitive for real-time budgets.

Figure 6: Rendering error of a neural surface cache and NIV compared against reference on Sponza. For the surface cache, error is dominated by variance from deferring; increasing the model capacity reduces bias but not overall error. The quality of NIV scales with allocated memory.

Training Methodology

NIV is trained by densely sampling position-direction pairs throughout the scene, with an explicit sampling strategy along surfaces to capture contact shadows and other high-frequency details. Samples with negligible indirect illumination (e.g., located within geometry) are pruned to maximize effective use of network parameters. Optimization is performed using relative L2 loss normalized by network prediction, which stabilizes convergence over irradiance values spanning large dynamic ranges. Convergence for complex scenes (e.g., Sponza) is achieved within 30 minutes on high-end GPUs, the majority of which is spent generating ground-truth samples with high-quality path tracing (Figure 7).

Figure 7: Visualization of the training procedure: position-direction pairs $(x, n)$ are uniformly sampled within the bounding-box of the scene, with direction sampled on the sphere. A fraction of points is sampled along surfaces. For each $(x,n)$ pair, reference indirect irradiance $E$ is estimated via path tracing and used to train the irradiance model.

Generalization and Applications: Variable Scenes and Higher-dimensional Irradiance Fields

NIV natively generalizes to arbitrary dynamic objects not seen at training, assuming their influence on the scene’s global illumination is minor—reflecting standard assumptions underlying probe-based approaches. Direct modeling of variable scene parameters (e.g., moving lights or occluders, time-of-day cycles) is achieved by adding additional inputs to the network; the compact encoding strategy allows real-time inference with up to 1-2 extra dimensions before compression quality suffers (Figure 8).

Figure 8: Adding the angle of a directional light to NIV lets us model time-of-day changes in Sponza without requiring further training at runtime. The Bunny is unseen during training but receives high-quality color bleed.

When compared to neural surface caches that explicitly encode variable scene elements, NIV achieves comparable or lower error at fixed memory for high-count variable parameter settings, without retraining or increasing runtime cost (Figure 9).

Figure 9: NIV compared with variable-scene neural methods. Unlike the compared methods, NIV is trained only once on the static scene and generalizes to arbitrary dynamic configurations with consistently low rendering error.

Limitations and Future Directions

While NIV delivers high-quality indirect diffuse illumination, it inherits the limitations of indirect irradiance volumes: moving dynamic geometry does not influence the bakened light field, higher-order interactions and glossy materials require ad hoc extensions (e.g., deferred queries, enhanced neural parameterization), and scaling beyond medium-sized scenes can become capacity-limited. Memory efficiency could further benefit from advances in compact neural primitive architectures and learning-based hash compression, though care must be taken to avoid training instabilities and reconstruction artifacts. Partitioning large scenes or integrating level-of-detail schemes presents promising future directions for scaling NIV to production-scale environments.

Practical and Theoretical Implications

NIV replaces artist-tuned probe grid heuristics with direct optimization via neural compression, facilitating integration into modern real-time engines through a G-buffer pipeline without disruptive runtime ray tracing or denoising. The representation is continuous, noise-free, and readily extensible to moderate numbers of variable scene parameters—beyond what regular probe grids or fully dynamic neural variable scene surface encodings can feasibly achieve for real-time applications. The differentiable nature of NIV additionally offers potential utility for inverse rendering, neural inverse graphics, and self-supervised relighting applications.

Figure 10: Overview of the indirect irradiance using NIV. Movable objects (Lucy, Bunny, Dragon) are unseen during training. NIV accurately immerses them in the precomputed light field, with dynamic ambient occlusion compensating local mutual occlusion effects.

Conclusion

Neural Irradiance Volume constitutes a significant advancement for real-time global illumination, achieving 10x improvement in memory-efficiency and offering a unified, noise-free solution for both static and dynamic geometry without runtime overhead of ray tracing or denoising. NIV’s direct loss-driven optimization, continuous representation, and scalable extension for moderate-dimensional variable scene parameters position it as a practical replacement for traditional irradiance volumes in game engines and interactive graphics, and an exploitable primitive for future differentiable, neural rendering and inverse graphics frameworks.