View-Dependent Gaussian Splatting

• View-dependent Gaussian splatting is a method that models 3D scenes using point-based Gaussians whose appearance changes with the viewing direction.
• Recent approaches integrate neural residuals, anisotropic spherical harmonics, and latent encoding to capture non-Lambertian effects like specular highlights and reflections.
• The technique efficiently supports applications in high-frequency rendering, segmentation, and uncertainty estimation while addressing limitations of fixed spherical harmonics models.

View-dependent Gaussian splatting refers to a class of techniques for novel view synthesis, 3D reconstruction, and scene representation in which the radiance, opacity, semantics, or other attributes associated with point-based 3D Gaussians explicitly depend on the viewing direction or related camera parameters. This methodology addresses the core limitation of classic 3D Gaussian Splatting (3DGS), whose use of low-order spherical harmonics or isotropic per-Gaussian color models is fundamentally ill-suited for reproducing high-frequency, non-Lambertian, or semantically ambiguous effects that vary with viewpoint—such as specular highlights, reflections, or view-conditional semantics. Recent work has systematically augmented the 3DGS paradigm through hybrid neural-residual models, anisotropic spherical Gaussian (ASG) mixtures, view-dependent opacity, neural-texture fields, and explicit camera-aware biasing, each targeting distinct classes of data (e.g., RGB, infrared, semantic) and application domains (rendering, segmentation, uncertainty estimation).

1. Foundations of 3D Gaussian Splatting and Its View-Dependence

The traditional 3DGS framework represents a scene as a set of $N$ anisotropic 3D Gaussians $\{\mathcal{G}_i\}_{i=1}^N$, each parameterized by spatial mean $\mu_i\in\mathbb{R}^3$, covariance $\Sigma_i$, opacity $o_i$, and a compact view-dependent color model (typically spherical harmonics coefficients $h_i$). For a view $\theta$ and pixel $(u,v)$, the color is computed by projecting each 3D Gaussian to a 2D image-plane Gaussian and compositing colors front-to-back:

$$C_{\text{base}}(u, v, \theta) = \sum_{i=1}^K w_i \cdot \Psi_l(h_i, v_i)$$

with $w_i = \alpha_i \cdot T_i$, $\alpha_i = o_i \cdot G^{2D}_i(p)$, $T_i = \prod_{j<i}\left(1-\alpha_j\right)$, and $\Psi_l$ the SH evaluator at view direction $v_i$ (Nguyen et al., 18 Nov 2025).

This scheme is lightweight and efficient but is strictly limited in its capacity to recover spatially varying, high-frequency, or view-conditional effects, due to the low degree ($\leq3$) and single coefficient set per Gaussian.

2. Explicit View-Dependence: Residual, Neural, and Harmonic Extensions

Recent advances address the limitations of fixed SH expansion by introducing parameterizations or predictors that model view-dependent appearance per Gaussian:

• IBGS (Image-Based Gaussian Splatting): IBGS adds a learned per-pixel residual $\Delta C(u,v,\theta)$ atop the standard 3DGS output, inferred from multi-view photographic evidence surrounding the target viewpoint. For a target ray, IBGS identifies proximal ("median") Gaussians, reprojects their closest points onto $M$ source images, samples the corresponding pixel values, and constructs per-view color and camera-difference features. These are passed through a multi-view feature encoder and CNN decoder to predict $\Delta C$. The final pixel color is:

$$C^{\text{final}}(u, v, \theta) = C_{\text{base}}(u,v,\theta) + \Delta C(u,v,\theta)$$

This allows the network to inject high-frequency, view-dependent detail inaccessible to the base SH representation, while storage remains limited to the original images (Nguyen et al., 18 Nov 2025).

• Neural/Latent Encoding: Multiple works replace or supplement SH coefficients with learned neural descriptors. Latent-SpecGaussian parameterizes each Gaussian by a 16-dimensional vector $z_i = [f_{d,i} \oplus f_{s,i}]$, with diffuse and specular features decoded separately by a Diffuse-UNet and a Specular-CNN, incorporating the view direction as a conditional input. Blending is achieved via a learned per-pixel, view-dependent mask $m(p)$, providing explicit control over diffuse/specular contribution (Wang et al., 2024).
• Global Neural Texture Fields: Neural Texture Splatting (NTS) utilizes a global triplane feature field and a neural decoder conditioned on spatial location, view, and (optionally) time, to generate per-Gaussian texture planes encoding high-frequency, view- and time-dependent variations in color and opacity (Wang et al., 24 Nov 2025).
• Directional Harmonics and Spherical Gaussians: View-dependent color and uncertainty can be efficiently modeled using spherical harmonics expansions up to a moderate order ($L=2$ or $3$), or, as shown in SG-Splatting and Spec-Gaussian, by mixtures of Spherical Gaussians (SG) or Anisotropic Spherical Gaussians (ASG). The latter enables highly localized, ellipsoidal lobes on the unit sphere, capturing both sharp and anisotropic highlights (Yang et al., 2024, Wang et al., 2024).

3. Rendering, Optimization, and Hierarchical Algorithms

View-dependent Gaussian splatting maintains compatibility with the efficient, sortable rasterization pipeline intrinsic to 3DGS. Each Gaussian is projected from 3D to 2D (incorporating any view-dependent deformation, as in Veta-GS for thermal images (Nam et al., 25 May 2025)), then alpha-composited according to depth. The compositing order can be enhanced for view-consistency by enforcing a per-pixel or tiled sort by true depth, as shown in “StopThePop,” avoiding blending and popping artifacts that could otherwise undermine view-dependent effects (Radl et al., 2024).

Training objectives include hybrid pixelwise and perceptual losses for color reconstruction, multi-view color warping for source image alignment, geometric consistency via normal maps or depth regularization, and specialized semantic or uncertainty calibration losses when modeling downstream tasks (Nguyen et al., 18 Nov 2025, Han et al., 10 Apr 2025).

4. Opacity, Uncertainty, and Semantic View-Dependence

Explicit modeling of view-dependent opacity and uncertainty expands the expressive power and applicability of Gaussian splatting:

• View-Opacity-Dependent Splatting (VoD-3DGS): Each Gaussian's opacity is modulated by a learned symmetric $3\times3$ matrix $\hat{S}_i$, enabling suppression or enhancement of Gaussian contributions along certain viewing directions:

$$\hat{\alpha}_i(\omega) = \sigma\left(\gamma_i + \omega^\top \hat{S}_i\, \omega \right)$$

This mechanism supports realistic reproduction of specular highlights and reflections (Nowak et al., 29 Jan 2025).

• View-Dependent Uncertainty: By treating uncertainty $u_i(\omega)$ as a spherical harmonic function of view direction, it is possible to expose which regions or directions are under-constrained by data, which is critical for downstream tasks such as next-best-view planning or reliable asset extraction (Han et al., 10 Apr 2025).
• View-Dependent Semantics: In open-vocabulary or language-grounded 3D applications, both CaRF and LaGa introduce explicit camera-aware or cross-view mechanisms to resolve the ambiguity and inconsistency of projected 2D semantic features. CaRF injects a camera encoding into each text interaction with a Gaussian and enforces cross-view consistency through paired-view loss (Tao et al., 6 Nov 2025). LaGa constructs multi-view, per-object cluster descriptors to resolve semantic variability and reweighting (Cen et al., 30 May 2025).

5. Comparison of Approaches and Empirical Performance

Table: Representative Approaches for View-Dependent Gaussian Splatting

Method	Key Contribution	View-Dependence Modeling
IBGS (Nguyen et al., 18 Nov 2025)	Per-pixel residual from images + base SH	CNN-predicted from multi-view warps
Spec-Gaussian (Yang et al., 2024)	ASG mixture per splat	Anisotropic SG lobes + MLP
NTS (Wang et al., 24 Nov 2025)	Neural field, tri-plane, local decoder	Neural RGBA/textures via triplane
VoD-3DGS (Nowak et al., 29 Jan 2025)	View-dependent opacity	Quadratic form in view vector
SG-Splatting (Wang et al., 2024)	Spherical Gaussian + SH	SG mixture, orthogonal axes
Latent-SpecGS (Wang et al., 2024)	Neural latent w/ parallel decoders	Parallel CNNs + MLP blending

Empirically, methods that leverage learned or neural view-dependent modeling (IBGS, NTS, Spec-Gaussian, Latent-SpecGS) achieve significant gains in PSNR (up to $+5.2$ dB in hard specular scenes for IBGS) and improved perceptual metrics over plain SH-based 3DGS. Memory and compute overheads are mitigated by compact neural fields or by sharing representation across Gaussians; for example, SG-Splatting achieves $35$–$50\%$ memory reduction with minimal quality drop (Wang et al., 2024).

6. Limitations, Open Problems, and Future Directions

Limitations include sensitivity to input-sampling density for methods reliant on image warps (IBGS), increased runtime and memory from neural decoders (NTS, Latent-SpecGS), possible training or inference complexity from high-order harmonic or SG mixtures (Spec-Gaussian, SG-Splatting), and limited effectiveness on purely diffuse or sparsely observed regions (VoD-3DGS, view-dependent uncertainty models).

Areas of active research and plausible future development encompass:

• Higher-Order Harmonics and Neural Directional Embeddings: For sharper or more discontinuous view effects, using higher-degree harmonics or neural alternatives is under exploration (Han et al., 10 Apr 2025, Gao et al., 11 Mar 2025).
• Explicit BRDF or Physically Based Parameterization: Integrating explicit physical reflectance models, potentially in combination with neural texture mechanisms, to handle multi-bounce or extreme illumination (Gao et al., 11 Mar 2025).
• Unified Dynamic and View-Dependent Models: Extending “7DGS” paradigms to jointly address spatial, temporal, and angular factors in a single, efficiently sliceable representation for real-time dynamic scene rendering (Gao et al., 11 Mar 2025).
• Semantic and Uncertainty-Aware Applications: Employing view-dependent splatting in open-vocabulary language grounding, segmentation, and downstream robotics or AR/VR understanding (Tao et al., 6 Nov 2025, Cen et al., 30 May 2025).
• Consistency and Rendering Artifacts: Hierarchical, per-pixel scene sorting to prevent blending artifacts ("popping") and support aggressive scene compression without sacrificing view-consistency or high-frequency effects (Radl et al., 2024).

View-dependent Gaussian splatting thus constitutes a rapidly evolving framework for photorealistic, semantically rich, and computationally efficient 3D scene modeling and rendering, with a diverse array of technical implementations addressing different facets of the view-dependence problem.