StructGS: Adaptive Spherical Harmonics and Rendering Enhancements for Superior 3D Gaussian Splatting

Abstract: Recent advancements in 3D reconstruction coupled with neural rendering techniques have greatly improved the creation of photo-realistic 3D scenes, influencing both academic research and industry applications. The technique of 3D Gaussian Splatting and its variants incorporate the strengths of both primitive-based and volumetric representations, achieving superior rendering quality. While 3D Geometric Scattering (3DGS) and its variants have advanced the field of 3D representation, they fall short in capturing the stochastic properties of non-local structural information during the training process. Additionally, the initialisation of spherical functions in 3DGS-based methods often fails to engage higher-order terms in early training rounds, leading to unnecessary computational overhead as training progresses. Furthermore, current 3DGS-based approaches require training on higher resolution images to render higher resolution outputs, significantly increasing memory demands and prolonging training durations. We introduce StructGS, a framework that enhances 3D Gaussian Splatting (3DGS) for improved novel-view synthesis in 3D reconstruction. StructGS innovatively incorporates a patch-based SSIM loss, dynamic spherical harmonics initialisation and a Multi-scale Residual Network (MSRN) to address the above-mentioned limitations, respectively. Our framework significantly reduces computational redundancy, enhances detail capture and supports high-resolution rendering from low-resolution inputs. Experimentally, StructGS demonstrates superior performance over state-of-the-art (SOTA) models, achieving higher quality and more detailed renderings with fewer artifacts.

Overview of StructGS: Enhancements in 3D Gaussian Splatting

The paper "StructGS: Adaptive Spherical Harmonics and Rendering Enhancements for Superior 3D Gaussian Splatting" presents a comprehensive approach to overcoming the limitations observed in traditional 3D Gaussian Splatting (3DGS) methods, particularly in the realms of 3D reconstruction and novel view synthesis. The introduced framework, StructGS, innovatively advances the field by enhancing the capabilities of 3DGS through the integration of adaptive algorithms and improved rendering techniques.

Key Contributions

StructGS addresses three significant limitations present in current 3DGS models: inefficiencies in training, limited structural fidelity, and high computational expenses for high resolutions. The paper outlines the shortcomings of existing models in accounting for non-local structural information, initialisation biases in higher-order spherical harmonics, and the extensive resource demands for rendering high-resolution images. Here are the notable contributions of the paper:

1. Patch-Based SSIM Loss: The paper introduces a patch-based SSIM loss metric that enhances the performance of 3D reconstruction by capturing non-local structural similarities more effectively. This innovation is aimed at remedying the oversight of conventional SSIM metrics that focus on individual pixel comparison, thus improving the rendering accuracy and detail resolution.

2. Dynamic Spherical Harmonics Initialisation: A significant aspect of the work is the proposal of a dynamic strategy for the initialisation of spherical harmonics. By tailoring higher-order harmonics based on opacity and distance metrics, StructGS improves computational efficiencies and reduces biases, leading to more effective training with reduced iterations.

3. Integration of a Multi-scale Residual Network (MSRN): The framework further employs a pre-trained MSRN model during the rendering process. This integration allows high-resolution renderings from low-resolution inputs, thereby mitigating the otherwise high memory and computational demands associated with traditional high-resolution training practices.

Numerical Results and Implications

The empirical evaluation evidences the efficacy of the proposed StructGS framework. On datasets such as Mip-NeRF 360 and Tanks {content} Temples, the framework outperforms existing state-of-the-art models, such as Scaffold-GS and Mip-Splatting, which were based on the original 3DGS. StructGS achieves superior results while effectively minimising artifacts and computational overhead.

In terms of quantitative metrics, StructGS demonstrates significant improvements in PSNR, SSIM, and LPIPS scores, marking a notable advancement in the field of photorealistic 3D rendering and reconstruction. These results not only cement the practical efficacy of StructGS but also open pathways for its application in complex rendering tasks, where computational efficiency and rendering quality are paramount.

Future Directions

Despite its advancements, the paper acknowledges certain limitations, focusing primarily on computational demands during the SSIM metric application and reliance on initial estimates for spherical harmonics adjustments. Future research may explore extending the utilisation of structural similarity metrics to encompass non-RGB attributes as well as dynamic adjustments during the training process to enhance flexibility and adaptive capabilities. Furthermore, integrating MSRN into the training phase could yield more optimised model performances.

Overall, this paper positions StructGS as a compelling tool in the arsenal of 3D rendering techniques, providing foundational insights and methodologies that suggest potential advancements in future neural rendering and reconstruction frameworks.