AGS-Mesh: Adaptive Gaussian Splatting and Meshing with Geometric Priors for Indoor Room Reconstruction Using Smartphones

Abstract: Geometric priors are often used to enhance 3D reconstruction. With many smartphones featuring low-resolution depth sensors and the prevalence of off-the-shelf monocular geometry estimators, incorporating geometric priors as regularization signals has become common in 3D vision tasks. However, the accuracy of depth estimates from mobile devices is typically poor for highly detailed geometry, and monocular estimators often suffer from poor multi-view consistency and precision. In this work, we propose an approach for joint surface depth and normal refinement of Gaussian Splatting methods for accurate 3D reconstruction of indoor scenes. We develop supervision strategies that adaptively filters low-quality depth and normal estimates by comparing the consistency of the priors during optimization. We mitigate regularization in regions where prior estimates have high uncertainty or ambiguities. Our filtering strategy and optimization design demonstrate significant improvements in both mesh estimation and novel-view synthesis for both 3D and 2D Gaussian Splatting-based methods on challenging indoor room datasets. Furthermore, we explore the use of alternative meshing strategies for finer geometry extraction. We develop a scale-aware meshing strategy inspired by TSDF and octree-based isosurface extraction, which recovers finer details from Gaussian models compared to other commonly used open-source meshing tools. Our code is released in https://xuqianren.github.io/ags_mesh_website/.

• The paper presents a dual-phase refinement process (DNC and ANR) that improves mesh fidelity using geometric priors.
• It leverages adaptive filtering and innovative meshing strategies, yielding superior results in Chamfer-L1, PSNR, and SSIM metrics.
• The method significantly enhances smartphone-captured indoor reconstructions, promising advances in AR and VR applications.

Adaptive Gaussian Splatting and Meshing with Geometric Priors for Indoor Room Reconstruction Using Smartphones

Introduction

This paper presents AGS-Mesh, a novel approach for improving 3D reconstruction of indoor scenes using low-resolution depth sensors commonly found in smartphones, such as iPhones. The method integrates geometric priors into Gaussian Splatting to refine mesh reconstruction, addressing the limitations of conventional depth maps for scenarios requiring dense detail capture. The approach leverages geometric consistency checks and adaptive regularization to mitigate the inaccuracies inherent in mobile device depth sensing.

Methodology

The core innovation of AGS-Mesh involves the combination of Gaussian Splatting techniques with geometric priors derived from low-resolution depth maps and pretrained monocular estimators. The algorithm distinguishes itself by employing a dual-phase refinement process: Depth Normal Consistency (DNC) and Adaptive Normal Regularization (ANR).

1. Depth Normal Consistency (DNC):

DNC evaluates the consistency of normal vectors derived from depth maps and monocular estimates. This comparison filters out imprecise depth values, improving the fidelity of reconstructed surfaces. Depth maps are first denoised using KNN-based normal estimation embedded in the optimization pipeline, enhancing the accuracy of subsequent geometric reconstructions.

2. Adaptive Normal Regularization (ANR):

ANR focuses on training with monocular normal estimates, reducing regularization impact where normal predictions deviate significantly. This adaptivity is achieved by comparing normals from Gaussian splats with monocular predictions, only reinforcing consistent, reliable estimates in the optimization loop.

3. Meshing Strategy:

The paper introduces a novel meshing approach using adaptive TSDF and IsoOctree algorithms. This method dynamically adjusts mesh detail based on scene characteristics, creating smoother and more geometrically accurate mesh outputs compared to traditional methods like Marching Cubes.

Figure 1: Pipeline Overview. Our approach leverages geometric consistency between normals derived from raw sensor depths and those predicted by a pretrained model to filter out noisy sensor depth data.

Experimental Results

AGS-Mesh demonstrates significant improvements in mesh detail and view synthesis quality across various datasets, including MuSHRoom. The introduction of adaptive filtering strategies in depth and normal evaluations substantiates the method's robustness in rendering realistic and geometrically precise indoor environments using consumer-grade devices.

Quantitative Benchmarks:

• Mesh Reconstruction: The proposed technique achieves superior performance in metrics like Chamfer-L1 and normal consistency, outperforming current state-of-the-art methods.
• View Synthesis: Evaluated through PSNR and SSIM metrics, AGS-Mesh exhibits remarkable gains, particularly when assessed on scenes captured at suboptimal angles or during challenging lighting conditions.

Figure 2: Qualitative comparison with our optimization strategy. IsoOctree mesh extraction can efficiently generate a smoother surface compared to the TSDF.

Implications and Future Work

The findings from AGS-Mesh emphasize the potential of integrating geometric priors with advanced rendering techniques to enhance 3D reconstruction capabilities using widely accessible mobile devices. The approach not only elevates mesh accuracy but also broadens the scope for implementing augmented reality (AR) and virtual reality (VR) applications on smartphones.

Future research may explore further optimization of regularization thresholds and extend the framework's applicability to outdoor or multi-environment scenarios, potentially benefiting navigation, simulation, and interactive media industries.

Conclusion

AGS-Mesh presents a significant advancement in the domain of indoor scene reconstruction using smartphone technology by integrating sophisticated geometric priors into Gaussian Splatting methods. Its dual-phase adaptive strategy effectively refines the meshing process, unlocking new possibilities for consumer-grade 3D reconstruction systems, thereby promising enhancements in accessibility and utility for immersive technological applications.

Figure 3: Qualitative mesh comparison for the "8b5caf3398" scene from ScanNet++ dataset.