NeuSG Framework: 3D Surface Reconstruction

• NeuSG Framework is a 3D surface reconstruction method that combines neural implicit SDFs with explicit 3D Gaussian splatting to recover highly detailed surfaces from multi-view images.
• The joint optimization leverages photometric, scale, and normal alignment losses to mutually refine both the SDF network and Gaussian parameters for enhanced accuracy.
• This approach overcomes the limitations of over-smoothed depth maps and sparse point clouds, delivering state-of-the-art performance with improved computational efficiency.

The NeuSG framework is a 3D surface reconstruction methodology that integrates neural implicit @@@@1@@@@ (SDF)-based surfaces with explicit 3D Gaussian Splatting, establishing a mutual optimization pipeline that enables the recovery of highly detailed surfaces from calibrated multi-view images. The framework exploits the dense geometric priors provided by Gaussian Splatting to regularize the implicit neural surface, while simultaneously leveraging the predicted surface normals of the implicit model to refine the explicit Gaussians. This approach enables NeuSG to overcome the limitations of over-smoothed depth maps and sparse point clouds in previous multi-view reconstruction methods, achieving state-of-the-art (SOTA) performance with improved computational efficiency (Chen et al., 2023).

1. NeuSG System Overview

NeuSG is designed to recover a complete, high-fidelity 3D surface from multi-view RGB images, combining the geometrical strength of neural implicit SDFs (NeuS) with the structural detail of 3D Gaussian Splatting. The core pipeline consists of three primary phases:

1. 3D Gaussian Splatting: Standard differentiable splatting is performed on the multi-view images, projecting them into a dense set of 3D Gaussians. Each Gaussian is parameterized by center $p_i$, covariance $\Sigma_i$, spherical-harmonic color coefficients $H_i$, and opacity $\alpha_i$.
2. Implicit SDF Network Construction: A neural implicit SDF $f(\mathbf{x};\theta)$, parameterized by a multi-layer perceptron (MLP), is erected to define the surface as the zero-level set, along with a radiance/color predictor $c(\mathbf{x}, \mathbf{d};\theta_n)$.
3. Joint Optimization: Parameters of both the SDF network and the Gaussians are refined together using a shared photometric loss and two new regularizers: a scale regularizer which flattens Gaussians onto the surface and a normal alignment term to align their principal axes with local surface normals.

By alternating or interleaving optimization steps for SDF parameters ($\theta$) and Gaussian parameters ($p_i, s_i, r_i, H_i, \alpha_i$), NeuSG establishes a bi-directional refinement: the splatted Gaussians feed dense structural guidance into the SDF network, while the SDF-derived normals improve Gaussian localization.

2. Mathematical Formulation

2.1 Neural Implicit Surface (NeuS)

A neural implicit surface is modeled as an SDF $f: \mathbb{R}^3 \to \mathbb{R}$, with the surface defined by $\{ \mathbf{x}\mid f(\mathbf{x})=0 \}$. The function $f(\mathbf{x};\theta)$ is represented by an MLP, potentially using hash encoding for efficiency. Volume rendering for view synthesis samples points $x_i = o + t_i d$ along each camera ray, with opacity computed as:

$$\Phi_s(u) = \frac{1}{1 + \exp(-u/s)}, \quad \alpha_i = \max\left( \frac{\Phi_s(f(x_i)) - \Phi_s(f(x_{i+1}))}{\Phi_s(f(x_i))}, 0 \right)$$

The synthesized color is:

$$\hat{C}(o,d) = \sum_{i=1}^N w_i c(x_i, d), \quad w_i = T_i \alpha_i, \quad T_i = \prod_{j=1}^{i-1} (1 - \alpha_j)$$

2.2 3D Gaussian Splatting

Each Gaussian is specified as:

• Center $p_i \in \mathbb{R}^3$
• Covariance $\Sigma_i = R_i S_i S_i^T R_i^T$, where scaling matrix $S_i = \operatorname{diag}(s_{i1}, s_{i2}, s_{i3})$, rotation $R_i$ (from quaternion $r_i$)
• Spherical harmonics color $H_i$, opacity $\alpha_i$

Differentiable splatting provides rendered color/opacity for each view, with rendering loss analogous to the SDF photometric loss.

2.3 Loss Function Components

Loss Term	Mathematical Formulation	Purpose
Photometric loss	$L_{RGB} = \sum_{rays} \| C - \hat{C}\|_1$	Color/radiance fidelity to ground-truth images
Eikonal regularizer	$$L_{eik} = \frac{1}{N}\sum_{i=1}^N ( \| \nabla f(x_i) \|_2 - 1 )^2$$	Gradient normalization of SDF
Point cloud prior	$L_{pt} = \sum_i | f(p_i) |_1$	Draws Gaussians’ centers to the true surface
Scale regularizer	$$L_{scale} = \sum_i | \min(s_{i1},s_{i2},s_{i3}) |_1$$	Flattens Gaussian ellipsoids on the surface
Normal alignment	$$L_{align} = \sum_i | 1 - | n_{w,i}^\top \nabla f(p_i) | |_1$$	Aligns thin axis of Gaussian with surface normal

The total optimization objective is:

$$L_{NeuSG} = L_{RGB} + \lambda_1 L_{eik} + \lambda_2 L_{pt} + \lambda_3 L_{scale} + \lambda_4 L_{align}$$

with recommended hyperparameters $\lambda_1=0.1$, $\lambda_2=1.0$, $\lambda_3=100$, $\lambda_4=1.0$ (Chen et al., 2023).

3. 3D Gaussian Splatting Guidance and Mutual Refinement

NeuSG’s innovation is in the iterative mutual refinement strategy:

• Gaussian Generation: Initial Gaussians are fit via standard differentiable 3D Splatting (Kerbl et al., 2023), producing a dense point cloud whose centers do not strictly coincide with the object’s surface.
• Flattening and Alignment: The scale regularizer ($L_{scale}$) pushes each ellipsoid to have a negligible axis, coercing its center to align with the SDF zero-crossing. Its principal (thin) axis, extracted as $$n_{c,i} = \mathrm{OneHot}(\arg\min (s_{i1}, s_{i2}, s_{i3}))$$, is rotated to world space ($n_{w,i} = R_i n_{c,i}$) and aligned to the SDF-derived normal via $L_{align}$.
• Parameter Updates: Gaussian parameters (scale, rotation) are updated by back-propagating $L_{scale}$ and $L_{align}$. Centers are moved through the photometric loss and $L_{pt}$. The color/shading coefficients remain under $L_{RGB}$ supervision.

This dual strategy ensures that the Gaussian set remains surface-faithful and that their implied normals denoise and enhance surface geometry, feeding back to the SDF for subsequent training rounds.

4. Training Protocols and Computational Details

• Preprocessing: Given $N$ calibrated RGB images with camera poses, the pipeline first generates the initial Gaussian set and randomly initializes the SDF+color MLPs (hash encoding optional).
• Optimization Schedule: Uses Adam optimizer ($\text{lr}=10^{-3}$ for both MLP and Gaussian parameters). Training comprises 500k SDF steps with, every 100k, a 30k-step Gaussian-centric mini-optimization focused on $L_{scale}+L_{align}$ regularization. Each iteration samples 1024 rays.
• Hardware: Employs a single NVIDIA RTX 4090 GPU, requiring approximately 16 hours for end-to-end optimization (significantly more efficient than prior SOTA approaches).
• This regimen allows SDF and Gaussian parameters to be updated under the shared, multi-component $L_{NeuSG}$ objective, realizing the mutual guidance principle.

5. Experimental Validation and Ablation Analysis

Quantitative Benchmarks

NeuSG was evaluated on the Tanks & Temples benchmark, providing F1-scores:

Method	Mean F1 Score	Training Time	Hash Size
NeuS (baseline)	0.38	-	-
Geo-Neus / MonoSDF / RegSDF	0.35–0.39	-	-
NeuralAngelo	0.50	128 hours	$2^{22}$
NeuSG	0.49	16 hours	$2^{19}$

NeuSG matches or exceeds SOTA while using $\sim 8\times$ fewer hash-encoding parameters and $\sim 8\times$ less training time (Chen et al., 2023).

Qualitative Assessments

NeuSG reconstructions exhibit substantially fewer geometric holes (e.g., in roofs and freestanding objects) and recover fine structural detail (panel seams, railings) that prior approaches routinely miss.

Ablation Results

Ablative experiments demonstrate:

• Raw Gaussian point cloud guidance alone can lower performance due to noise ($F1=0.59$).
• Introducing only the scale regularizer yields $F1=0.69$.
• Full scale and normal alignment brings mean $F1$ to $0.73$, with regularizers converging rapidly ($L_{scale} \to 10^{-8}$, $L_{align} \to 10^{-3}$).

6. Contributions, Limitations, and Future Directions

Key Contributions

1. Mutual Optimization: A bi-directional training loop where neural implicit surfaces and explicit 3D Gaussian splatting refine each other.
2. Scale Regularization: Enforces thin ellipsoids, automating the process of aligning Gaussians with the true surface.
3. Normal Alignment: Refines Gaussian orientations using SDF-derived normals, denoising guidance to the SDF.
4. Computational Efficiency: Demonstrates robust, SOTA-quality reconstruction on large scenes within practical compute budgets.

Limitations

• The reliance on dense, multi-view coverage renders the method less effective under sparse or single-view setups.
• Though substantially more efficient than comparators, NeuSG still requires approximately 16 hours of GPU compute per scene.
• In under-constrained regions with limited photometric or geometric information, both Gaussians and the SDF may drift, potentially producing artifacts.

This suggests avenues for future research could include reducing data requirements, accelerating convergence, or enabling real-time updates.

(Chen et al., 2023)