Mobile Scene Modeling Paradigm

• Mobile scene modeling is a paradigm that enables on-device, high-fidelity 3D rendering by co-designing geometry priors and differentiable operations while meeting strict constraints in training time and memory.
• It employs techniques like information-gated subsampling, GPU-native bundle adjustment, and single-reference MVS to extract robust geometric cues from sparse mobile inputs.
• Advancements such as prior-conditioned Gaussian parameterization and hardware-aligned differentiable splatting ensure competitive perceptual quality on diverse datasets within mobile resource limits.

Mobile scene modeling refers to the paradigm enabling on-device, high-fidelity 3D scene modeling and rendering by leveraging mobile hardware subject to tight resource constraints in terms of memory and training time. This paradigm addresses fundamental conflicts in adapting state-of-the-art scene modeling frameworks—such as 3D Gaussian Splatting (3DGS)—which were developed for unconstrained workstation environments, to the severely limited computational budgets of mobile devices. PocketGS exemplifies this paradigm, providing a complete on-device pipeline for 3DGS that delivers competitive or superior perceptual quality to high-end baselines while meeting strict requirements (minute-scale training time, <3 GB memory usage), by co-designing geometric, statistical, and hardware-aligned operators (Guo et al., 24 Jan 2026).

1. Paradigm Objectives and Constraints

The mobile scene modeling paradigm is defined by the need to satisfy three central, tightly coupled constraints simultaneously:

• Training time: Must complete in minute-scale durations (∼4 minutes, 500 iterations).
• Peak memory: Entire pipeline must execute within strict budgets (e.g., <3 GB on an iPhone 15, A16).
• Perceptual Quality: Visual modeling fidelity must match or exceed workstation 3DGS baselines.

These constraints introduce three fundamental contradictions:

1. The difficulty in acquiring sufficient geometric information from sparse, low-power mobile inputs.
2. The need for model parameterizations that guarantee rapid convergence from minimal data while avoiding early conditioning gaps.
3. The tension between memory-efficient differentiation (essential for mobile) and the computational patterns expected by hardware backends.

PocketGS resolves these contradictions by co-designing scene modeling, parameter initialization, and differentiable rendering in an integrated pipeline.

2. Geometry-Prior Construction (Operator 𝒢)

The geometry prior stage is engineered to maximize geometric signal quality from minimal input, reliably extracting pose and dense surface priors from mobile image sequences.

• Information-Gated Frame Subsampling: Frames are selected for processing using displacement and sharpness gates. Displacement gating accepts a frame only if the camera translation between candidate and previously accepted frames exceeds a threshold ($\tau_d=0.05$ m). Sharpness gating evaluates $S$—the mean absolute difference of intensities along x and y axes over a sparse pixel grid $\Omega$—to ensure new frames yield increased salience via $S_{\text{new}} > (1+r) S_{\text{best}}, r=0.05$.
• GPU-Native Global Bundle Adjustment: Poses $\{\mathbf T_i\}$ and sparse points $\{\mathbf P_j\}$ are jointly optimized via robust reprojection loss using Huber penalty, with the normal equations

$$\left(H_{cc} - H_{cp} H_{pp}^{-1} H_{pc}\right)\Delta_c = b_c - H_{cp} H_{pp}^{-1} b_p$$

solved efficiently in parallel on the GPU, inverting only block-diagonal submatrices.

• Single-Reference Cost-Volume MVS: Depth range quantiles guide plane-sweep MVS from a reference view maximizing an exposure and focus metric, using Census-based cost volumes and SGM aggregation. Pixels with confidence ≥0.4 are fused into a dense point cloud prior $P$.

Output: refined camera poses $\{T_t\}$ and a dense point cloud $P = \{\mathbf p_i\}$, which serve as the geometric foundation for subsequent model initialization.

3. Prior-Conditioned Gaussian Parameterization (Operator ℐ)

To expedite convergence and mitigate degenerate initializations, the prior-conditioned parameterization injects local surface statistics into the initialization of the scene representation.

• Local Surface Statistics: For each point $\mathbf p_i \in P$, principal component analysis (PCA) over its $K=16$ nearest neighbors yields a local covariance matrix $C_i$ and surface normal $\mathbf n_i$ (smallest eigenvector).
• Anisotropic Covariance Initialization: Tangential scale $s_t$ is computed as the mean distance to the three nearest neighbors; normal scale $s_n=r_\text{normal} s_t$, with $r_\text{normal}=0.3$. The Gaussian covariance is then set as

$$\Sigma_i = R_i \begin{pmatrix} s_t^2 & 0 & 0 \ 0 & s_t^2 & 0 \ 0 & 0 & s_n^2 \end{pmatrix} R_i^T$$

where $R_i$ aligns local axes with $\mathbf n_i$.

• Primitive Parameter Initialization: Each Gaussian primitive is assigned mean $\mu_i=\mathbf p_i$, color $c_i$ (from nearest view), opacity $\alpha_i$ (logit, initialized to 0.1), with $\log s$ parameterized in log-space.

The resulting set $$\Theta = \{ \theta_i = (\mu_i, \Sigma_i, c_i, \alpha_i) \}$$ forms the starting point for differentiable splatting optimization.

4. Hardware-Aligned Differentiable Splatting (Operator 𝒯)

Rendering and optimization are adapted to minimize both memory footprint and computational divergence on mobile GPUs.

• Forward Pass: Unrolled alpha compositing processes visible Gaussians sorted by depth at each pixel, producing the output color per view by iteratively applying

$$C^{(\ell)}_{\text{out}} = C^{(\ell)}_{\text{in}}(1-\alpha_\ell) + \alpha_\ell c_\ell$$

while caching only $C^{(\ell)}_{\text{in}}, \alpha_\ell$ per pixel, with a per-pixel counter, thereby reducing memory bandwidth.

• Backward Pass: Analytic gradients are applied to compositing operations, requiring only a minimal set of cached intermediates. Gradients for each Gaussian parameter are accumulated using an index-mapped scatter mechanism, eliminating CPU-GPU synchronization and in-place reordering.
• On-GPU Optimization: Adam optimizer moment buffers $m, v$ are maintained in GPU memory and updated in a single fused compute pass. The parameterization uses log-space for scale, logit-space for opacity, and tangent-space for rotations, enabling stable convergence in low-precision FP16 arithmetic.

5. Objective Functions and Regularization

The primary training objective is the per-pixel photometric L2 loss over $T$ views,

$$L_{\text{photo}}(\Theta) = \sum_{t=1}^T \sum_{u,v} \left\| C_{\text{final}}^t(u,v;\Theta) - I_t(u,v) \right\|_2^2$$

with additional weak regularization terms on $\log s$ (encouraging scale smoothness) and $\alpha$ (opacity sparsity),

$$L(\Theta) = L_{\text{photo}}(\Theta) + \lambda_s \sum_i \| \log s_i \|^2 + \lambda_\alpha \sum_i | \alpha_i |$$

to prevent degenerate solutions.

6. Experimental Evaluation and Benchmarks

PocketGS was evaluated on LLFF (forward-facing, real), NeRF-Synthetic, and MobileScan (iPhone 15, realistic noise/motion blur) datasets. For all datasets, the following budgets and setups were enforced: 500 iterations, tile resolution matched across methods, processing on iPhone 15 (A16) with Swift + Metal, and peak memory under 3 GB.

Performance Table (Average over datasets):

Dataset	Method	PSNR ↑	SSIM ↑	LPIPS ↓	Time ↓ (s)	Primitive Count
LLFF	3DGS-SFM-WK	21.01	0.641	0.405	108.0	18k
	3DGS-MVS-WK	19.53	0.637	0.387	313.1	40k
	PocketGS	23.54	0.791	0.222	105.4	33k
NeRF-Synthetic	3DGS-SFM-WK	21.75	0.800	0.243	83.7	12k
	3DGS-MVS-WK	24.47	0.887	0.128	532.1	50k
	PocketGS	24.32	0.858	0.144	101.4	47k
MobileScan	3DGS-SFM-WK	21.16	0.687	0.398	112.8	23k
	3DGS-MVS-WK	20.85	0.781	0.281	534.5	165k
	PocketGS	23.67	0.791	0.225	255.2	168k

On MobileScan, geometry prior memory footprint averages 1.53 GB (range: 1.19–2.22 GB) and full 3DGS training 2.21 GB (1.82–2.65 GB). Real-time rendering on iPhone 15 consistently achieves sharp perceptual quality.

7. Ablation Studies and Operator Contributions

Ablations were conducted on the MobileScan dataset to isolate the contribution of each operator:

Variant	PSNR ↑	SSIM ↑	LPIPS ↓	Time ↓ (s)
Full PocketGS	23.67	0.791	0.225	255.2
w/o ℐ (isotropic init)	22.49	0.7696	0.253	319.5
w/o global BA (𝒢)	23.45	0.7517	0.232	251.1
w/o MVS (𝒢)	21.07	0.6461	0.414	124.8

Ablation results confirm that:

• Operator ℐ accelerates convergence and improves PSNR (+1.18 dB).
• Global BA boosts pose consistency (SSIM increase of +0.04).
• Omission of dense MVS reduces overall perceptual quality (LPIPS increase of +0.19).

This suggests the paradigm's advantage arises from the interplay of geometry priors, prior-conditioned anisotropic initialization, and memory-optimal differentiable renderers adapted for mobile platforms.

---

The mobile scene modeling paradigm, as instantiated by PocketGS, systematically combines information gating, geometric optimization, prior-based surface parameterization, and hardware-aligned differentiable processing to achieve efficient, high-fidelity, and practical scene capture-to-rendering workflows entirely on resource-constrained mobile devices (Guo et al., 24 Jan 2026).