PocketGS: On-Device Training of 3D Gaussian Splatting for High Perceptual Modeling

Abstract: Efficient and high-fidelity 3D scene modeling is a long-standing pursuit in computer graphics. While recent 3D Gaussian Splatting (3DGS) methods achieve impressive real-time modeling performance, they rely on resource-unconstrained training assumptions that fail on mobile devices, which are limited by minute-scale training budgets and hardware-available peak-memory. We present PocketGS, a mobile scene modeling paradigm that enables on-device 3DGS training under these tightly coupled constraints while preserving high perceptual fidelity. Our method resolves the fundamental contradictions of standard 3DGS through three co-designed operators: G builds geometry-faithful point-cloud priors; I injects local surface statistics to seed anisotropic Gaussians, thereby reducing early conditioning gaps; and T unrolls alpha compositing with cached intermediates and index-mapped gradient scattering for stable mobile backpropagation. Collectively, these operators satisfy the competing requirements of training efficiency, memory compactness, and modeling fidelity. Extensive experiments demonstrate that PocketGS is able to outperform the powerful mainstream workstation 3DGS baseline to deliver high-quality reconstructions, enabling a fully on-device, practical capture-to-rendering workflow.

• The paper introduces PocketGS, a framework that enables on-device training of 3D Gaussian Splatting through geometry-prior construction, improved initialization, and hardware-conscious optimization.
• It employs techniques like information-gated frame subsampling, GPU-native global bundle adjustment, and single-reference cost-volume MVS to efficiently generate robust geometric priors and anisotropic Gaussian representations.
• Experimental results on mobile datasets demonstrate reduced runtime, lower memory usage, and superior perceptual quality compared to traditional workstation-level implementations.

On-Device Training of 3D Gaussian Splatting: The PocketGS Paradigm

Introduction and Motivation

The PocketGS framework presents a methodical approach for training 3D Gaussian Splatting (3DGS) models directly on mobile devices, precisely targeting the constraints that arise in such resource-limited environments: strict runtime ceilings ($<$5 min), memory budgets ($<$3 GB), and hardware limitations of tile-based mobile GPUs. Standard 3DGS methods operate under workstation-level resources, leading to contradictions when ported to mobile contexts—such as unreliable geometric priors, inefficient optimization due to naïve initialization, and prohibitive memory requirements for backpropagation during differentiable rendering.

PocketGS resolves these contradictions via three tightly coupled operators: geometry-prior construction ($\mathcal{G}$), prior-conditioned Gaussian parameterization ($\mathcal{I}$), and hardware-aligned differentiable splatting optimization ($\mathcal{T}$). This co-design enables perceptually high-fidelity scene modeling in realistic capture-to-reconstruction workflows executable fully on commodity smartphones.

Figure 1: Overview of the PocketGS framework, demonstrating the integration of geometry-prior construction ($\mathcal{G}$), prior-conditioned Gaussian parameterization ($\mathcal{I}$), and hardware-aligned splatting optimization ($\mathcal{T}$).

PocketGS System Architecture

Geometry Prior Construction ($\mathcal{G}$)

PocketGS initiates modeling via robust and compact geometry-prior construction, mitigating the noisy initialization endemic to mobile-captured input. This is achieved through:

• Information-Gated Frame Subsampling: Real-time selection of frames based on displacement, sharpness heuristics, and candidate windowing, ensuring only geometrically informative, high-fidelity frames are processed.
• GPU-Native Global Bundle Adjustment (BA): Joint pose and 3D point optimization using a robustified reprojection objective, solved efficiently on GPU using a Schur complement to exploit the sparse problem structure. This ensures global coherence and consistency under constrained memory.
• Single-Reference Cost-Volume MVS: Construction of a dense, geometry-faithful point cloud via memory-efficient depth inference, selecting references based on baseline and viewing angle and leveraging Census Transform plus Semi-Global Matching.

By combining these stages, $\mathcal{G}$ produces priors of sufficient density and accuracy to avoid iterative densification, consequently reducing computational and memory load while enhancing scene fidelity.

Prior-Conditioned Gaussian Parameterization ($\mathcal{I}$)

Transitioning away from isotropic Gaussian initialization, PocketGS leverages geometry priors to seed spatially balanced anisotropic Gaussians. For each point, local surface statistics are estimated via KNN, yielding normal vectors and scales. Gaussians are then seeded as disc-like ellipsoids, tangent to these estimated normals and scaled according to local point density. This initialization:

• Improves optimization conditioning
• Reduces the reliance on photometric optimization to discover structure
• Accelerates convergence within limited iteration budgets

Hardware-Aligned Differentiable Splatting ($\mathcal{T}$)

A critical innovation of PocketGS is its hardware-conscious reengineering of differentiable rendering for mobile GPUs. Key components include:

• Unrolled Alpha-Compositing with Forward Replay Cache: Manual unrolling of compositing, caching only essential intermediate values ($\{C_{in}, \alpha\}$), dramatically reducing bandwidth and memory pressure while supporting correct gradient flow.
• Index-Mapped Gradient Scattering: Decoupling the depth-sorted access for rendering from the canonical layout in memory, with gradients scattered appropriately via GPU index maps, preserving optimizer consistency without redundant data movement.
• Fully GPU-Resident Parameter Optimization: Adam optimizer states and parameter updates are maintained on-device, utilizing specialized representations (logit-space for opacity, log-space for scale, tangent-space projection for rotation) to ensure numerical stability and avoid CPU–GPU synchronization overhead.

These hardware-aligned refinements enable PocketGS to execute the entire training loop in strict mobile budgets, maintaining stability and correctness in the presence of tile-based deferred pipelines.

Experimental Analysis

PocketGS was evaluated on representative datasets: NeRF Synthetic (clean geometry), LLFF (challenging real-world captures), and a self-collected MobileScan dataset (mobile-native conditions). All models were trained end-to-end on an iPhone 15, with 500 optimization iterations for uniform comparison. Baselines include workstation-level 3DGS implementations seeded with either sparse SfM or dense MVS priors using COLMAP.

Figure 2: Qualitative comparison on the MobileScan dataset, revealing PocketGS's superior texture sharpness and structural fidelity relative to baselines.

PocketGS consistently outperforms baselines on perceptual quality and runtime efficiency:

• LLFF: PocketGS produces sharper textures and finer details, outperforming the dense prior baseline despite fewer Gaussians.
• NeRF Synthetic: PocketGS nearly matches the best metrics achieved by the highest-capacity baseline, with a $>5\times$ reduction in wall-clock runtime due to streamlined geometry and hardware alignment.
• MobileScan: PocketGS achieves a lower LPIPS (0.225) and approximately half the runtime (~255s) compared to the dense-MVS baseline (0.281 LPIPS, ~535s), evidencing stronger noise robustness and structure placement.

  Figure 3: Qualitative results on LLFF; PocketGS recovers thin structures and textures with high fidelity, closely matching ground truth.

  

  Figure 4: Qualitative results on NeRF Synthetic; PocketGS generates high-fidelity renderings consistent with baseline workstation outputs.

Mobile Deployment and Representation Efficiency

Figure 5: On-device real-time renderings on iPhone 15, showing consistent high-fidelity reconstruction and robust generalization across datasets and scene conditions.

PocketGS fulfills real-time deployment requirements by maintaining peak memory usage below 3 GB for all stages (median training: 2.21 GB, geometry prior: 1.53 GB), further validating practical feasibility on commodity smartphones.

Ablation and Component Contribution

Figure 6: Ablation of information-gated frame subsampling, highlighting its role in preserving detail fidelity.

Ablation studies confirm each PocketGS operator's necessity:

• Removing $\mathcal{I}$ (anisotropic initialization) substantially degrades perceptual metrics and slows convergence.
• Disabling global BA ($\mathcal{G}$) yields SSIM drops and undermines pose/structure coherence.
• Eliminating lightweight MVS causes the largest loss in perceptual quality, evidenced by significant declines in PSNR and SSIM, reinforcing the importance of dense, geometry-faithful priors.

Theoretical and Practical Implications

PocketGS demonstrates that with principled co-design encompassing geometric, initialization, and hardware-level constraints, high perceptual quality in 3D scene modeling is achievable on mobile platforms. This unlocks professional-grade content creation, enabling mixed reality and digital twin workflows with dramatically reduced time and energy costs. The framework suggests future directions in:

• End-to-end differentiable reconstruction coupling geometry and appearance learning
• Emergent mobile-native protocols for streaming and federated 3D content generation
• Extension to wider hardware classes via further kernel-level optimizations and resource-aware scheduling

Conclusion

PocketGS establishes a technically rigorous and practical benchmark for on-device training of 3D Gaussian Splatting, advancing both the theory and deployment of neural scene representations in resource-constrained environments. Its synergetic approach to prior construction, initialization, and differentiated rendering achieves workstation-comparable visual fidelity within rigid mobile hardware budgets, paving the way for democratization of high-resolution 3D scene capture and rendering in consumer devices.