ONNX-Gaussian Generator for Real-Time 3DGS

• ONNX-based Gaussian Generator is a neural module in ONNX format that produces 3D Gaussian primitives for rendering applications.
• It integrates dynamic inference with GPU-accelerated pipelines, enabling real-time 3D Gaussian Splatting and eliminating CPU bottlenecks.
• The system leverages strict I/O contracts and efficient pre-processing to achieve sub-millisecond frame times in advanced WebGPU rendering.

An ONNX-based Gaussian Generator is a standardized neural network module, exported in Open Neural Network Exchange (ONNX) format, for producing frame-specific Gaussian primitives within a neural rendering pipeline. Deployed primarily in the context of real-time 3D Gaussian Splatting (3DGS), these generators integrate dynamic inference with GPU-accelerated rendering, eliminating legacy pipeline constraints and CPU bottlenecks. The ONNX-based Gaussian Generator is a core element in platforms such as Visionary, which unifies the inference and rendering of generative and reconstructive world models directly in the browser, leveraging WebGPU for high-throughput interactive synthesis and visualization (Gong et al., 9 Dec 2025).

1. Interface Schema and Data Contract

The ONNX-based Gaussian Generator adheres to a strict I/O schema per animation frame, ensuring compatibility and efficient interoperation with WebGPU-based rendering engines.

Inputs:

• camera_extrinsic: float32[4,4], representing the world-to-camera transformation matrix.
• camera_intrinsic: float324,4, encoding the camera’s projection parameters.
• Optional control variables: sequence index, timestamp, or latent vector, each as float32 scalar or 1D tensor.

Outputs:

• N (int32 scalar): specifies the number of Gaussian primitives produced per frame.
• gaussians: float16[N,13] packed tensor, each row parameterizing a 3D Gaussian with:
  • Mean: $\mu_x, \mu_y, \mu_z$ (3)
  • Covariance (upper-triangular): $$\Sigma_{xx}, \Sigma_{xy}, \Sigma_{xz}, \Sigma_{yy}, \Sigma_{yz}, \Sigma_{zz}$$ (6)
  • RGB color: $color_r, color_g, color_b$ (3)
  • Opacity: $\alpha$ (1)
• meta (optional): Dictionary with fields such as "packed_dtype" ("FP16" or "FP32") and upper bound for N.

The contract enforces that exactly N Gaussian tuples are output per frame. The use of a packed float16 layout minimizes upload bandwidth requirements; unpacking occurs on the GPU at runtime. Covariance is expressed in a compressed 6-value format corresponding to the independent elements of the symmetric $3 \times 3$ matrix.

2. Mathematical Foundations of Gaussian Splatting

Each rendered primitive $G_i$ is modeled as an anisotropic Gaussian in $\mathbb{R}^3$, parameterized by weight $w_i$, mean vector $\mu_i \in \mathbb{R}^3$, and covariance matrix $\Sigma_i \in \mathbb{R}^{3\times 3}$. Under camera projection $\Pi(\cdot)$, the 3D Gaussian maps to a 2D ellipse in image space:

• Projected mean: $x_i = \Pi(\mu_i)$
• Projected covariance: $S_i = J_i \Sigma_i J_i^{T}$, where $$J_i = \frac{\partial \Pi}{\partial x}\Big|_{x=\mu_i}$$

The influence on pixel $x \in \mathbb{R}^2$ is given by:

$$g_i(x) = \alpha_i \cdot \exp \left( -\frac{1}{2} (x - x_i)^T S_i^{-1} (x - x_i) \right)$$

with view-independent color $c_i \in [0,1]^3$. Final color compositing proceeds front-to-back:

$$C(x) = \sum_i \left[ g_i(x) \prod_{j < i} (1 - g_j(x)) \right] c_i$$

The generator network maps its outputs to these parameters: $\alpha_i \equiv w_i \in (0,1)$ (output via sigmoid); $\mu_i$ (3-vector head); $\Sigma_i$ (6-vector head, upper-triangular entries, positivity enforced via softplus); $c_i$ (3-vector head, postprocessed if using spherical harmonics coefficients).

3. ONNX Inference Pipeline and Runtime Integration

Model deployment and inference occur directly within WebGPU-enabled browser contexts. The pipeline comprises several phases:

• Model Loading: Performed once at application initialization using onnxruntime-web. The ONNX model is loaded, often with parameters quantized to FP16 and constants baked in at export (opset ≥ 14, constant folding enabled).
• Warm-Up: A dummy inference primes the WebGPU execution graph and cache, optimizing subsequent per-frame executions.
• Per-Frame Schedule:
  • Camera and optional control data are packed into WebGPU buffers.
  • Synchronous/asynchronous forward pass via ONNX runtime, with outputs streamed directly as GPU buffers (avoiding CPU roundtrips).
  • Gaussian buffer is bound as storage for GPU compute pre-processing: transform, cull, and radix-sort the Gaussian splats.
  • Instanced draw call issues one triangle-strip per splat; fragment shader evaluates $g_i(x)$ and performs compositing.

Reusing persistent WebGPU bind groups and ONNX session bindings enables predictable low-latency execution and amortized dispatch costs within the frame budget.

The following table summarizes the operational pipeline:

Step	Action	Technology
Model load	ONNX import, session instantiate	onnxruntime-web
Warm-up	Dummy inference to cache WebGPU execution graph	WebGPU
Per-frame inference	Input assembly, neural decoding, GPU buffer bind	WebGPU, ONNX
Preprocessing	Splat preprocess, cull, radix-sort (O(N))	WebGPU compute
Rendering	Instanced draw, fragment compositing	WebGPU fragment

4. Network Architectures and Training Paradigms

Several network designs are supported:

• MLP-based 3DGS: Follows Scaffold-GS style. Input is per-anchor feature ($f_i$, dimension 256–512) concatenated with view direction ($d_{view}$, dimension 3). 4–6 fully connected layers with hidden units 256–512 (GELU or ReLU activations). Four output heads: $H_{\mu}$ (3 floats), $H_{\Sigma}$ (6 floats via softplus), $H_c$ (3 floats, either direct RGB or SH coefficients), and $H_\alpha$ (1 float via sigmoid). Trained via L2 or L1 pixel loss plus regularization on $\Sigma$ scale. ONNX export uses dynamic axes for batch/view dimensions.
• 4D Gaussian Splatting: Temporal input (timestamp $t$) with canonical parameters and multi-scale feature planes. Feature lookup (bilinear sampling) is concatenated and fed to a small MLP (2–3 layers, 128 units). Outputs are delta mean (3), delta quaternion for rotation (4), and delta scale (3). The covariance updates as $R \Sigma_{can} R^T + \operatorname{diag}(\Delta s)$.
• Animatable Avatars (e.g., LHM, R3-Avatar): Input is SMPL-X pose $\theta$ (72D), shape $\beta$ (10D), and optional frame index. Internally, canonical $\mu$, $\Sigma$, and per-Gaussian skinning weights $W_i$ are used. Forward kinematics and LBS are included within the ONNX graph. Output is the deformed $\mu_i$ and $\Sigma_i$ in the current observation frame.

Training commonly employs mixed-precision (fp16) weights, opset ≥ 14, and constant folding. Large Concats/Splits are refactored in export to conform with WebGPU operational constraints.

5. Browser-Based Integration and Application Example

Integration is facilitated by a concise TypeScript API compatible with three.js and the “visionary-webgpu” library. The typical workflow involves initializing a WebGPU renderer and Gaussian renderer, loading the ONNX model, warm-up, and a main loop that, per frame, updates camera matrices, performs ONNX inference, streams results to GPU, and dispatches render calls.

import * as THREE from 'three'; import { InferenceSession, Tensor } from 'onnxruntime-web'; import { WebGPURenderer, GaussianRenderer } from 'visionary-webgpu'; async function initVisionary(canvas: HTMLCanvasElement) { const renderer = new WebGPURenderer(canvas, { useCompute: true }); const gaussRenderer = new GaussianRenderer(renderer); const session = await InferenceSession.create('/models/gauss_gen.onnx', { executionProviders: ['webgpu'] }); const dummyExtrinsic = new Tensor('float32', new Float32Array(16), [4,4]); const dummyIntrinsic = new Tensor('float32', new Float32Array(16), [4,4]); await session.run({ camera_extrinsic: dummyExtrinsic, camera_intrinsic: dummyIntrinsic }); function frame(time: number) { const cam = camera.matrixWorldInverse; const proj = camera.projectionMatrix; const extrinsicTensor = new Tensor('float32', cam.toArray(), [4,4]); const intrinsicTensor = new Tensor('float32', proj.toArray(), [4,4]); session.run({ camera_extrinsic: extrinsicTensor, camera_intrinsic: intrinsicTensor, // optional control... }).then((outputs) => { const gaussBuffer = outputs.gaussians.data as Uint16Array; const count = outputs.N.data[0] as number; gaussRenderer.updatePrimitiveBuffer(gaussBuffer, count); renderer.beginFrame(); gaussRenderer.draw(); renderer.endFrame(); }); requestAnimationFrame(frame); } requestAnimationFrame(frame); }

This workflow maintains all data and computation on the GPU, minimizing latency and eliminating CPU-GPU roundtrips.

6. Performance Characteristics and Throughput

The ONNX+WebGPU approach yields high throughput and low latency:

• Per-frame preprocessing—frustum culling, $\mu \rightarrow$clip, $\Sigma \rightarrow$ellipse axes—runs in a single WebGPU compute pass.
• Depth keys and indices are updated via GPU atomics, and a GPU radix sort organizes splats in $O(N)$ runtime, achieving sub-millisecond times for millions of points.
• Instanced triangle-strip draws and fragment-based Gaussian summation enable efficient compositing and blending.
• Captured and replayed ONNX execution graphs result in stable dispatch overhead, with reported per-frame decoding times:
  • Scaffold-GS: 2.5M splats $\rightarrow$ 9 ms
  • 4DGS: 0.06M splats $\rightarrow$ 8 ms
  • Avatars: 0.2M splats $\rightarrow$ 30 ms
• Aggregate end-to-end frame times for static 6M Gaussians are $\sim$2 ms, representing approximately 100$\times$ speedup over CPU-based WebGL sorting.
• The entire pipeline remains within a single-frame time budget, leveraging unified memory architectures for device-local execution and rapid updates (Gong et al., 9 Dec 2025).

A plausible implication is that the ONNX-based Gaussian Generator contract enables real-time, browser-native world model rendering and generative processing suitable for both reconstruction and synthetic content, with substantial practical advantages for reproducibility and deployment.