FlowSSC: Generative Refinement for Monocular SSC

• FlowSSC is a generative refinement framework for monocular Semantic Scene Completion that infers dense 3D voxel grids, including occluded regions.
• It integrates feed-forward SSC backbones with Shortcut Flow-matching to perform one-step latent diffusion in a highly compressed triplane latent space.
• FlowSSC achieves state-of-the-art performance on SemanticKITTI with improved IoU and mIoU scores while enabling real-time inference for autonomous systems.

FlowSSC is a generative refinement framework for monocular Semantic Scene Completion (SSC), addressing the challenge of inferring dense 3D semantic voxel grids, including both visible and fully occluded regions, from a single RGB image. It formulates SSC as a conditional generation task, integrating with existing feed-forward SSC backbones to enhance high-fidelity reasoning over occluded 3D structure. The core innovation is Shortcut Flow-matching, a mechanism enabling high-quality single-step latent diffusion in a highly compressed triplane latent space. FlowSSC achieves state-of-the-art performance and real-time inference, marking it as the first generative method directly applicable to monocular SSC (Xi et al., 21 Jan 2026).

1. Problem Formulation and Motivation

The monocular SSC task is defined as predicting a dense 3D voxel grid $X \in \{voxel_1, \ldots, voxel_N\}$, where each voxel holds occupancy information and a semantic label, from a single input image $I$. The central challenge is the "one-to-many" ambiguity: given occlusions in RGB images, many plausible 3D scene completions exist behind visible surfaces. Existing feed-forward architectures (such as MonoScene, VoxFormer, OccFormer, ET-Former) minimize per-voxel regression loss but collapse high-frequency occluded details to oversmoothed means, failing to generate plausible 3D structure in occluded regions. This motivates a shift toward generative reasoning based on learned 3D priors to hallucinate fine details and maintain spatial relationships, which are essential for practical deployment in autonomous systems and robotics.

2. System Architecture

FlowSSC is a plug-in generative refinement module that augments any feed-forward monocular SSC method. Its architecture consists of three principal stages:

• (a) Coarse Prediction: A backbone SSC network $F_{pred}$ processes the image $I$, producing a coarse semantic occupancy grid $$X_{coarse} \in \mathbb{R}^{H \times W \times D \times C_{sem}}$$.
• (b) Latent Compression via VecSet VAE: The encoder $E_{ae}(\cdot)$ compresses occupancy grids into triplane latents $h = (h_{xy}, h_{xz}, h_{yz})$, each with shape $\mathbb{R}^{H_{tp} \times W_{tp} \times C}$. Compression is accomplished via a cross-attention VecSet mechanism, treating active voxels as tokens, attended by a 2D positional query set. The decoder $D_{ae}(\cdot)$ reconstructs the voxel grid from any triplane latent via bilinear sampling plus MLP or a shallow 3D CNN. This achieves roughly $100\times$ compression: e.g., $256 \times 256 \times 32 \times C_{sem}$ to $3 \times 128 \times 128 \times 64$.
• (c) Shortcut Latent Diffusion: A Triplane Diffusion Transformer (DiT) equipped with Adaptive LayerNorm (AdaLN) takes as input a noisy latent $h_t \sim p_t$ and conditional information $h_{coarse} = E_{ae}(X_{coarse})$. It learns a shortcut flow field $s_\theta(h_t, t, d)$ that can move the latent along the generative flow in an arbitrary step size $d$. In the one-step case ($d=1$), this enables direct generation of a clean refined latent from noise in a single forward pass. The final output is produced by decoding the refined latent with $D_{ae}$.

3. Mathematical Foundation

FlowSSC leverages conditional continuous-flow models in triplane latent space:

• Flow-Matching ODE: For $h_0 \sim \mathcal{N}(0, I)$ and latent data distribution $p_1(h)$, the ODE

$\frac{dh_t}{dt} = v_t(h_t)$

defines a probability path $\{p_t\}$ connecting noise and data. The velocity field $v_\theta$ is optimized via

$$L_{\rm FM} = \mathbb{E}_{t, h_t \sim p_t} \big\| v_\theta(h_t, t) - u_t(h_t) \big\|^2,$$

where $u_t$ is the target flow.

• Shortcut Flow-Matching: Rather than learning instantaneous velocity, a shortcut field

$$s_\theta(h_t, t, d) \approx \frac{h_{t+d} - h_t}{d}$$

is estimated for any $d \in [0, 1-t]$, with $d=1$ corresponding to a one-step jump from noise to clean data latent.

• Training Objective: The shortcut loss combines instantaneous flow-matching $(L_{\rm FM})$ and a self-consistency term $(L_{\rm SC})$ enforcing additivity of jumps:

$L_{\rm S} = L_{\rm FM} + \lambda L_{\rm SC}.$

The self-consistency loss ensures that a jump of size $2d$ approximates two consecutive jumps of size $d$.

4. Integration with Existing SSC Frameworks

FlowSSC is designed for plug-and-play integration with any existing monocular feed-forward SSC backbone, without requiring any modification to the original network architecture:

• The feed-forward backbone $F_{pred}$ provides initial output $X_{coarse}$ for each input image.
• Latent encoding: $h_{coarse} = E_{ae}(X_{coarse})$.
• A standard normal latent $h_0 \sim \mathcal{N}(0, I)$ (or optionally, a noised version of $h_{coarse}$) is sampled.
• One-step refinement: $h_1 = h_0 + s_\theta(h_0, 0, 1, h_{coarse})$.
• The final refined 3D scene $\widehat{X}$ is decoded via $D_{ae}(h_1)$.

The VecSet VAE is typically pre-trained and then frozen; only the DiT is trained on paired $(X_{coarse}, X_{gt})$ data using the shortcut objective. Backpropagation into $F_{pred}$ is not necessary for effective operation, though end-to-end fine-tuning is possible.

5. Inference, Efficiency, and Practical Considerations

FlowSSC achieves real-time performance by performing a single-step latent refinement:

• Inference: With $d=1$, a single DiT forward pass maps from a noisy latent to a clean one using the shortcut field, followed by VAE decoding.
• Runtime: On an 8×H20-3e system, DiT refinement takes approximately $66$ ms, VAE decoding $150$ ms, for a total of $216$ ms per image (approximately $4.6$ FPS).
• Deployment: Triplane latent compression (approximately $100\times$) ensures that the diffusion model remains lightweight. The mask-based AdaLN mechanism for parameterizing $d$ supports both one-step (fastest) and multi-step (higher fidelity, optional) sampling within a unified architecture.
• DiT computation accounts for about $30\%$ of end-to-end latency, the remainder dominated by voxel decoding.

6. Experimental Validation and Performance

Experiments on the SemanticKITTI SSC split, evaluated using Geometric IoU (Intersection-over-Union) and Semantic mIoU for 20 classes, demonstrate the effectiveness of FlowSSC:

• Quantitative Results: FlowSSC achieves $56.97\%$ IoU and $19.52\%$ mIoU on the test set, surpassing ET-Former ($51.49\%$/$16.30\%$) and all other baselines.
• Ablation Studies:
  • Adding FlowSSC yields $+3.65\%$ IoU and $+5.83\%$ mIoU over the coarse feed-forward baseline.
  • Peak performance is reached at one-step generation ($56.98\%$ IoU); additional refinement steps slightly degrade performance due to deviation from the optimized path.
  • The cross-attention VecSet VAE reconstructs ground-truth at $91.10\%$ IoU, compared to $84.51\%$ for a convolutional VAE, which directly translates to improved generative refinement.
• Qualitative Analysis: Outputs display sharper semantic boundaries and plausible structural hallucination in occluded areas, such as constructing buildings occluded by vegetation and vehicles obscured by roadside elements. Feed-forward baselines tend to yield blank or blurry completions in such cases.
• Practicality: The design satisfies real-time constraints for autonomous driving and robotics, with a route to further latency reduction by optimizing the voxel decoding stage.

---

FlowSSC represents a generative refinement paradigm for monocular SSC, leveraging one-step shortcut diffusion in compressed triplane latent space to enhance the fidelity and plausibility of 3D scene completions over existing deterministic approaches (Xi et al., 21 Jan 2026).