Probabilistic Fusion for Semantic & Geometric Info

• Probabilistic fusion is a set of techniques that integrate semantic probabilities and geometric measurements using Bayesian principles.
• These methods underpin robust 3D mapping in applications like autonomous driving and SLAM by addressing sensor noise and occlusion.
• Implementations include volumetric mapping, surfel‐based fusion, sensor-projection, and attention-guided neural approaches that refine uncertainty over time.

Probabilistic fusion for semantic and geometric information refers to a set of techniques that combine uncertainty-aware semantic predictions (e.g., class probabilities from visual perception) and geometric information (e.g., 3D structure from depth, lidar, or SLAM) into coherent representations such as semantic maps, 3D reconstructions, or object-aware environments. These methods are foundational in robotics, autonomous driving, mixed reality, and computer vision, where robust interpretation of complex, noisy, multi-modal sensor data is required for decision making and scene understanding. The core principle is the adoption of Bayesian or probabilistically principled updates to maintain, integrate, and refine semantic and geometric evidence over time, accommodating sensor noise, label ambiguity, occlusion, and dynamics.

1. Bayesian Formulation for Semantic-Geometric Fusion

Probabilistic fusion is commonly grounded in recursive Bayesian estimation, where semantic and geometric evidence are modeled as latent random variables updated as new measurements arrive. The canonical update, as applied to semantic mapping tasks, utilizes the Bayes filter:

$$p(\ell_t | z_{1:t}) \propto p(z_t | \ell_t)\,p(\ell_{t} | z_{1:t-1})$$

where $\ell_t$ is the latent semantic label (or class-distribution) at a location or instance, and $z_t$ comprises the semantic evidence (e.g., segmentation scores, open-set tags) and geometric evidence (e.g., depth, pose, 3D association) at time $t$.

This approach underlies per-instance, per-voxel, or per-surfel fusion in dense mapping pipelines as well as landmark-based probabilistic SLAM (Liu et al., 2024, McCormac et al., 2016, Michael et al., 2022). Measurement likelihoods are factorized over the available semantic cues, such as open-set foundation-model labels and their corresponding similarity scores, and geometric cues, such as instance associations or 3D overlaps.

2. Fusion Architectures and Algorithmic Pipelines

Modern systems implement probabilistic semantic–geometric fusion via diverse algorithmic pipelines:

• Volumetric and Surfel-based 3D Maps:

Pipelines such as FM-Fusion (Liu et al., 2024) and SemanticFusion (McCormac et al., 2016) maintain a geometric backbone (TSDF volumes, surfel clouds) where each spatial primitive carries a semantic probability mass function (pmf) over closed-set classes:

- FM-Fusion: Maintains both a global TSDF and per-instance TSDF grids, updating their class pmfs using open-set measurements and a learned label-likelihood matrix, via recursive Bayes and running-average smoothing. - SemanticFusion: Attaches pmfs to surfels, updating via Bayesian multiplication of per-pixel CNN softmax probabilities with the surfel's prior, projected under tracked SLAM correspondences.

• Sensor-Projection Fusion:

For multi-modal data (e.g., camera-lidar fusion), probabilistic fusion involves projecting semantic probabilities from 2D camera images into 3D (lidar) space, correcting for pose, motion, and occlusion, then registering the resulting class distributions in 3D voxels, octrees, or point clouds (Berrio et al., 2020, Berrio et al., 2020, Paz et al., 2020). Uncertainty is propagated through geometric transformations (including using the Unscented Transform for pose and projection models), and information from multiple frames is recursively integrated using cell-wise or voxel-wise Bayesian updates.

• Probabilistic Data Association in Semantic SLAM:

For SLAM with ambiguous object correspondence, frameworks compute marginal assignment probabilities over measurement-landmark pairs using k-best assignment enumeration, then use association-weighted measurement likelihoods combining semantic (e.g., class softmax) and geometric (e.g., ellipsoid or bounding-box overlap) factors in the joint inference objective (Michael et al., 2022).

• Text-Guided and Multistream Attention Fusion:

In neural approaches such as TextSplat, semantic–geometric fusion occurs at feature level: depth, semantic, and cross-view features are aligned, aggregated, and fused using attention mechanisms guided by textual embeddings, producing probabilistic Gaussian splats in 3D whose parameters (mean, covariance, opacity, color) are inferred via differentiable neural regression (Wu et al., 13 Apr 2025).

3. Measurement Uncertainty, Label Likelihoods, and Occlusion Handling

Most frameworks explicitly model and propagate measurement uncertainty:

• Semantic likelihoods:

Measurement likelihoods are extracted from CNN softmax outputs, open-set detection scores (from foundation models), or confusion matrices empirically estimated on held-out data (Liu et al., 2024, Paz et al., 2020). For mapping, superpixel-based temperature scaling is often applied to flatten over-confident predictions in low-purity regions (Berrio et al., 2020, Berrio et al., 2020).

• Geometric uncertainty:

In lidar-camera fusion, point-level covariances are propagated through transform chains using the Unscented Transform, yielding uncertainty ellipses for projection into image space (Berrio et al., 2020).

• Occlusion and visibility:

Dedicated viewpoint validation algorithms mask out occluded 3D points by simulating projective masks along depth ordering (front-to-back) in the image plane, ensuring label fusion only for visible, unoccluded points (Berrio et al., 2020, Berrio et al., 2020).

• Instance association and merge:

Data association steps exploit IoU in 2D and 3D, per-instance semantic similarity, and temporally linked open-set measurements to robustly associate observations or merge over-segmented regions (Liu et al., 2024).

4. Map Representation, Update Rules, and Refinement

Semantic–geometric probabilistic fusion produces map representations such as:

• Dense voxel or surfel grids:

Each unit stores a class pmf, updated recursively as new observations arrive—either by strict Bayes rule or by running-average smoothing to reduce over-confidence. Occupancy probabilities are similarly tracked for geometric reasoning (McCormac et al., 2016, Berrio et al., 2020).

• Instance volumes:

FM-Fusion maintains per-instance TSDFs, performing data-association and implementing merge/prune strategies for over-segmentation and geometric noise. Instance semantic pmfs are updated using a learned (matrix) probability of mapping open-set label and detection scores to closed-set classes (Liu et al., 2024).

• Bird's-Eye-View (BEV) Grids:

In mapping for autonomous driving, 3D point clouds "painted" from camera semantics are collapsed into 2D BEV cells, whose class pmfs are updated via a confusion-matrix likelihood and intensity priors—supporting probabilistic outputs necessary for downstream planners or control modules (Paz et al., 2020).

• Gaussian Splatting:

Representations such as in TextSplat treat each "splat" as a probabilistically inferred Gaussian kernel in $\mathbb{R}^3$, with geometric parameters and RGB attributes fused from multimodal features via neural, attention-based mechanisms (Wu et al., 13 Apr 2025).

Refinement modules in these pipelines include:

• Boundary polishing:

Merge heuristics based on semantic similarity and 3D IoU, deletion of noisy or unsupported voxels based on global geometry, and optional spatial or temporal smoothing (Liu et al., 2024, Berrio et al., 2020).

• Regularization:

CRF-based spatial smoothing across adjacent surfels or voxels, and propagation of pmf fractions to neighbors for denoising (McCormac et al., 2016, Berrio et al., 2020).

5. Empirical Performance and Evaluation

The efficacy of probabilistic semantic–geometric fusion is consistently demonstrated across various benchmarks:

• Instance-aware 3D semantic mapping:

FM-Fusion yields 40.3 mAP@50% on ScanNet—a significant +15.6 point gain over the strong open-set baseline (Kimera + RAM-Grounded-SAM), with discrete ablations indicating that probabilistic likelihood integration, prompt augmentation, and instance merge refinement each contribute 3–10 mAP (Liu et al., 2024).

• Dense RGB-D reconstruction:

SemanticFusion demonstrates a 3–5% increase in class and pixel accuracy over single-frame segmentation on NYUv2 and office scenes, with additional gains from CRF regularization (McCormac et al., 2016).

• Sensor fusion benchmarks:

In fused camera–lidar settings, proper motion correction and occlusion masking boost F1 scores by 4–5 points for thin or complex object classes; full probabilistic pipelines further reduce class confusion and improve map consistency (Berrio et al., 2020, Berrio et al., 2020).

• SLAM scalability and accuracy:

Probabilistic data association via k-best enumeration achieves near-exact marginal assignment probabilities in sub-millisecond time even with 50×50 assignment matrices; resulting semantic SLAM backends run in real time and maintain robustness in ambiguous, repetitive environments (Michael et al., 2022).

• Text-guided reconstructions:

TextSplat achieves 0.39 dB PSNR and 0.010 SSIM gains over previous best GGS on RealEstate10K/ACID, with a reported 3.2% IoU improvement for semantic consistency in part labeling (Wu et al., 13 Apr 2025).

6. Limitations, Assumptions, and Future Directions

Published works discuss several limitations:

• Dynamic Environments:

Most approaches assume static class probabilities per cell or voxel. Handling persistent dynamic objects or temporal label changes (e.g., FreMEn models, temporal decay) is an open direction (Berrio et al., 2020).

• Sparse Sensing and Noise:

Sparse lidar or limited camera views reduce label fusion quality, especially at range or in occluded zones. Early mislabeling may persist in unsensed regions.

• Computational Load:

Uncertainty propagation via the Unscented Transform and dense per-point Bayesian integration are computationally intensive; hybrid EKF linearizations or parallelization are suggested extensions (Berrio et al., 2020).

• End-to-End Learning:

Boundary and instance refinement remain largely heuristic; fully differentiable, loss-driven geometric-semantic polishing could further improve consistency and sharpness (Liu et al., 2024, Wu et al., 13 Apr 2025).

This suggests that future research will increasingly focus on scalable, uncertainty-aware, differentiable fusion architectures, robust to cross-domain shift, dynamic scenes, and open-set conditions.

7. Comparative Summary of Representative Frameworks

Framework	Semantic Inputs	Geometric Representation	Fusion Mechanism
FM-Fusion (Liu et al., 2024)	Open-set VLM detections	TSDF (global + per-inst)	Bayesian update, label-likelihood matrix, instance merge/refinement
SemanticFusion (McCormac et al., 2016)	CNN softmax, per-pixel	Surfel cloud	Per-surfel Bayes rule, CRF reg.
Camera-Lidar Fusion (Berrio et al., 2020)	2D semantic with uncertainty	Octree voxels	Bayes rule via integration over uncertain projections, occlusion-checking
BEV Mapping (Paz et al., 2020)	Image segm. with conf. matrix	Grid/BEV	Per-cell Bayesian update w/ confusion matrix, intensity prior
Probabilistic SLAM (Michael et al., 2022)	Softmax scores, bbox/ellipsoid	Object landmarks	Joint semantic-geometric measurement model, k-best data association
TextSplat (Wu et al., 13 Apr 2025)	Depth, semantic, cross-view	3D Gaussian splats	Neural attention, text-guided fusion, end-to-end learning

Each of these frameworks embodies the fundamental principles of probabilistic semantic–geometric fusion: explicit modeling of uncertainty, principled Bayesian updates, strong coupling between geometry and semantics, and robust integration over time, sensors, and ambiguous evidence.