3D Semantic Scene Graphs in Robotics

• 3D semantic scene graphs are structured, heterogeneous graphs that represent object instances and semantic relationships using geometric and label embeddings.
• A heterogeneous GNN with type-specific message passing fuses local observations with historical data, ensuring robust incremental mapping.
• The framework supports real-time updates through instance matching and cross-layer edge formation, facilitating planning, interaction, and embodied AI applications.

A 3D semantic scene graph is a structured and compact representation of a physical environment in which nodes correspond to object instances—augmented with attributes and geometric properties—and edges encode semantic relationships among these entities. This abstraction enables high-level reasoning, incremental mapping, efficient integration of prior observations, and downstream applications such as planning, interaction, and question answering in robotics and embodied AI (Renz et al., 15 Sep 2025).

1. Graph Structure and Formal Definition

A 3D semantic scene graph (3DSSG) is formulated as a heterogeneous, attributed graph $G = (V, E)$, where:

• Nodes: An individual node $i$ represents an object instance, which can exist in either the local subgraph (current perception, e.g., from a new RGB-D frame) or the global subgraph (integration of prior observations across time). Nodes are partitioned by type—LocalObject and GlobalObject.
• Edges:
  • Within-layer edges (intra-global or intra-local): Connect object pairs whose centroids are less than 0.5 meters apart, modeling spatial adjacency.
  • Cross-layer edges: Link matched node pairs $(i,j)$ where the local instance $i$ is associated (via instance matching) to the global node $j$, integrating new and historical observations (see Fig. 1 in (Renz et al., 15 Sep 2025)).

Node feature vector $x_i \in \mathbb{R}^d$ consists of:

• PointNet embedding of 256 sampled object points ($f_p(i)\in\mathbb{R}^{128}$), conveying geometric shape.
• Geometric descriptor $d_g(i)\in\mathbb{R}^7$: centroid $c$, coordinate-wise std, box dimensions $(l,w,h)$, maximum side $L$, volume $V$.
• Label embeddings ($f_{lbl}(i)$): global nodes only, encoded as one-hot (27 classes) or as a 512-dimensional CLIP embedding for open-vocabulary semantic information.

Edge feature $e_{ij}\in\mathbb{R}^f$ encodes spatial differentials:

• $$e_{ij} = [c_j-c_i, \mathrm{std}_j-\mathrm{std}_i, \log\textstyle\frac{l_j}{l_i}, \log\frac{w_j}{w_i}, \log\frac{h_j}{h_i}, \log\frac{L_j}{L_i}, \log\frac{V_j}{V_i}]$$
• Processed by a two-layer MLP (Renz et al., 15 Sep 2025).

2. Heterogeneous GNN Architecture and Message Passing

The 3DSSG employs a heterogeneous graph neural network (GNN) to jointly reason about local and global information flow:

• Types:
  • Node types $$\tau \in \{\mathrm{LocalObject}, \mathrm{GlobalObject}\}$$
  • Edge types $$\rho \in \{\mathrm{local-local}, \mathrm{global-global}, \mathrm{local-global}\}$$
• Message passing: For each GNN layer $l$,
  • For each edge type $\rho$, compute messages using type-specific MLP $f_\rho$:
  • $$m_{ij}^{(l),\rho} = f_\rho(h_i^{(l)}, h_j^{(l)}, e_{ij}, p_{ij})$$, where $p_{ij}$ is a binary prior-observation indicator for cross-layer match.
  • Aggregate incoming messages per node $i$:

  $$a_i^{(l)} = \sum_\rho\sum_{j\in\mathcal{N}_\rho(i)} m_{ij}^{(l),\rho}$$ - Node update: $$h_i^{(l+1)} = \mathrm{UPDATE}(h_i^{(l)}, a_i^{(l)})$$ where UPDATE consists of a two-layer MLP, ReLU, layer-norm, and dropout.

Classifier heads predict node classes and edge relations by forwarding the final layer embeddings through an MLP.

Incorporation of semantic embedding modules (e.g., CLIP) enables effective cross-modal fusion, as label embeddings are concatenated into $h_j^{(0)}$ for global nodes (Renz et al., 15 Sep 2025).

3. Incremental Pipeline for Graph Construction and Update

The framework operates incrementally at each timestep $t$:

1. Local reconstruction from the latest RGB-D frame:

   • Segment objects to create local nodes and intra-local edges based on spatial proximity (0.5 m threshold).
2. Prior observation integration:
   • Perform instance matching between new detections $V_{\mathrm{loc}}^t$ and the current global graph $V_{\mathrm{glob}}^{t-1}$, supervising with ground-truth during training.
   • Insert cross-layer edges between matched object instances.
3. Heterogeneous GNN forward pass on the union graph:
   • Two message-passing layers update features of all nodes and edges in the combined $$\left(V_{\mathrm{loc}}^t \cup V_{\mathrm{glob}}^{t-1}, E_{\mathrm{loc}}^t \cup E_{\mathrm{glob}}^{t-1} \cup E_{\mathrm{cross}}^t\right)$$.
4. Prediction: Classify objects and infer updated predicates on local nodes/edges.
5. Graph merge: Integrate matched and unmatched local objects by downsampling points, updating descriptors, and appending as necessary to the global state (Renz et al., 15 Sep 2025).

No global point-cloud history or full scene reconstruction is required, supporting scalability in long-horizon, real-world deployments.

4. Training Objectives and Loss Functions

Training is supervised through two primary losses:

• Node classification: Weighted cross-entropy over object classes in the local graph, with class weighting $w_n^c = 10/\log(n_n^c)$ to counter class imbalance.

$$\mathcal{L}_n = -\sum_{i \in V_{\mathrm{loc}}^t} w_n^{y_i} \log \mathrm{Softmax}(\hat{y}_i)_{y_i}$$

• Edge relation classification: Multi-label binary cross-entropy, with relation class weights $w_e^r = 10/\log(n_e^r) + 1$, and positive-class scaling ($\alpha=40$).

$$\mathcal{L}_e = -\sum_{(i,j) \in E_{\mathrm{loc}}^t} \sum_r w_e^r \left[ y_{ij}^r \log \sigma(\hat{r}_{ij}^r) + (1 - y_{ij}^r) \log (1-\sigma(\hat{r}_{ij}^r)) \right]$$

• The total batch loss:

$\mathcal{L} = \mathcal{L}_n + \alpha \mathcal{L}_e$

(Renz et al., 15 Sep 2025)

5. Experimental Evaluation and Results

The evaluation is performed on a 3DSSG/RIO27 split (1320 scenes, 0.8/0.1/0.1 train/val/test):

• Metrics:
  • Node classification: Acc@1, Acc@5, and unseen-node Acc@k.
  • Relationship inference: mean edge recall (Rec); ng-Recall@k (fraction of ground-truth (subject, predicate, object) triples among top-k predictions).
• Baselines: Compared to SGFN, homogeneous GraphSAGE, and ablated models (plain, one-hot, CLIP, with corrupted labels).
• Key results ([Table 1, (Renz et al., 15 Sep 2025)]):
  • Heterogeneous + CLIP + HGT attains ng-R@50 = 0.80; ng-R@100 = 0.84 (relationship recall).
  • Homogeneous SAGE+CLIP gives Acc@1 = 0.98 and Acc@5 = 0.99 but fails to predict relationships (relationship recall = 0).
  • Adding one-hot or CLIP label embeddings notably boosts all metrics; CLIP further enhances relationship prediction.
  • Additional edge types (e.g., harmonic-centrality) yield minor further improvements, demonstrating architectural flexibility.

6. Advantages, Limitations, and Future Directions

Advantages:

• Prior observations are integrated in real time at the GNN level without necessitating storage of the full historical scene or point cloud.
• The heterogeneous GNN captures the semantic heterogeneity of objects and relationships, with flexible node/edge types facilitating multimodal fusion.
• The model can serve as a generic backbone for adding new modalities (e.g., symbolic knowledge bases).

Limitations:

• Performance depends on accurate instance matching; in real deployments, segmentation and tracking errors can introduce errors into the incremental graph.
• Off-the-shelf GNN layers (GraphSAGE/HGT) do not use edge features natively; precise handling of richer relational encodings requires explicit customization.
• Corruption of global semantic labels significantly degrades relationship recall (up to –0.46 in ng-Recall), highlighting the sensitivity to high-quality semantic embedding.

Future directions:

• Developing robustness to segmentation noise and imperfect matching.
• Designing learned, transformer-based edge-feature integration for better relational reasoning.
• Incorporating temporal dynamics using recurrent GNNs for non-static environments.
• Real-world validation with robotic systems, incremental SLAM, and semantic mapping (Renz et al., 15 Sep 2025).

---

References:

• Integrating Prior Observations for Incremental 3D Scene Graph Prediction (Renz et al., 15 Sep 2025)