Hierarchical 4D Scene Graphs

• Hierarchical 4D Scene Graphs are unified multi-level structures that combine spatial, semantic, and temporal dynamics to model evolving real-world environments.
• They employ hierarchical decomposition with spatial-semantic and temporal edges, using techniques like spectral parameterization to capture motion flows and activity patterns.
• Applications include autonomous navigation, interactive 3D scene generation, and real-time robotic decision-making by integrating dynamic observations with static spatial contexts.

Hierarchical 4D Scene Graphs encode spatial structure and temporal evolution together in a unified, multi-level abstraction, serving as a critical foundation for autonomous navigation, 3D scene generation, and robotic decision-making in dynamic, real-world environments. These structures extend conventional 3D scene graphs by incorporating temporal dynamics, capturing not only geometric and semantic properties of environments but also the evolution of object relations, motion flows, and activity patterns over time.

1. Formal Definitions and Core Structure

A hierarchical 4D scene graph is defined as a tuple $\mathcal{G}_{4D} = (V, E, H, \mathcal{T})$, integrating hierarchical node organization, rich edge semantics, and explicit temporal modeling (Catalano et al., 10 Dec 2025, &&&1&&&, Hou et al., 30 May 2025). At time $t$, the instantaneous 3D scene graph is denoted $\mathcal{G}^t = (V^t, E^t)$, where $V^t$ is the set of graph nodes and $E^t$ is the set of edges. Nodes are partitioned into disjoint hierarchy levels $\ell=1,\dots,L$ such that $V^t = \bigsqcup_{\ell=1}^L V^{(t)}_\ell$.

Hierarchy Levels:

• Lower levels typically encode object instances and their fine-grained relationships.
• Higher levels capture coarser spatial partitions (e.g., rooms, functional areas).
• Some frameworks introduce a dedicated “navigational” layer ($\ell=n$), where nodes carry aggregated motion statistics and temporal activity descriptors (Catalano et al., 10 Dec 2025).

Edge Semantics:

• Spatial-semantic edges ($E_S$): encode physical adjacency, hierarchical containment (e.g., object-in-room), and navigational connectivity.
• Temporal edges ($E_t$): maintain identity across time steps or link states at $t-1$ and $t$, carrying information about the evolution or transitions between static and dynamic states (Liu et al., 2024).

The temporal component is represented by a mapping $\mathcal{T} = \{s_i(t)\}_{v_i\in V_n}$, where $s_i(t)$ are temporal descriptors (e.g., motion histories or state vectors) for each navigational node. The global structure is thus a sequence $\{\mathcal{G}^t\}_t$, with temporal evolution encoded either explicitly through edges or implicitly through node states.

2. Methods for Temporal Dynamics and Motion Representation

Temporal evolution in hierarchical 4D scene graphs is realized by integrating dynamic observations and modeling motion patterns at multiple resolutions.

Sparse Map of Dynamics (MoD):

• Each navigational node $v_i \in V_n$ is assigned a $B$-bin orientation histogram over $\lambda$ temporal descriptors: $s_i(t) \in \mathbb{R}^{B \times \lambda}$ (Catalano et al., 10 Dec 2025).
• Incoming observations $(p_h, \theta_h, t)$ are hashed spatially; their activity is incremented in the corresponding orientation bin and spatial cell.
• After a stability window $\tau$, historical data is anchored to the nearest graph node and cleared from the sparse hash table.

Spectral (FreMEn) Parameterization:

• Temporal channels $s_{i,b}(t)$ are modeled as sums of $K$ sinusoids plus a bias, enabling representation of periodicity and non-stationarity:

$$s_{i,b}(t) \approx c_{i,b} + \sum_{k=1}^K [a_{i,b,k}\cos(\omega_k t) + b_{i,b,k}\sin(\omega_k t)]$$

Coefficients are fit by incremental least squares in the frequency domain (Catalano et al., 10 Dec 2025).

Motion-Flow Function:

• The predicted flow from node $u$ to $v$ over $\Delta t$ is defined as the probability $F_{\Delta t}(u \to v)$ that an agent currently at $u$ is observed to move towards $v$ at $t+\Delta t$, mapping directional motion to histogram bins and normalizing across neighbors.

Other frameworks, such as GraphCanvas3D, use temporal edges $E_t$ and recurrent neural units (e.g., GRU) to propagate dynamic context alongside message passing, supporting both discrete and continuous modeling of object trajectories (Liu et al., 2024).

3. Hierarchical Decomposition and Anchoring Strategies

Hierarchical decomposition partitions the graph across spatial, semantic, and functional levels, promoting efficient reasoning and scalability.

Global and Local Partitioning:

• Hi-Dyna Graph defines a global static graph $G^g = (V^g,E^g)$ capturing room/layout and persistent objects, and a local dynamic subgraph $G^d_t = (V^d_t,E^d_t)$ representing moving entities, transient objects, and evolution of relations within a time window (Hou et al., 30 May 2025).
• Spatial and semantic anchoring aligns dynamic nodes and relations to their global static context, either by 3D bounding box intersection (spatial anchoring) or semantic label matching (semantic anchoring).
• After each update window, dynamically observed instances are either merged with existing global nodes (when overlap or label agreement occurs) or introduced as new nodes, maintaining persistent hierarchical connectivity.

GraphCanvas3D employs multilevel message passing and optimization, from raw object features (level $\ell=0$) to subgraph and global cluster levels, enabling structured propagation of spatial, semantic, and temporal context (Liu et al., 2024).

4. Algorithms, Objective Functions, and Computational Considerations

Construction and maintenance of hierarchical 4D scene graphs involve several key algorithmic steps.

Aion (Hierarchical MoD):

• SLAM-derived geometry and semantics are registered as 3DSG nodes and edges.
• Dynamic-object detections increment orientation histograms in a sparse hash structure.
• After temporal stabilization, data is mapped to graph nodes, periodic models are fit to temporal channels, and spectral coefficients updated in batches.
• No deep-learning losses are required for temporal modeling; FreMEn provides a fully spectral/probabilistic fit (Catalano et al., 10 Dec 2025).
• Computational complexity is $O(1)$ per observation for online updates, $O(|V_n| \cdot B \cdot K)$ for spectral updates, and linear in the number of active navigational nodes for memory.

GraphCanvas3D (Controllable Scene Generation):

• In-context LLMs construct and dynamically edit the graph on the fly, parsing language prompts into node/edge sets and temporal instructions without retraining.
• Hierarchical layout optimization proceeds via gradient-based updates, constrained by semantic and spatial relations at multiple levels, and guided by learned compatibility scores or neural message passing.
• Temporal smoothness is realized via recurrent processing and explicit edge weighting (Liu et al., 2024).

Hi-Dyna Graph:

• Dynamic subgraph maintenance employs segmentation, tracking, transformer-based relation prediction, and sliding-window graph construction.
• Object and region anchoring is handled by 3D IoU or semantic class matching.
• An LLM-based reasoning module serializes the unified graph into a prompt, facilitating affordance-aware planning and action sequencing.

5. Applications to Autonomy, Planning, and Interactive Generation

Hierarchical 4D scene graphs have direct implications for autonomous navigation, task-driven manipulation, and interactive scene synthesis.

• Aion demonstrates improved dynamic navigation by integrating predicted flow probabilities $F_{\Delta t}(u \to v)$ into A* planning edge costs. Empirical results show reduced traversal through high-entropy or opposing-flow regions and increased path safety compared to distance-only baselines (Catalano et al., 10 Dec 2025).
• Hi-Dyna Graph enables long-horizon autonomy in human-centric environments: a robot agent reasons over the composite scene graph to infer tasks, plan navigation and manipulation sequences, and execute affordance-constrained actions without further training or external reward shaping (Hou et al., 30 May 2025).
• GraphCanvas3D supports flexible, run-time controllable generation of dynamic 3D environments driven by language instructions, with adaptation achieved without retraining and multi-level graph optimization ensuring layout and temporal coherence (Liu et al., 2024).

6. Evaluation Metrics and Empirical Validation

Evaluation of hierarchical 4D scene graph systems encompasses both statistical and task-driven metrics.

Aion’s Hierarchical MoD is quantitatively assessed using Jensen–Shannon divergence, Bhattacharyya distance, Wasserstein (angular) distance, and circular correlation, measuring both entropy and directionality in historical and predicted motion flows. Across 20 scenes and 6 agents, hierarchical MoD achieves comparable or improved accuracy to grid-based baselines while using fewer spatial units (e.g., JS differences $<0.2$ for entropy prediction) (Catalano et al., 10 Dec 2025).

GraphCanvas3D is evaluated by:

• CLIP Score for image–text alignment per rendered frame.
• MLLM Score for multi-view semantic consistency.
• User-rated scene quality, geometric fidelity, and temporal smoothness.
• A quantitative temporal-coherence metric:

$$C_{\mathrm{temp}} = 1 - \frac{1}{T-1}\sum_{t=2}^T \frac{\|\mathbf{H}^t-\mathbf{H}^{t-1}\|_2}{\|\mathbf{H}^{t-1}\|_2}$$

A higher $C_{\mathrm{temp}}$ reflects smoother trajectories (Liu et al., 2024).

Hi-Dyna Graph demonstrates real-world effectiveness by enabling robots to autonomously execute complex tasks as a cafeteria assistant without retraining, backed by qualitative video demonstrations and persistent, efficiently updated graph structures (Hou et al., 30 May 2025).

7. Practical Considerations and System Scalability

Hierarchical 4D scene graph systems must maintain representational richness while scaling efficiently to large, dynamic environments.

• Sparse data structures (e.g., C++ unordered_maps for visited hash cells; parallel arrays of temporal model coefficients) reduce storage and computation to only active, semantically relevant regions.
• Updates upon each observation are constant-time; periodic batch model updates scale linearly with the number of navigational nodes.
• Hierarchical anchoring and local/global partitioning minimize the update impact of local changes and localize the computational burden (Catalano et al., 10 Dec 2025, Hou et al., 30 May 2025).
• Dynamic, in-context, LLM-based interfaces facilitate flexible interaction and editing without retraining, applicable to human-robot collaboration and procedural content generation (Liu et al., 2024, Hou et al., 30 May 2025).

In summary, hierarchical 4D scene graphs unify spatial, semantic, and dynamic aspects of environments in an interpretable, scalable, and operationally proven structure, enabling advanced capabilities in autonomy, interactive scene generation, and context-aware planning in both simulated and real-world domains (Catalano et al., 10 Dec 2025, Liu et al., 2024, Hou et al., 30 May 2025).