ArtiSG: Functional 3D Scene Graph Construction via Human-demonstrated Articulated Objects Manipulation

Abstract: 3D scene graphs have empowered robots with semantic understanding for navigation and planning, yet they often lack the functional information required for physical manipulation, particularly regarding articulated objects. Existing approaches for inferring articulation mechanisms from static observations are prone to visual ambiguity, while methods that estimate parameters from state changes typically rely on constrained settings such as fixed cameras and unobstructed views. Furthermore, fine-grained functional elements like small handles are frequently missed by general object detectors. To bridge this gap, we present ArtiSG, a framework that constructs functional 3D scene graphs by encoding human demonstrations into structured robotic memory. Our approach leverages a robust articulation data collection pipeline utilizing a portable setup to accurately estimate 6-DoF articulation trajectories and axes even under camera ego-motion. We integrate these kinematic priors into a hierarchical and open-vocabulary graph while utilizing interaction data to discover inconspicuous functional elements missed by visual perception. Extensive real-world experiments demonstrate that ArtiSG significantly outperforms baselines in functional element recall and articulation estimation precision. Moreover, we show that the constructed graph serves as a reliable functional memory that effectively guides robots to perform language-directed manipulation tasks in real-world environments containing diverse articulated objects.

• The paper introduces ArtiSG, a framework that integrates human demonstration-derived kinodynamic priors to embed actionable functional information into 3D scene graphs.
• It employs a three-stage process—semantic initialization, viewpoint-robust articulation estimation, and interaction-augmented graph refinement—to significantly boost functional element recall.
• Experimental results demonstrate major performance gains, with functional recall improving from 55.8% to 88.5% and substantial reduction in trajectory estimation errors in dynamic conditions.

Functional 3D Scene Graph Construction from Human Demonstrations for Robotic Manipulation

Introduction

The construction of scene graphs for robotic applications has predominantly focused on semantic and geometric representations, often omitting the crucial functional information necessary for physically grounded interaction with articulated objects. The paper "ArtiSG: Functional 3D Scene Graph Construction via Human-demonstrated Articulated Objects Manipulation" (2512.24845) introduces a framework that directly addresses this gap. It proposes encoding human demonstration data—specifically, manipulation trajectories and articulation mechanisms—into hierarchical, open-vocabulary 3D scene graphs. This approach establishes a robust robotic memory for subsequent task planning and manipulation, effectively linking visual perception to actionable affordances.

Figure 1: Human demonstration-derived articulation trajectories are extracted and registered as functional elements, enabling open-vocabulary localization and action priors for manipulation in the constructed scene graph.

System Architecture

ArtiSG is structured in three sequential stages:

1. Scene Graph Initialization: The system constructs an initial semantic and geometric scene graph using multi-view RGB-D mapping. Object-level nodes and candidate functional elements are identified utilizing instance segmentation and clustering, complemented by top-$k$ frame selection for optimal viewpoint aggregation. Semantic features are extracted using powerful vision-language encoders, and functional elements are detected with promptable object detectors followed by fine-grained mask segmentation models. Results from multiple views are fused to improve robustness and recall.
2. Viewpoint-Robust Articulation Estimation: The crucial innovation is a hardware-assisted data collection pipeline. A head-mounted RGB-D camera tracks a UMI gripper with a polyhedral ArUco marker sphere, enabling robust 6-DoF pose estimation even under dynamic operator ego-motion. The system fuses articulated manipulation trajectories with SLAM-derived camera poses and applies PCA/SVD-based routines to classify prismatic and revolute articulations and recover respective axes.

   Figure 2: System overview of ArtiSG, depicting the staged progression from multi-view semantic aggregation to viewpoint-robust kinodynamic estimation and final interaction-driven graph refinement.

   

   Figure 3: The marker-equipped UMI gripper and optitrack hardware foundation enable precise 6-DoF articulation trajectory acquisition and evaluation.

3. Interaction-Augmented Graph Refinement: Manipulation-derived trajectories and articulation parameters are geometrically aligned with the graph nodes. When human interactions expose functional elements missed in visual initialization due to occlusion or implicitness, new nodes are dynamically instantiated. For elements already detected, articulation and kinematic priors are explicitly attached to the corresponding graph nodes, enhancing the graph's actionability.

Functional Element Detection and Scene Graph Quality

Evaluation on both simulation (Behavior-1k) and real-world (kitchen, pantry, tabletop) environments demonstrates ArtiSG's clear superiority in functional element recall and generalizability over baselines including OpenFunGraph and Lost&Found. Notably, introducing human demonstration data boosts functional element recall in real-world scenes from 55.8% to 88.5% and overall F1 score, reflecting effective compensation for the limitations of static vision-only approaches.

ArtiSG's open-vocabulary querying mechanism, supported by top-$k$ frame aggregation and advanced vision-LLMs, yields high-fidelity node representations. This enables precise instance retrieval and supports complex language-driven manipulation tasks. Visualization results illustrate the enhanced localization accuracy and completeness achieved, particularly with inconspicuous or ambiguous elements.

Figure 4: Qualitative comparison shows ArtiSG's superior localization and recall of ground-truth functional elements over baselines in both real and simulated domains.

Articulation Axis and Trajectory Estimation

Articulation parameter estimation is quantitatively validated using an OptiTrack motion capture system. Compared to vision-based hand/keypoint tracking (Mediapipe, CoTracker) and static vision approaches (GFlow), the marker-based trajectory acquisition delivers an order-of-magnitude improvement in trajectory and axis estimation error, especially in dynamic conditions and for revolute joints. The approach is robust to operator movement and viewpoint changes, outperforming state-of-the-art trackers even under challenging conditions (trajectory RMSE reduced to 0.82 cm in dynamic scenarios).

Figure 5: Viewpoint-robust articulation tracking is visualized for both prismatic and revolute joints; the decoupled marker system enables high-precision 6-DoF pose recovery.

Robotic Manipulation via ArtiSG Memory

Downstream utility is demonstrated by integrating ArtiSG with a Franka Research 3 arm for language-guided opening tasks. The system is evaluated on objects with inconspicuous functional elements, atypical kinematics, or ambiguous mechanisms. While even advanced VLMs frequently mispredict affordance locations and articulation types under visual ambiguity or partial observability, ArtiSG leverages demonstration-derived graph memory to retrieve precise interaction trajectories and axis constraints. This enables successful, physically grounded execution of complex manipulation tasks.

Figure 6: Robots actuate accurately on objects with subtle or visually ambiguous mechanisms by following trajectories stored in ArtiSG, in contrast to pure VLM-based approaches that fail under limited visual cues.

Theoretical and Practical Implications

ArtiSG advances the state of embodied scene understanding in two significant directions:

• The integration of human-interaction-derived kinodynamic priors explicitly augments perception-based scene representations, bridging the gap between static detection and actionable affordances.
• Robustness to diverse viewpoints and unconstrained environments is realized via hardware decoupling and dynamic graph refinement, pushing the limits of manipulation policy generalizability and reliability.

Practically, ArtiSG supports a broader repertoire of manipulation tasks, especially in open, unstructured settings where static perception inevitably misses implicit or occluded affordances. The approach constitutes a step toward more universally applicable functional scene representations critical for lifelong robot learning and task transfer.

Future Work

Future prospects include removing dependence on marker-based hardware in favor of markerless, vision-based pose estimation to further facilitate in-the-wild deployment, and tighter integration of the functional prior graph into generalist robot manipulation policy learning. Such integration would allow explicit kinodynamic constraints and affordance knowledge stored in scene graphs to guide diffusion policies and task sequences in complex, dynamic human environments.

Conclusion

The ArtiSG framework systematically enhances 3D scene graphs by embedding actionable functional information derived from human demonstrations, offering a robust, interaction-augmented foundation for next-generation robot manipulation in complex real-world environments. Its design, combining scene semantics, viewpoint-robust kinodynamic capture, and open-vocabulary functional querying, enables precise, language-guided interventions and provides a scalable pathway for future advances in embodied AI and robotic autonomy.