Leveraging VR Robot Games to Facilitate Data Collection for Embodied Intelligence Tasks

Abstract: Collecting embodied interaction data at scale remains costly and difficult due to the limited accessibility of conventional interfaces. We present a gamified data collection framework based on Unity that combines procedural scene generation, VR-based humanoid robot control, automatic task evaluation, and trajectory logging. A trash pick-and-place task prototype is developed to validate the full workflow.Experimental results indicate that the collected demonstrations exhibit broad coverage of the state-action space, and that increasing task difficulty leads to higher motion intensity as well as more extensive exploration of the arm's workspace. The proposed framework demonstrates that game-oriented virtual environments can serve as an effective and extensible solution for embodied data collection.

• The paper presents a novel gamified VR framework that integrates procedural scene generation, teleoperation, and automatic task evaluation for scalable data collection.
• It demonstrates robust state-action coverage in a trash pick-and-place simulation, showcasing measurable improvements in task diversity and arm manipulation.
• The system reduces expert workload and offers extensibility for complex, multi-agent embodied tasks, paving the way for advanced imitation and reinforcement learning.

Leveraging VR Robot Games for Scalable Embodied Data Collection

Introduction

The paper "Leveraging VR Robot Games to Facilitate Data Collection for Embodied Intelligence Tasks" (2604.16903) proposes a systematic framework for embodied data collection that utilizes a gamified approach within a Unity-based simulation environment. The framework integrates procedural scene generation, VR-based humanoid robot teleoperation, automatic task evaluation, and comprehensive trajectory logging to address the persistent challenge of acquiring high-quality, diverse, and scalable datasets for embodied AI. The authors validate their methodology via a prototype implementation for a trash pick-and-place task and provide empirical analysis on data quality, diversity, and the role of task difficulty in eliciting broad behavioral coverage.

Figure 1: A screenshot of the VR-based robot control prototype.

System Architecture

The proposed architecture consists of four tightly linked modules: procedural scene generation, VR-based human-robot interaction, episode/task management with automatic assessment, and structured data aggregation. The modularity and extensibility of the system allow seamless adaptation to new embodied tasks and multi-agent settings.

Procedural scene generation is achieved by automatically populating indoor environments according to randomized templates and validity constraints, ensuring physically plausible and diverse task instantiations. VR-based robot teleoperation leverages a PICO Neo3 headset, where controller signals are mapped to robot motion primitives, including RL-pretrained lower-limb locomotion, clutch-mode arm control with inverse kinematics, and fine-grained gripper articulation. Task progression is handled via a state machine encompassing task initialization, interactive manipulation, completion detection (through object-goal zone contact), abort handling, and data logging.

Figure 2: System overview—procedural generation, VR interaction, task evaluation, and logging pipeline.

Figure 3: Example of a procedurally generated environment with varied furniture, objects, and goal conditions.

Task completion triggers per-episode data saving that includes full state-action sequences, kinematic parameters, contextual metadata, and leaderboard rankings, supporting both behavioral analysis and reinforcement/imitation learning.

Data Collection Pipeline and Interaction Design

Demonstrations are gathered through an interactive pick-and-place game where users operate a simulated Unitree G1 humanoid to retrieve trash items and deposit them into a bin. The VR interface's clutch-mode mapping, incremental wrist rotation, and action filtering facilitate smooth, naturalistic control, broadening system accessibility to non-experts while preserving demonstration quality.

The pipeline achieves consistent and robust coverage of the robot's 44-dimensional state space and a 33-dimensional action space per frame, with all trajectories annotated by task context and assessed by completion validity and timing. Only successful (goal-attained) episodes are retained, ensuring that recorded behaviors are viable for downstream policy learning.

Empirical Analysis

Dataset Coverage and Quality

The resulting dataset demonstrates extensive coverage: over 17 episodes in the Easy setting, major body and arm subspaces attain 95–100% bin coverage, with slightly lower (expected) coverage for wrist rotation due to the minimal role of rotation in the specific task. Action analysis confirms effective engagement of both locomotor and manipulator subspaces.

The demonstration set encompasses a wide range of action sequence lengths (28.4 to 58.1 seconds), capturing natural variation in user strategy and skill level, thus engendering valuable heterogeneity for imitation learning.

Task Difficulty and Behavioral Diversity

Task difficulty is modulated via two experimental axes: initial object pose (upright/easy vs. inclined/hard) and goal bin size. Hard tasks yield non-overlapping and substantially longer task durations ($78.6 \pm 8.8$ s vs. $40.2 \pm 8.7$ s), with increased arm IK activity (82.5% vs. 72.5% of frames). State-action coverage analysis suggests that higher difficulty compels more extensive arm workspace exploration, beneficial for learning more robust manipulation policies.

Figure 4: Side-by-side game screenshots of Easy and Hard task conditions.

Procedural Generation and Scene Diversity

Procedurally generated environmental variation is ensured through a well-bounded randomized pipeline. Empirical evidence demonstrates meaningful spatial and object placement diversity, with robot base positions spanning a 5.8 m by 6.0 m region and coverage metrics exceeding 88% across all major subspaces. The process is robustly reproducible, with all randomness controlled by fixed seeds and episode-level parameter logging.

Implications and Future Directions

The demonstrated framework reduces expert burden and labor cost by reframing the data collection process as an accessible, engaging, and automated game-like activity. This model directly addresses long-standing limitations of conventional LfD pipelines, notably in terms of diversity, accessibility, and annotation overhead. The system is readily extensible to more complex, competitive, or multi-agent settings, and supports a straightforward path toward richer incentive mechanisms to sustain participation and data throughput.

From the perspective of AI policy learning, the collected demonstrations are well-suited for both behavior cloning and long-horizon imitation learning, and the integration of procedural generation fosters generalization over a broad range of tasks and environmental conditions. Additionally, the VR-based approach streamlines the transition from simulated to real robot domains by maximizing realism and user engagement, thereby supporting more effective sim-to-real transfer when integrated with high-fidelity simulators.

Conclusion

This work establishes an effective and extensible paradigm for embodied data collection by amalgamating gamification, procedural generation, VR-based teleoperation, and automatic structured logging within a unified pipeline. The system provides strong empirical evidence for broad state-action space coverage, covers a gradient of task complexity, and enables scalable generation of high-quality demonstration data with minimal expert intervention. The theoretical and practical implications point toward a generalizable and sustainable avenue for large-scale embodied intelligence research, with clear expansion pathways into richer task landscapes, incentive structures, and real-world deployment.