WorldSample: Closed-loop Real-robot RL with World Modelling

Abstract: Reinforcement learning (RL) can overcome the demonstration-coverage limitation of imitation learning (IL) by allowing robots to improve through trial-and-error interaction beyond the states observed in demonstrations. However, deploying RL on real robots remains constrained by high interaction costs, since each physical rollout is costly and reflects only one realized action-outcome path. To address this challenge, we propose WorldSample, a physically grounded data augmentation framework for real-robot RL that closes a real-synthetic loop between physical rollouts, world-model generation, and policy improvement. Grounded on real rollouts, WorldSample generates high-fidelity synthetic transitions through a post-trained world model, which greatly lowers the visual hallucination. Specifically, rather than simply using these transitions as real-world experience, WorldSample introduces Policy-Paced Learning (PPL) to regulate the training process through sample selection and scheduling, balancing useful augmentation against value overestimation and mitigating the hallucination-induced noise. Experiments on robot manipulation tasks involving contact-rich and precise tasks show that WorldSample improves policy success rate by 28% while reducing training steps by 59% compared with baselines. Furthermore, WorldSample improves world model visual fidelity by 19.4dB in PSNR and 0.47 in SSIM over demonstration-only post-training, validating the effectiveness of the real-synthetic loop for both policy and world model performance.

• The paper introduces WorldSample, a closed-loop framework that augments real robot rollouts with synthetic, physically grounded trajectories generated by an action-conditioned world model.
• The paper employs Policy-Paced Learning combining Q-aware sample selection and uncertainty-guided scheduling to mitigate risks like Q-value overestimation and distribution shifts.
• The paper demonstrates significant empirical gains, reducing real-world rollouts by 59% and improving success rates from 56% to up to 95% across complex manipulation tasks.

WorldSample: Physically Grounded Closed-Loop Real-Robot Reinforcement Learning with World Modelling

Introduction

"WorldSample: Closed-loop Real-robot RL with World Modelling" (2607.02431) addresses the persistent challenge of sample inefficiency in real-world reinforcement learning (RL) for robotics. The framework focuses on reducing the high cost and risk of robot-environment interactions by introducing a closed-loop system wherein a world model generates physically realistic synthetic rollouts to augment policy learning. The central innovation is the real-synthetic data loop, meticulously regulated by Policy-Paced Learning (PPL), which balances the benefits of data augmentation against risks such as Q-value overestimation and distribution shift due to model hallucinations.

Methodology

WorldSample Framework

WorldSample consists of two principal modules: the real-synthetic data loop and Policy-Paced Learning.

Real-Synthetic Data Loop: WorldSample extends each physical robot rollout into a family of synthetic, counterfactual trajectories using an action-conditioned world model. These trajectories are grounded by the statistics of actual robot data and generated by locally perturbing real action sequences, then simulating the physical consequences within the learned dynamics model. The generated rollouts are labeled by a task reward model, ensuring their relevance and physical plausibility.

Figure 1: The real-synthetic data loop in WorldSample expands each physical rollout into multiple grounded synthetic trajectories for efficient policy improvement.

Figure 2: Policy-Paced Learning coordinates the incorporation of synthetic data via Q-aware sample selection and uncertainty-guided scheduling.

Crucially, model fine-tuning is asynchronous: as new real data arrives, the world model updates in parallel, aligning the generative distribution with reality. This iterative, asynchronous adaptation maximizes throughput and keeps the synthetic data distribution anchored to reality.

Policy-Paced Learning

Naive inclusion of generated transitions is prone to destabilizing value estimation, leading to Q-value overestimation and poor generalization. To address this, WorldSample deploys PPL, which operates via two orthogonal mechanisms:

• Q-Aware Sample Selection: By splitting generated trajectories into successful and failed sets (via an independent reward model), PPL ensures balanced sampling. The selection criterion aligns the distribution of Q-values between synthetic and real data to within a prescribed margin, thereby suppressing value shift and hallucination artifacts.
• Uncertainty-Guided Scheduling: The effective incorporation rate of synthetic samples is controlled by the policy’s entropy on real robot states. During early training/current high uncertainty, synthetic samples contribute minimally, reducing risk of transferring world model-induced noise. Their influence grows as the policy becomes more certain, effectively minimizing synthetic bias throughout training.

  Figure 3: Uncertainty-guided scheduling adaptively tunes synthetic sample influence based on actor entropy, preventing early-stage bias.

  

  Figure 4: Q-aware sample selection maintains stable Q-value estimates by balancing the composition of synthetic rollouts.

Experimental Analysis

Evaluation Setup

Experiments are conducted on a 6-DoF Galaxea A1X arm equipped with dual RGB-D cameras covering side-view and wrist-mounted perspectives. Tasks include pushing, insertion, sorting, pick-and-place, and assembly, spanning a spectrum of requirement for visual grounding, contact-rich manipulation, and precision.

Figure 5: Task suite used for evaluating WorldSample, demonstrating manipulation tasks of varying complexity and sensory grounding.

Baselines include HIL-SERL (human-in-the-loop RL), VLAW (world-model-driven imitation learning), and WMPO (model-based RL without real interaction). All methods are provided with identical demonstration and hardware budgets.

Policy Improvement and Sample Efficiency

WorldSample yielded strong empirical gains across all tasks:

• Success rate: Average improvement from 56% (HIL-SERL) to 82%, with significant boosts on insertion and sorting (from 63% to 95% and 66% to 95%, respectively).
• Training efficiency: Number of real-world rollouts required for convergence reduced by 59% on average (from 56k to 23k).
• Training wall-clock: WorldSample achieved equivalent or better policy performance with 23% less wall-time and required fewer human interventions compared to RL and model-based baselines.

  Figure 6: Success and intervention rates by task; WorldSample improves efficiency and final performance particularly on precision and contact-rich tasks.

WorldSample converges faster and sustains higher learning rates early in training, reflecting effective exploitation of synthetic data without deteriorating policy reliability.

World Model Grounding and Prediction Quality

A core hypothesis is that physically grounded synthetic rollouts are higher quality and better aligned with true dynamics. Quantitative metrics confirm this:

• PSNR increased from 10.3 (demo-only) to 29.7 (demo+rollout).
• SSIM increased from 0.43 (demo-only) to 0.91 (demo+rollout).

Incorporating online rollout adaptation led to 19.4 dB boost in PSNR and 0.47 in SSIM over demonstration-only fine-tuning.

Figure 7: Rollout-adapted world models generate visually accurate, physically plausible cross-time predictions compared to real frames.

Figure 8: Fine-tuning with both demonstrations and online rollouts consistently enhances world model video fidelity.

Ablation and Mechanisms

PPL’s contribution is isolated via ablation. Removing uncertainty-guided scheduling or Q-aware sample selection degraded task success and increased intervention rates, particularly in complex tasks with high sensitivity to policy bias:

• Without scheduling: success dropped from 95% to 61%.
• Without Q-selection: success dropped to 76%.
• Without full PPL: required 3× more steps for equivalent performance.

  Figure 9: Ablations on Policy-Paced Learning reveal necessity of both scheduling and sample balancing for effective exploitation of synthetic rollouts.

Implications and Future Directions

WorldSample demonstrates that maintaining a closed, physically anchored data loop between real-world robot interaction and world model-based synthetic trajectory generation allows for dramatic improvement in sample efficiency and overall training efficacy in complex robotic RL settings. PPL’s dual-balancing design is critical in ensuring the safe, stable, and robust integration of synthetic data.

The results imply that world-model augmentation, when rigorously anchored to real sensorimotor experience and paced by policy uncertainty, can extend RL applicability to lower-resource, safety-critical, or high-variability domains that are otherwise out of reach for pure online or simulation-heavy techniques.

Key avenues for future research include:

• Task and Instance Generalization: Extending the real-synthetic loop to operate over variable tasks, scene distributions, and robotic platforms.
• Adaptive World Model Expansions: Integrating more sophisticated uncertainty quantification, online validation, or hierarchical world modelling.
• Real-Time and Scalable RL Deployment: Leveraging distributed, asynchronous synthetic data pipelines for ever-larger, more diverse task sets.

Conclusion

WorldSample defines a systematic approach toward bridging the gap between data-rich RL paradigms and resource-constrained real-world robot learning. By closing the feedback loop from reality to model and back, with careful pacing of synthetic knowledge incorporation, it achieves superior empirical performance and advances the practical deployment boundary of RL in physical agents (2607.02431).