GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation

Abstract: Evaluating embodied robot foundation models remains a critical bottleneck; unlike LLMs efficiently assessed via digital benchmarks, robotic policies require slow, costly real-world rollouts limited by hardware and human supervision, which has driven interest in world models as surrogate policy evaluators, yet the key properties that make a world model reliable for policy assessment remain poorly understood. This work presents a systematic study of world models for robotic policy evaluation and introduces WMBench, a benchmark constructed from real-robot teleoperation data and matched policy rollouts covering diverse manipulation tasks to enable controlled comparisons across model families, action encodings, rollout horizons, and evaluation metrics. Using WMBench, we analyze 7 video world models, 4 action representation schemes, and over 324,000 simulated policy rollouts paired with real robot executions, further enriching our analysis with large-scale community submissions from the CVPR 2026 GigaBrain Challenge, curated synthetic trajectories, and a training videos spanning more than 12,000 hours. Our experiments deliver three core insights: evaluator quality is dominated by long-horizon, action-faithful rollout consistency rather than short-term visual realism; pretraining gains stem not only from data scale but from balancing general world knowledge with robot-specific controllability; and architectural choices including action encoding, memory design, and evaluator-focused post-training strongly determine alignment with real-world robot behavior. Drawing on these results, we derive a practical design roadmap and realize it in \textit{GigaWorld-1}, a world model specially optimized for policy evaluation, and we fully release our code, models, datasets, and toolkits to advance scalable evaluation research for embodied foundation models.

• The paper demonstrates that long-horizon, action-faithful rollouts enable reliable policy evaluation using learned world models.
• It introduces the WMBench benchmark with metrics like Subject Consistency and Perspectivity (up to ρ=0.88) to ensure evaluator alignment.
• The system uses a scalable VLM-based pipeline with explicit action control and memory modules to mitigate error accumulation in rollouts.

GigaWorld-1: A Systematic Roadmap for World Models in Robot Policy Evaluation

Introduction and Motivation

Robotic policy evaluation fundamentally diverges from the paradigms established by large language or vision models. While LLMs are evaluated on efficient digital benchmarks, embodied robot policies require repeated real-world rollouts, incurring high overhead and limiting the rate and scale of evaluation. This bottleneck has intensified with the rise of large-scale multi-modal robot foundation models, motivating the exploration of learned world models as surrogate policy evaluators. However, the specific properties that ensure a world model's reliability for policy assessment were largely unresolved. "GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation" (2607.02642) addresses this gap through a rigorous empirical and methodological analysis, culminating in a scalable, open-source framework—GigaWorld-1—for evaluator-centric world model design.

Figure 1: Large-scale comparative analysis of world-model-simulated rollouts, video world models, and action representation paradigms in GigaWorld-1.

Framing World Models as Evaluators

The core contribution of the paper is treating world model policy evaluation as a first-class research objective, independent of standard video generation or simulation fidelity. The system is formalized as a recurrent closed-loop between the robotic policy and the learned world model: the policy produces an action conditioned on observations, and the world model predicts future observations based on this action, iteratively producing a rollout trajectory.

Figure 2: Conceptualization of a world model serving as a policy evaluator—requiring fidelity in both visual quality and accurate, action-faithful rollouts.

Critically, reliable evaluation hinges not on short-horizon video realism, but on the preservation of action-faithful causality over long-horizon rollouts and calibration with real-world policy outcomes. The primary alignment metric is the correlation of empirical success rates between world-model-generated and real-world rollouts, emphasizing agreement in policy-level conclusions across conditions.

WMBench: Benchmarking the Policy Evaluation Task

To systematically evaluate evaluator fidelity, WMBench is introduced—constructed from extensive real-robot teleoperation and policy rollout data, with rigorous data cleaning and annotation, supporting multi-view and multi-task robotic manipulation.

Figure 3: WMBench's evaluation protocol: from real-world policy collection to world model training, closed-loop rollout, and comprehensive alignment assessment.

The benchmark features a four-level ordinal World Model Evaluator Score (WMES) aligned with both outcome faithfulness and visual/motional fidelity, and is complemented by a taxonomy of automatic metrics spanning frame quality, geometry, semantics, interaction, and long-horizon stability.

Empirical Insights: Metric Predictivity and Design

Large-scale evaluation on 324,000 annotated rollouts yields several key findings:

• Visual and geometric fidelity metrics—such as Subject Consistency, Perspectivity, and Instruction Following—are the most reliable predictors of evaluator alignment, with group-level correlations up to $\rho=0.88$. Semantic labels alone are insufficient for robust evaluation.
• Metrics emphasizing appearance stability (e.g., Background Consistency, Photometric Consistency) can be misleading, as trivial static rollouts can score highly despite being action-ignorant.

  Figure 4: Metric group correlations illustrating the dominance of visual fidelity and geometry in outcome prediction.

  

  Figure 5: Pearson correlation heatmap—subject consistency, geometric, and semantic fidelity most strongly predict WMES, while appearance stability is unreliable.

• Evaluator quality must be assessed over long-horizon autoregressive rollouts, as models with plausible short-horizon outputs degrade under iterative feedback, leading to compounding drift and misalignment.

VLM-Based Evaluation Pipeline

A scalable, VLM-assisted evaluator (LoRA-tuned Qwen3-VL) is deployed to predict WMES and provide structured rationale, achieving over 87% exact agreement and 99% adjacent agreement with human judges, enabling cost-effective scaling of policy evaluation without sacrificing reliability.

Figure 6: Qwen3-VL-based rollout evaluator: structured aspect-level and outcome score predictions from multi-view video rollouts.

Impact of Pretraining, Data Composition, and Architecture

Pretraining and Data Ablation

• Transferable physical priors and compatibility with robot-conditioned rollout are more critical than model size or generic data scale. Models with domain-aligned pretraining (e.g., Cosmos, Wan) surpass larger generic video backbones lacking suitable priors.
• Mixing broad physical videos into the training corpus yields the best trade-off for generalization and evaluator fidelity. This blend preserves task-relevant robot biases from in-domain data (GigaData) and the world knowledge and dynamics from internet-scale physical videos.
• Narrowly specialized robot demonstrations enhance embodiment fidelity but may worsen generalization and structure-sensitive metrics, indicating an embeddability–generalization trade-off.

  Figure 7: Multi-source data pipeline—filtering and balancing to compose a diverse, high-quality training corpus for world-model learning.

Architecture: Action Control and Memory

• Spatially aligned, explicit action control is essential for trajectory accuracy and rollouts faithfully following policy intent. Channel-wise concatenation of action representations (e.g., control maps, pose maps) outperforms token-based or attention-side control.
• Long-horizon memory modules, featuring hierarchical history anchoring, are pivotal for preventing spatio-temporal drift. These components stably anchor the generated rollouts to consistent scene identity and dynamics, mitigating error accumulation over time.

  Figure 8: The GigaWorld-1 architecture—autoregressive diffusion-transformer world generator with explicit, temporally aligned control injection and persistent hierarchical memory.

Spherical Linear Prompt Interpolation

• Spherically interpolated prompt transitions, rather than abrupt switching, lead to smoother task-phase and behavior transitions during long-horizon rollouts. This method prevents jerkiness and semantic inconsistencies during multi-stage manipulation.

  Figure 9: Prompt transition via spherical interpolation—smooth, continuous semantic transitions across task phases during autoregressive rollouts.

Training and System Optimization

A progressive curriculum covers robot-domain pretraining, autoregressive rollout learning, optional scene adaptation, and few-step DMD2 distillation—enabling practical few-step inference with negligible compromise to generation fidelity. System-level optimizations (SageAttention, custom Triton kernels, memory/parallelism techniques) yield up to 36× speedup in inference throughput, crucial for scaling policy evaluation pipelines.

Figure 10: Multi-stage training pipeline from foundation pretraining to distillation, with modular, pluggable adaptation and optimization stages.

Quantitative Results and Empirical Validation

GigaWorld-1-Plus and GigaWorld-1-Nano achieve the highest scores on all relevant evaluator metrics in head-to-head comparison with established open-source backbones. Compared to the strongest robot/auto and generic baselines, GigaWorld-1-Plus improves core evaluator-alignment metrics by 11.6%–14.9%. Gains are concentrated in JEPA Similarity, Semantic Alignment, and Trajectory Accuracy—metrics tightly coupled to action-faithful policy evaluation.

Figure 11: Per-metric model architecture comparison—GigaWorld-1 variants score highest on evaluator-alignment, semantic, and trajectory metrics.

Long-horizon rollout experiments demonstrate GigaWorld-1's superior resistance to error accumulation and visual/semantic drift, outperforming baselines across PSNR, FID, and FVD over 40-second rollouts.

Figure 12: Long-horizon rollout fidelity dynamics—GigaWorld-1 sustains higher PSNR, lower FID/FVD over extended sequences.

Figure 13: Comparative visual rollout—GigaWorld-1 maintains consistent identity and physics, avoiding collapse observed in generic models.

Ablations confirm the ablation impact of explicit memory and interpolated prompts on scene stability and semantic transition smoothness.

Figure 14: Memory and prompt interpolation ablation—combination yields optimal semantic and geometric consistency.

Policy-Consistent Evaluation: Closed-Loop and OOD Robustness

GigaWorld-1 demonstrates strong OOD generalization, accurately simulating physical outcomes under object, background, and action distribution shifts. Most significantly, in closed-loop policy evaluation, success rates under world-model rollouts closely match real-robot executions, markedly outperforming challenge baselines (reduced optimistic/overconfidence bias).

Figure 15: OOD generalization—robust preservation of causal physical dynamics and outcome under diverse shifts.

Figure 16: Task-level success-rate alignment—GigaWorld-1 closely tracks the real-world diagonal; baselines frequently overestimate.

Figure 17: Subtask-level success-rate bias: GigaWorld-1 achieves closer agreement and lower error than competitors.

Theoretical and Practical Implications

• The study reorients the definition of world model fidelity from visual realism to policy-level outcome alignment under long-horizon, action-conditioned rollouts.
• Evaluator-centric design must balance broad world knowledge, robot-specific controllability, and persistent causal grounding.
• Relying on misleading metrics or static video generation for evaluator selection can result in severe mis-calibration and failure to recognize poor policies.
• The public release of code, trained models, and benchmark tools establishes WMBench and GigaWorld-1 as critical infrastructure for scalable, community-driven robot policy evaluation.
• The paper's insights inform both simulator design (e.g., new benchmarks beyond WMBench, 3D/physically-grounded world models) and future research directions in uncertainty estimation, richer reward modeling, and omnimodal embodied world models.

Conclusion

GigaWorld-1 sets a new standard for world-model-based robotic policy evaluation by unifying empirical rigor, scalable benchmarking, architectural design, and system efficiency. Its open-source release catalyzes follow-up work in embodied foundation model evaluation and simulator research. The paper’s findings underscore that long-horizon, action-faithful consistency—not superficial visual metrics or model scale—is central to reliable surrogate policy evaluation. Building on these insights, future advances may integrate richer modalities, certified uncertainty, and even counterfactual evaluation, bringing embodied AI foundation models closer to rapid, trustworthy, and scalable iteration.