WoW-Bench: Embodied AI Benchmark Suite

• WoW-Bench is a unified benchmark suite that evaluates embodied AI and fine-grained audio-language models on tasks involving physical, causal, and perceptual reasoning.
• It integrates large-scale, diverse datasets from robotic manipulation and marine mammal acoustics with detailed metrics and dual evaluation protocols.
• The benchmark challenges models with tasks in perception, predictive reasoning, planning, and generalization, highlighting critical gaps in current generative models.

WoW-Bench defines a class of evaluation benchmarks targeting embodied world models and fine-grained audio-LLMs, with a focus on physical, causal, and perceptual reasoning. Originating in response to the inadequacies of prior benchmarks—often limited to visual fidelity or shallow semantics—WoW-Bench protocols systematically test abilities across perception, prediction, planning, execution, and generalization, in both video-based and audio-based modalities. This suite operates in domains from robot manipulation video to out-of-domain marine mammal acoustics, combining large curated datasets, fine-grained metrics, and robust human/machine evaluation pipelines to illuminate the boundaries of current generative models and their grounding in physical reality (Chi et al., 26 Sep 2025, Fan et al., 7 Jan 2026, Kim et al., 28 Aug 2025).

1. Rationale and Conceptual Foundations

WoW-Bench emerged from the observation that existing benchmarks—such as FVD, SSIM, and PhysBench—do not jointly assess physical consistency, causal reasoning, instruction compliance, and generalization. For vision-based models, this led to the design of benchmarks grounded in robot interaction, where each sample comprises an initial image and a natural-language instruction, requiring the model to “imagine” and execute physically plausible action sequences. In audio, the motivation was to challenge Large Audio LLMs (LALMs) on low-level listening tasks (pitch, duration, temporal patterning) via rare, real-world sounds beyond the scope of traditional semantic classification.

2. Dataset Composition and Task Diversity

WoW-Bench in robotic world modeling consists of over 2 million trajectories sourced from 12 robot embodiments spanning both real and simulation environments. The final test split—606 video-prompt pairs (WoWBench) or 609 manipulation sequences (WoW-World-Eval)—sources ground-truth from open-data (RoboMIND, DROID), in-house robot operations, and style-transferred scenes. Tasks are partitioned to probe four core abilities: Perception (objects, relations, affordances), Predictive Reasoning (object permanence, collision, dynamics), Planning (long-horizon DAG decomposition, atomic actions), and Generalization (out-of-distribution artistic renderings).

In the audio domain, WoW-Bench assembles clips from the Watkins Marine Mammal Sound Database, sampling a broad frequency range (20 Hz to >20 kHz) and testing models on categorization and cognitively demanding tasks (Remember, Understand, Apply, Analyze, with and without distractors).

Task Breakdown in Embodied Video Benchmarks

Ability	Subtasks Included	Sample Count
Perception	Attributes, Spatial relations, Affordances	~299
Predictive Reasoning	Permanence, Collision, Multi-object	~327
Planning	DAG decomposition, Long-horizon	25
Generalization	OOD scenes, Style-transfers	20

3. Evaluation Protocols and Metrics

WoW-Bench deploys dual evaluation protocols—expert human raters and autonomous vision-LLMs (VLMs)—across grouped metric families.

• Visual Fidelity & Temporal Consistency: FVD, SSIM, PSNR, DreamSim; mask-guided DINOv3 embedding consistency.
• Semantic Correctness: Caption Score, Sequence Match Score, Execution Quality, leveraging GPT-4o and VLM alignment.
• Physical & Causal Reasoning: Metrics such as MED ($$\frac{1}{N} \sum_{i=1}^{N} \|x^{gen}_i - x^{gt}_i\|$$), Dynamic Time Warping, Fréchet Distance; Physical Common-Sense scores from specialized VLM QA probes.
• Planning Reasoning: DAG step comparison with metrics $R_k$, $R_s$, $P_k$, aggregated as $S_{\rm plan}$.
• Score Aggregation: Empirical CDF/z-score remapping, inverse intra-group metric correlation weightings, geometric means for group and overall scores.

Audio WoW-Bench computes micro-averaged MCQ accuracy and, when relevant, per-class precision, recall, and F1 as:

$$\mathrm{Accuracy} = \frac{1}{N} \sum_{i=1}^N \mathbf{1}(\hat y_i = y_i),\quad \mathrm{F1}_c = 2\,\frac{\mathrm{Precision}_c\,\mathrm{Recall}_c}{\mathrm{Precision}_c + \mathrm{Recall}_c}$$

Distractor-designed MCQ variants test true perceptual fidelity by preventing language or format shortcuts.

4. Comparative Analysis with Baselines

Evaluation against baselines reveals a pronounced advantage for models trained on embodied data. WoW achieves substantial gains over model Sora and others in physical causality metrics:

Model	PCS ↑	CDE ↓	OPA ↑
Sora	0.72	1.35 s	68.4%
Cosmos-P1	0.78	0.92 s	74.1%
WoW	0.85	0.47 s	82.7%

On WoW-World-Eval, closed-source and open-source models struggle with long-horizon planning (best results below 18/100). Real-world execution via GC-IDM highlights a significant gap: most models fail ($\sim$0%) while WoW variants reach up to 40.74% execution success, underscoring the challenge of translating visual rollouts into executable actions (Fan et al., 7 Jan 2026, Chi et al., 26 Sep 2025).

In audio, top commercial models (Gemini 2.5 Flash) attain 47% accuracy (vs. human ≈70%) on Cognition but near-random performance on Perception, particularly on distractor tasks, evidencing shallow acoustic feature use and reliance on format heuristics (Kim et al., 28 Aug 2025).

5. Diagnostic Findings and Failure Modes

WoW-Bench reveals several strengths:

• Closed-loop SOPHIA + FM-IDM architectures excel in instruction understanding (96.5%) and adherence to physical laws (80.2%).
• Dynamic Critic + Refiner loops effectively correct physical hallucinations, substantiating VLM-guided refinement strategies.
• IDM translation achieves 94.5% success on simple, 75.2% on medium robot tasks.

However, persistent weaknesses include stochastic instabilities (collisions, penetration artifacts), suboptimal performance on hard physical tasks (dual-arm or fluid dynamics), and erratic results on distractor audio MCQs, indicating reliance on surface cues rather than true grounded understanding.

6. Implications for Embodied AI and Future Development

WoW-Bench provides evidence for the necessity of large-scale, causally rich embodied datasets in fostering physical intuition and generalization in world models. A notable implication is that metrics aligned with human preference (Pearson $r$ > 0.93) support reliable Turing Test-style evaluation. The standardized protocols offer a foundation for benchmarking new models, exposing multidimensional failure modes across perception, planning, causality, and execution.

Key open challenges include scaling benchmarks to multi-agent settings, ameliorating residual hallucinations via explicit physics or neuro-symbolic techniques, and extending coverage to domains such as deformable manipulation and transparent media.

In audio-language modeling, future benchmarking must address architectural limitations in sensing fine acoustic features, suggest pretraining on bioacoustic corpora, and promote multi-task objectives combining low-level listening losses and interpretation.

The ongoing open-sourcing effort (https://wow-world-model.github.io/) aims to catalyze research in physically grounded, instruction-aware world models—with WoW-Bench establishing itself as a unified, robust, and ecologically valid standard for the evaluation of embodied AI (Chi et al., 26 Sep 2025, Fan et al., 7 Jan 2026, Kim et al., 28 Aug 2025).