WALA Learning Executable Latent Actions from Action-Labeled Demonstrations and Action-Free Videos

Abstract: Generalizable robot policies typically rely on action-labeled robot demonstrations, which are expensive to collect and difficult to scale. In contrast, large-scale human and robot videos contain rich physical interactions but often lack executable robot action labels. We present WALA, a framework for learning executable latent actions from both action-labeled demonstrations and action-free videos. WALA first pretrains a semantic-geometric latent action model from videos by modeling the evolution between current observations and sparsely sampled future observations. Instead of reconstructing raw pixels, WALA predicts future deltas in the DINOv3 feature space and dense depth space, preserving task-relevant semantic and geometric structure while reducing sensitivity to appearance details. During policy training, the pretrained encoder provides stable latent action targets, and the decoder serves as a trainable latent world model. The latent actions generated by the vision-language backbone are jointly supervised by robot action prediction, latent action target matching, and future dynamics prediction. This enables action-labeled demonstrations to provide executable control supervision, while action-free videos contribute dynamics supervision without requiring robot action annotations. Experiments show that WALA achieves strong performance on RoboTwin, sets a new state-of-the-art result on RoboCasa with 75.2% average success, and improves both policy performance and generalization in real-world manipulation tasks.

• The paper introduces WALA, a framework that learns executable latent actions using both action-labeled and action-free videos to enhance robot control.
• It leverages semantic-geometric pretraining with DINOv3 features and depth maps to align scene evolution with manipulation dynamics.
• Empirical results on benchmarks and real-world tasks show significant improvements, including effective zero-shot transfer performance.

Learning Executable Latent Actions from Action-Labeled Demonstrations and Action-Free Videos: The WALA Framework

Problem Statement and Motivation

WALA addresses a critical bottleneck in generalizable robot policy learning: reliance on action-annotated robot demonstrations, which are expensive and insufficient for covering the breadth of real-world physical interactions. Previous vision-language-action models (VLAs) and world action models (WAMs) have advanced manipulation policies by integrating vision, language, and robot action data (Brohan et al., 2022, Brohan et al., 2023, Kim et al., 2024), but neither class leverages large-scale action-free videos for precise control or dynamics supervision. WALA proposes a novel scheme—jointly learning executable latent actions from both action-labeled and action-free videos by modeling a bridge between scene dynamics and robot actions.

Semantic-Geometric Latent Action Model Pretraining

WALA's pretraining phase operates in semantic and geometric latent spaces using DINOv3 features and dense depth maps to capture scene evolution (Figure 1). This model incorporates a frozen DINOv3 encoder for patch-wise semantic extraction and a depth estimator for spatial geometry, avoiding raw pixel reconstruction and suppressing task-irrelevant appearance noise.

Figure 1: Overview of semantic-geometric latent action model pretraining, extracting DINOv3 features and dense depth maps to learn action-relevant deltas.

Latent action tokens are encoded based on observed relative semantic and geometric deltas from sparsely sampled future frames. The decoder predicts future DINOv3 feature changes and depth deltas, supervised via $\ell_1$, cosine, and depth gradient loss terms. This ensures that the latent actions reflect action-induced transitions in tasks, not static visual detail.

Policy Training with Latent World Supervision

During policy training (Figure 2), the frozen latent action encoder provides stable target representations, while the world model decoder is integrated as a trainable component. The vision-language backbone (instantiated with Qwen3-VL-4B) processes multi-view observations, instructions, robot states, and action queries. Policy-generated latent actions are supervised through three distinct losses:

• Robot action prediction: maintaining executable control alignment
• Latent action target matching: aligning with pretrained latent action targets
• Future dynamics prediction: ensuring consistency with future scene evolution

Action-free videos participate by providing latent action and dynamics supervision without robot action labels. This design tightly binds WALA's latent actions to both executable control and scene dynamics.

Figure 2: Policy training in WALA using a frozen encoder for latent action targets and world model decoder as dynamics supervisor; inference omits all pretraining components.

Empirical Evaluation and Benchmark Results

WALA's performance was validated across major benchmarks: RoboTwin 2.0 and RoboCasa-GR1-Tabletop.

• On RoboTwin 2.0, WALA achieves 90.6% under Clean settings and 92.8% under Random, outperforming recent VLA and WAM-based baselines (2607.11397).
• On RoboCasa-GR1-Tabletop, WALA reaches an average success rate of 75.2%, surpassing DIAL (70.2%) and all reported alternatives.

Qualitative analysis demonstrates that WALA's predictions in DINOv3 feature and depth spaces robustly mirror task-relevant semantic and spatial outcomes, verifying that latent actions are grounded in manipulative dynamics (Figure 3).

Figure 3: Visualization of future semantic and geometric predictions, showing predicted DINOv3 features and depth changes closely match ground truth, preserving action-induced structure.

Data Scaling Studies: Action-Free Video Utility

WALA leverages action-free videos for policy scaling (Figure 4). With only 10% action-labeled data, WALA achieves 53.9% success; supplementing with action-free video increases performance up to 67.8%. Empirically, action-free data injects meaningful dynamics supervision that compensates for limited robot action labels.

Figure 4: Success rate scaling as action-labeled data and action-free videos are increased; action-free videos consistently boost performance.

Analysis of Latent Action Space

Retrieval experiments show that WALA's latent actions cluster transitions with similar manipulation primitives—grasping, placing, releasing—across variations in object, layout, and viewpoint (Figure 5). Attention maps demonstrate that the encoder focuses on hands, object contact, and manipulated regions (Figure 6).

Figure 5: Latent action retrieval visualizes clustering by manipulation semantics, invariant to scene or viewpoint changes.

Figure 6: Attention visualization indicates encoder focuses on contact and interaction regions during action token extraction.

Real-World Manipulation and Zero-Shot Transfer

WALA was evaluated on four real-world multi-task manipulation settings (Figure 7), achieving significant gains over strong baselines ($\pi_0$, $\pi_{0.5}$). Notably, combining 50 robot demonstrations with 400 action-free human videos yields near parity (74.2%) with the performance attained via 200 robot demonstrations (75.0%).

Figure 7: Real-world manipulation tasks evaluated in multi-task policy; sampled rollouts show robust execution by WALA.

WALA also demonstrated zero-shot transfer: action-free egocentric human videos enabled successful completion of a novel bread pick-and-place task without any robot demonstrations, increasing success from 3/10 (baseline) to 9/10 (with human videos) (Figure 8).

Figure 8: Zero-shot transfer: action-free human video supervision enables novel manipulation task execution by WALA policy.

Implications, Limitations, and Future Directions

From a theoretical standpoint, WALA advances the interface between action-labeled robot control and large-scale action-free dynamics data, demonstrating that semantically and geometrically grounded latent actions can serve as executable controllers. Practically, WALA's architecture enables robust policy training, rapid deployment, and generalization across tasks by leveraging video-scale data without incurring inference overhead.

Strong empirical results indicate that future dynamics (in DINOv3 and depth space) provide a direct, effective training signal for manipulation policies, even in the absence of action labels. Action-free human videos were shown to compensate for limited robot demonstrations and support zero-shot generalization.

Limitations include the finite scope of LAM pretraining data and limited zero-shot evaluation. Future directions include scaling LAM pretraining on broader video datasets, expanding zero-shot study to progressively more difficult tasks, and optimizing the latent action space for further cross-embodiment transfer.

Conclusion

WALA introduces a methodology for learning executable latent actions from both labeled robot demonstrations and unlabeled, action-free videos, connecting scene dynamics supervision to robot policy learning via a semantic-geometric latent action space. Experiments on simulated and real-world settings establish WALA as a practically efficient and theoretically grounded approach, with implications for scaling manipulation policy learning and broadening the horizons of zero-shot, task-centric robot control.