Cross-Hand Latent Representation for Vision-Language-Action Models

Abstract: Dexterous manipulation is essential for real-world robot autonomy, mirroring the central role of human hand coordination in daily activity. Humans rely on rich multimodal perception--vision, sound, and language-guided intent--to perform dexterous actions, motivating vision-based, language-conditioned manipulation systems for robots. However, training reliable vision-language-action (VLA) models for dexterous manipulation requires large-scale demonstrations across many robotic hands. In addition, as new dexterous embodiments appear rapidly, collecting data for each becomes costly and impractical, creating a need for scalable cross-embodiment learning. We introduce XL-VLA, a vision-language-action framework integrated with a unified latent action space shared across diverse dexterous hands. This embodiment-invariant latent space is directly pluggable into standard VLA architectures, enabling seamless cross-embodiment training and efficient reuse of both existing and newly collected data. Experimental results demonstrate that XL-VLA consistently outperforms baseline VLA models operating in raw joint spaces, establishing it as an effective solution for scalable cross-embodiment dexterous manipulation.

• The paper introduces an unsupervised latent action space (XL-VLA) that enables consistent dexterous manipulation across diverse robotic hands.
• It employs multi-headed VAE autoencoders with reconstruction, retargeting, and KL losses to align disparate hand kinematics in a unified latent manifold.
• Results show significant performance gains and robust zero-shot generalization, paving the way for scalable plug-and-play robotic design.

Cross-Hand Latent Representation for Vision-Language-Action Models: Technical Analysis

Introduction

Robotic dexterous manipulation poses significant challenges due to the heterogeneity and evolving design of robotic hands. While recent advancements in Vision-Language-Action (VLA) models have enabled robots to follow language instructions and perceive complex scenes, action modeling remains bottlenecked by the divergent morphology of robot effectors. The work "Cross-Hand Latent Representation for Vision-Language-Action Models" (2603.10158) introduces XL-VLA: a unified VLA pipeline with an embodiment-invariant latent action space, enabling multi-task, multi-embodiment dexterous manipulation with strong cross-embodiment generalization. This essay provides a detailed technical summary and critical analysis of the architecture, methodology, numerical results, and the broader theoretical and practical implications.

XL-VLA Architecture and Cross-Hand Latent Action Space

The central innovation is the introduction of a cross-embodiment latent action space, instantiated as a shared latent manifold trained via an unsupervised autoencoder framework. For each dexterous hand $h \in \mathcal{H}$, hand-specific encoders $E_h$ map joint trajectories to the shared latent space, and decoders $D_h$ reconstruct the trajectory from latent embeddings. These components are integrated with a VLA backbone (building on $\pi_0$), which is conditioned on image, language, and a latent-coded proprioceptive history for sequence modeling.

Figure 1: XL-VLA's model pipeline: Vision and language encoders, a latent action expert, and hand-specific latent encoders/decoders enable embodiment-invariant policy learning.

Critically, (1) the latent space is trained independently and frozen during downstream VLA fine-tuning, (2) only the encoder/decoder selection changes across embodiments—not the policy model—and (3) the framework supports plug-and-play inclusion of new dexterous hands, eliminating the need for per-hand model retraining.

Latent Space Construction and Training Objective

The latent space is realized using multi-headed VAE-style autoencoders. Three joint constraints regularize the latent space:

1. Reconstruction Loss ($L_1$): Ensures per-hand encoder--decoder pairs can faithfully reconstruct the original joint trajectory within each hand's kinematic manifold.
2. Retargeting Loss ($L_2$): Differential FK-based loss to enforce that latent-to-joint decoding yields geometrically consistent fingertip (thumb-finger pair) positions across different hands, despite mechanical and kinematic disparities.
3. KL Regularization ($L_3$): Constrains the latent distribution to be isotropic Gaussian, facilitating generative sampling and smoothness across the latent manifold.

Training uses synthetic joint-space sampling, enabling fully self-supervised alignment of disparate hands without requiring paired cross-hand demonstration trajectories.

Figure 2: Latent space pretraining pipeline, querying joint positions through encoders into a shared latent vector and reconstructing via decoders, with explicit placement of $L_1$ (reconstruction), $L_2$ (retargeting), and $L_3$ (KL) losses.

Experimental Protocol

A multi-hand dataset is collected via teleoperation (utilizing Apple Vision Pro and dedicated mocap for both xArm and humanoid platforms), encompassing 10 canonical manipulation tasks executed by the Ability Hand, Paxini DexH13, X-Hand1, and Inspire Hand. The XL-VLA and baselines are trained to predict 64-step joint action chunks at 20 Hz, conditioned on recent proprioception, language description, and visual observation.

Figure 4: Task visualizations exemplifying the diversity and complexity of the dexterous tasks used in evaluation.

Numerical Results and Zero-Shot Generalization

Cross-Embodiment Performance:

XL-VLA consistently surpasses the strong $\pi_0$ baseline (which nominally handles embodiment diversity via sequence length adaptation) in cross-hand multi-task success rate. Mean success rates improve from 0.32 ($\pi_0$) to 0.72 for XL-VLA, with improvements spanning from relatively simple hands (Ability, from 0.37 to 0.73) to mechanically divergent hands (XHand, from 0.29 to 0.70). Gains are particularly pronounced in tasks necessitating coordinated, dexterous motion (e.g., hand-over, box rearrangement).

Cross-System Scaling:

On both the xArm bimanual robot and the Unitree G1 humanoid, co-training with the shared latent space improves policy performance versus raw-action models, even with substantial variation in embodiment and sensory setup.

Figure 5: G1 cross-robot performance: Co-training with aligned latent actions outperforms raw-joint-space supervision on various tasks.

Zero-Shot Generalization:

A salient property of the latent space is demonstrated: XL-VLA achieves robust zero-shot success in transferring policies to unseen hand-task combinations without additional data or optimization. On held-out tasks with unseen hand-task pairs, XL-VLA outperforms both direct retargeting baselines and a strong cross-hand action-diffusion method (LAD [bauer2025latentdiff]) in mean replay and execution success.

Figure 6: Zero-shot generalization to unseen hand-task combinations reveals XL-VLA's capacity for transfer without explicit pairwise alignment.

Latent Space Quality and Ablation

Comprehensive ablations assess:

• The necessity of each loss term: removing reconstruction or retargeting elements degrades joint and tip reconstruction, diminishes cross-hand pinch alignment, and introduces latent-space discontinuity.
• Architectural choices: compact latent sizes ($L=32$) generate robust retargeting and smooth latent interpolations; excessively large latent spaces reduce embodiment-invariant structure.
• Compared to supervised latent-diffusion baselines, XL-VLA's unsupervised latent alignment yields superior or comparable cross-hand replay rates and stability, especially on fine-grained tasks.

  Figure 8: Latent trajectory visualization: consistent decoding of a grasping trajectory across all four hands, indicating invariant skill encoding.

Visualizations and Hardware Generalization

The framework is validated on four hardware platforms (Ability, Paxini, X-Hand1, Inspire), spanning mechanical and kinematic variety.

Figure 10: The four dexterous hands used in experiments—Ability, Inspire, X-Hand1, and Paxini DexH13—demonstrating embodiment diversity.

Figure 9: A diverse object set used for manipulation tasks, emphasizing the robustness of the policy to object and scene variability.

Implications and Future Outlook

Theoretical Implications:

This work substantiates the feasibility of learning compact, geometry-aligned, and embodiment-invariant latent action spaces for dexterous robotic manipulation via self-supervised, unpaired training. The latent space bridges morphology gaps without requiring explicit demonstration/retargeting data pairs, and supports smooth transfer, interpolation, and replay—even for hands with divergent kinematics.

Practical Implications:

Plug-and-play deployment of new robotic hands is viable, requiring only a new encoder/decoder pair into the established latent manifold. Empirical results indicate high data efficiency and robust transfer—a significant enabling property for scalable multi-embodiment robotic fleets and for modular robot design.

Contradictory & Highlighted Claims:

• The paper claims boldly that its unsupervised, geometry-based latent action alignment surpasses supervised approaches (e.g., LAD) in both replay consistency and real-world task performance.
• It demonstrates zero-shot successful skill transfer without cross-hand joint-space demonstration pairs—contrasting directly with most cross-embodiment policies that require paired data or explicit kinematic retargeting.

Outlook:

Future directions include extending latent alignment to broader classes of manipulators (e.g., legged robots or multi-limb morphologies), improving latent representation expressivity without loss of invariance, and integrating generative latent sampling for skill composition. Integration with foundation VLA models points toward scalable, generalist robotic manipulation in unstructured environments and continual adaptation to new hardware.

Conclusion

XL-VLA establishes a formal paradigm shift in vision-language-action modeling for dexterous robot manipulation via a unified, geometry-regularized latent action space. Its design enables cross-embodiment scalability, high policy success rates, robust zero-shot generalization, and seamless integration of new hardware—all validated in real-world teleoperated and autonomous experiments. The empirical and architectural results underscore the utility of unsupervised latent action spaces and suggest a path toward foundation models for general-purpose robotic manipulation across a rapidly evolving landscape of dexterous embodiments.