Dex4D: Task-Agnostic Point Track Policy for Sim-to-Real Dexterous Manipulation

Abstract: Learning generalist policies capable of accomplishing a plethora of everyday tasks remains an open challenge in dexterous manipulation. In particular, collecting large-scale manipulation data via real-world teleoperation is expensive and difficult to scale. While learning in simulation provides a feasible alternative, designing multiple task-specific environments and rewards for training is similarly challenging. We propose Dex4D, a framework that instead leverages simulation for learning task-agnostic dexterous skills that can be flexibly recomposed to perform diverse real-world manipulation tasks. Specifically, Dex4D learns a domain-agnostic 3D point track conditioned policy capable of manipulating any object to any desired pose. We train this 'Anypose-to-Anypose' policy in simulation across thousands of objects with diverse pose configurations, covering a broad space of robot-object interactions that can be composed at test time. At deployment, this policy can be zero-shot transferred to real-world tasks without finetuning, simply by prompting it with desired object-centric point tracks extracted from generated videos. During execution, Dex4D uses online point tracking for closed-loop perception and control. Extensive experiments in simulation and on real robots show that our method enables zero-shot deployment for diverse dexterous manipulation tasks and yields consistent improvements over prior baselines. Furthermore, we demonstrate strong generalization to novel objects, scene layouts, backgrounds, and trajectories, highlighting the robustness and scalability of the proposed framework.

• The paper introduces Dex4D, a task-agnostic policy that leverages object-centric point tracks and paired point encoding for zero-shot sim-to-real dexterous manipulation.
• The paper details a curriculum-based RL training with transformer-based world modeling and extensive domain randomization to enhance policy robustness and generalization.
• The paper reports over 16% improvement in success rates and significant gains against baselines in high-DoF manipulation tasks in both simulation and real-world experiments.

Dex4D: Task-Agnostic Point Track Policy for Sim-to-Real Dexterous Manipulation

Introduction

The paper "Dex4D: Task-Agnostic Point Track Policy for Sim-to-Real Dexterous Manipulation" (2602.15828) proposes a task-agnostic learning paradigm that addresses challenges in sim-to-real transfer for dexterous manipulation. By eschewing task- and language-specific policies, Dex4D leverages simulated training across thousands of object-pose configurations to learn an Anypose-to-Anypose (AP2AP) policy conditioned on object-centric, geometry-aware point tracks. At deployment, this policy can be directly zero-shot transferred to diverse real-world manipulation tasks by extracting and tracking object point trajectories from videos generated by large video models. Dex4D introduces key architectural and representational innovations—including a Paired Point Encoding for effective goal specification, extensive domain randomization, and an efficient closed-loop control system—yielding significant improvements in robustness, generalization, and task success rates over prior state-of-the-art methods.

Dex4D Framework and Policy Design

Dex4D's core insight is the decomposition of dexterous manipulation into two interacting components: (1) high-level open-world planning via video generation and 4D reconstruction that outputs object-centric point tracks, and (2) a robust low-level sim-to-real policy for closed-loop control.

Figure 1: Overview of the Dex4D teacher (RL, privileged information, dense points) and student (masked points, transformer) architectures, incorporating Paired Point Encoding to map partial observations to robot action and dynamics predictions.

Paired Point Encoding

A central contribution is Paired Point Encoding, a representation that preserves the correspondence between the current and target object points, crucial for accurate pose tracking. Rather than encoding current/target features independently (which discards critical geometric information under object rotations), Paired Point Encoding fuses corresponding 3D point pairs $\bm{q}^i_t = [\bm{p}^i_t \ \bm{\bar{p}}^i_t]$, yielding a permutation-invariant feature amenable to robust action conditioning.

Figure 2: Paired Point Encoding maintains pointwise correspondence and permutation invariance, outperforming decoupled or MLP-based encodings for policy learning.

Training Methodology

Dex4D applies a curriculum-based, large-scale RL (PPO) procedure for the teacher policy in simulation, using full privileged state and dense object points. The student policy is distilled from the teacher using DAgger, under partial and realistic observation perturbations (random point masking, noise, and occlusion), with a transformer-based action world model jointly predicting actions and next-state dynamics.

Figure 3: Mean reward trajectories during teacher RL curriculum, with Paired Point Encoding consistently surpassing alternative representations.

Ablations confirm the benefit of each pipeline component: Paired Point Encoding, use of self-attention in the student, and auxiliary world modeling for dynamics prediction all contribute materially to task performance.

Sim-to-Real Deployment

Dex4D closes the sim-to-real gap by leveraging advances in video generation and 4D scene understanding:

1. Given a natural language prompt and initial observation, a foundational video model (e.g., Wan2.6) generates a sequence of RGB frames depicting intended manipulation.
2. 2D point tracking (CoTracker3), segmentation (SAM2), and relative video depth estimation (Video Depth Anything) reconstruct the object-centric 3D point tracks.
3. At each control timestep, real-time online tracking updates current points, and Dex4D's policy receives the set of paired current/target points, proprioceptive state, and last action to predict the next action.
4. Execution advances along the generated trajectory, updating targets when sufficient geometric alignment is achieved.

This approach yields closed-loop feedback—which is proved to be critical for dexterous manipulation—substantially enhancing robustness over open-loop or heuristic planning baselines.

Experimental Results and Analysis

Dex4D achieves strong quantitative and qualitative results in both simulation and the real world. On six high-DoF tasks with significant variation in objects, scenes, and trajectories, Dex4D outperforms NovaFlow and its closed-loop extension by more than 16% in Success Rate and 10% in Task Progress (Table 1 in the paper).

Extensive ablation studies demonstrate:

• MLP-based or decoupled point encodings significantly reduce success rates.
• Removing self-attention or world modeling in the student architecture degrades policy performance.

  Figure 4: Overview of real-world dexterous manipulation setups. Each column depicts two task frames.

Real-world experiments on unseen objects and four tasks confirm that the learned policy maintains robustness under severe observation noise, point occlusion, and environment perturbations. Dex4D increases real-world success rate by over 22% compared to the strongest baseline, primarily due to its ability to maintain reactivity in both the hand and arm without reliance on fragile pose estimation or strictly motion-planned actions.

Figure 5: Baseline (left) succumbs to object dropping and post-grasp errors due to lack of hand feedback, while Dex4D (right) remains robustly in control even with noisy/occluded point tracks.

Theoretical and Practical Implications

Dex4D advances the state-of-the-art in generalizable sim-to-real dexterous manipulation by demonstrating that:

• Policies trained on large-scale, task-agnostic pose-to-pose transformations can perform zero-shot trajectory imitation in the real world when conditioned on video-extracted point tracks.
• Explicitly modeling current-target point correspondence in the goal representation is essential for high-DoF, geometry-centric control tasks.
• Video generation models, when coupled with robust 4D tracking and a properly conditioned closed-loop policy, can serve as high-level planners for manipulation, decoupling recognition and control and unlocking new synergies between foundational models and RL in robotics.

Practical limitations include the lack of incorporation of rich human grasp priors (due to data scarcity and embodiment mismatch), and the restriction to single-object manipulation. The primary observed failure mode is point tracking loss under occlusion or rapid motions in cluttered scenes.

Future Directions

Potential avenues include:

• Incorporating human demonstration priors from large-scale human-object-interaction datasets, especially with more anthropomorphic hands.
• Extending AP2AP policy conditioning and tracking to multi-object or articulated object manipulation.
• Augmenting state estimation with additional sensory modalities (e.g., tactile).
• Developing faster and more robust vision-based online tracking for improved sim-to-real reliability and scaling.

  Figure 6: Depiction of simulated tasks, highlighting the diversity in object types and required manipulation behaviors.

  

  Figure 7: Relative depth estimation yields more spatio-temporally stable tracking results than metric depths, which is crucial for clean sim-to-real transfer.

Conclusion

Dex4D demonstrates that a task-agnostic, point track-conditioned sim-to-real policy, robustly trained in simulation and leveraging closed-loop perception and novel goal representations, is highly effective for zero-shot dexterous manipulation across a wide spectrum of tasks and objects. The practical decoupling of high-level planning (via video foundation models and 4D tracking) from low-level control enables scalable and compositional reuse, suggesting a compelling path forward for generalizable, open-ended robot manipulation research.