ChronoFlow-Policy: Unifying Past-Current-Future Interaction Flow in Visuomotor Policy Learning

Abstract: Visual signals play a crucial role in policy learning by enabling models to capture object motion and interaction dynamics. Just as humans reason about actions using both past experience and anticipated outcomes, effective policies should integrate past interactions with future predictions. However, existing visuomotor policies typically model either historical context or future dynamics in isolation, lacking a unified temporal representation of interaction dynamics. In this work, we introduce ChronoFlow, a temporally unified representation that captures past, current, and future interaction dynamics through sparse 3D keypoints of both objects and the gripper. Based on this representation, we propose ChronoFlow-Policy, a diffusion-based visuomotor policy that jointly learns ChronoFlow and action sequences through a co-training objective. Experiments on 14 simulated tasks and 5 real-world manipulation tasks demonstrate that ChronoFlow-Policy consistently outperforms strong diffusion-policy baselines and improves robustness in long-horizon and non-Markovian manipulation scenarios. Our project page is available at https://the-kamisato-sii.github.io/ChronoFlow-Policy-project-page/.

• The paper presents ChronoFlow-Policy, a unified diffusion-based visuomotor policy that integrates historical and future keypoint flows for enhanced interaction modeling.
• It demonstrates superior performance across benchmarks, achieving up to 93% stage-wise success in challenging real-world and simulated tasks.
• Ablation studies confirm that ChronoFlow supervision and spatial augmentation are critical for managing non-Markovian dynamics and deformable object manipulation.

ChronoFlow-Policy: Unified Temporal Interaction Flow for Visuomotor Policy Learning

Motivation and Temporal Interaction Representation

The paper "ChronoFlow-Policy: Unifying Past-Current-Future Interaction Flow in Visuomotor Policy Learning" (2606.31493) introduces ChronoFlow, a temporally unified representation that leverages sparse 3D keypoints of objects and grippers to encode interaction-centric dynamics across past, current, and future timeframes. Unlike prior visuomotor policies, which independently model future or historical context, ChronoFlow integrates both temporal directions in a shared spatial frame. This representation is explicitly supported by advances in 3D point tracking (e.g., TAPIP3D), and is designed to filter redundant scene motion, enhancing focus on task-relevant interactions.

Figure 1: Overview of ChronoFlow-Policy. Top: Illustration of ChronoFlow: unified past-current-future object-gripper keypoint flows. Bottom: The network architecture.

ChronoFlow-Policy: Diffusion-Based Policy Coupled with Interaction Flow

ChronoFlow-Policy is proposed as a diffusion-based visuomotor policy that co-trains ChronoFlow prediction alongside action generation, aligning shared latent representations with temporally grounded manipulation dynamics. The policy factorizes action prediction as conditional on both historical ChronoFlow trajectory and present observation, predicting future keypoint flows and subsequently decoding actions. The backbone employs either Unet1D or Diffusion Transformer architectures, leveraging denoising diffusion probabilistic modeling for trajectory refinement.

Training is supervised by two objectives:

• ChronoFlow loss (standard diffusion noise prediction) enforces spatiotemporal modeling of interaction-centric trajectories.
• Action loss (MSE) aligns partially-denoised hidden states and reconstructed interaction flows with ground-truth actions.

Inference leverages asynchronously tracked historical ChronoFlow and proceeds via iterative denoising, followed by decoding of both future interaction trajectories and actions.

Experimental Evaluation and Numerical Results

ChronoFlow-Policy is evaluated across 14 simulated tasks (MetaWorld, RoboTwin 2.0) and 5 real-world manipulation tasks encompassing long-horizon, deformable-object, and non-Markovian conditions. The method is compared with action-only diffusion policies (DP3, RISE), history-aware variants (HistRISE), and future-dynamics models (MBA, 3D-FDP).

Simulation Benchmarks:

ChronoFlow-Policy achieves marked improvements in task success rates compared to all baselines. On MetaWorld, CFP (Unet) attains an average success rate of 72% and CFP (DiT) 70%, while DP3 and MBA reach 30% and 47% respectively. In tasks requiring precise contact and insertion (Hand-Insert, Sweep Into), gains are particularly significant, with CFP outperforming object-pose and scene-flow models due to its interaction-centric representation.

Real-World Tasks:

In long-horizon and memory-intensive tasks (Swap-Easy, Swap-Hard, Prepare Breakfast), ChronoFlow-Policy maintains high stage-wise completion rates (up to 93% across stages), while action-only and object-pose baselines degrade sharply in later stages due to lack of explicit historical modeling. On deformable object manipulation (Fold Towel), ChronoFlow-Policy achieves 87% success in both folding stages, surpassing object-pose baselines which are unable to robustly represent deformations.

Figure 2: Learning curves on simulation benchmarks. CFP (Unet) generally converges faster and achieves higher or comparable final success rates than DP3 across representative tasks, demonstrating the effectiveness of ChronoFlow supervision.

Ablation Analysis

Ablation studies validate critical ChronoFlow design choices: removing ChronoFlow supervision, object flows, or stochastic keypoint subsampling severely impairs performance, especially on deformable manipulation. Spatial augmentation with transformation jittering is essential for real-world robustness, with disabling such augmentation resulting in catastrophic failure rates. Object flows dominate the signal for tracking deformable geometry, while gripper flows enhance fine-grained contact encoding.

Theoretical and Practical Implications

ChronoFlow introduces a compact, interaction-centric abstraction of temporal dynamics that bridges the gap between future prediction and history integration in policy learning. This design addresses non-Markovian dependencies, stage-conditioned planning, and complex interaction scenarios (including soft-body manipulation), yielding improved robustness and sample efficiency. The co-training framework ensures that learned latent spaces encode consistent physical relationships rather than image-level patterns or action labels in isolation.

Practically, this enables more reliable deployment of visuomotor policies in real-world settings, where occlusion, deformability, and asynchronous observations are prevalent. Theoretically, ChronoFlow highlights a principled pathway for embodied AI: prioritizing flow-based representations that capture both the causal structure and progression of interaction events.

Future Directions

The paradigm established by ChronoFlow-Policy suggests several promising avenues:

• Extension to Multi-Agent/Tool Interactions: Generalization of ChronoFlow to multi-entity interaction with tools or collaborative agents.
• Integration with Semantic Spatial Priors: Augmenting ChronoFlow with explicit semantic segmentation or affordance modeling.
• Uncertainty and Occlusion Robustness: Incorporating uncertainty-aware tracking and adaptive keypoint sampling for handling tracking failures, occlusions, and missing data.
• Foundation Model Alignment: Coupling ChronoFlow with vision-language-action models for scalable, generalizable policy pretraining.

Conclusion

ChronoFlow-Policy establishes a unified framework for temporally consistent, interaction-centric policy learning, demonstrating substantial empirical gains over prior action-only and object-centric architectures. Its co-training regime and sparse flow representation contribute to robust temporal reasoning in both simulated and real-world settings, especially for non-Markovian and deformable manipulation. The approach motivates further research into flow-based representation learning for embodied AI, targeting scalable, generalizable solutions that encode causal interaction structure at all temporal horizons.