Keypose-Conditioned Coordination Policy

• The paper introduces a hierarchical visuomotor framework that segments long-horizon bimanual tasks using keyposes as explicit subgoal milestones.
• It employs a consistency model to generate temporally coherent and physically aligned action trajectories, ensuring effective coordination and error recovery.
• Empirical results show state-of-the-art success rates in both simulated and real-world tasks, demonstrating improved efficiency and robustness.

A Keypose-Conditioned Coordination-Aware Consistency Policy is a class of hierarchical visuomotor policies, primarily for dual-arm (bimanual) robotic manipulation, that decomposes long-horizon multi-stage tasks into segments coordinated by keyposes—semantically meaningful milestones in the robot’s joint or end-effector state space. At each segment, a subgoal keypose provides an explicit target for a consistency-model-based action generator, which emits action trajectories guaranteed to be physically and temporally coherent given both history and the keypose. Coordination-aware modules manage synchronization between arms and stages, enforcing consistency and robustness across task phases. This architecture unifies the strengths of discrete keyframe planning, end-to-end continuous policy learning, and modern generative modeling, enabling efficient, accurate, and synchronized dual-arm manipulation in both simulated and real environments (Yu et al., 2024, Zhao et al., 24 Jun 2025, Xu et al., 17 Jan 2026, Yang et al., 24 Apr 2025).

1. Hierarchical Architecture and Keypose Conditioning

The hallmark of this policy is its explicit two-level hierarchy. The high-level keypose predictor, typically a neural network (such as a transformer atop a ResNet-18 or a 1D-U-Net), receives the environment observation (multi-view RGB, proprioception) and the most recently achieved keypose, and predicts the next subgoal keypose. These keyposes encode joint-space (or end-effector) configurations marking the boundaries of task sub-stages in bimanual operations, such as “pre-grasp,” “handover,” or “align-for-insertion” (Yu et al., 2024, Xu et al., 17 Jan 2026).

Keypose extraction from demonstration data employs both low-level heuristics—detecting gripper events, stalling, or spatial thresholds per arm—and high-level, potentially VLM-assisted, contact parsing. Coordination-awareness is achieved via merging rules: in phases requiring synchronization (such as handover or simultaneous contact), keyposes are aligned across both arms, while in independent phases, each arm proceeds on its own milestone schedule (Xu et al., 17 Jan 2026). This explicit segmentation is crucial for error recovery and confining the impact of sub-task failure or drift.

2. Consistency Model-Based Trajectory Generation

The low-level policy is a consistency model (CM), mapping the recent history of multi-modal observations and the forthcoming keypose to a sequence of actions over a short horizon. Let $o_t$ denote the current observation, $k_{j+1}$ the next predicted keypose, and $a_{t:t+H_a-1}$ a sequence of actions. The CM directly models the conditional distribution

$p(a_{t:t+H_a-1} \mid o_t, k_{j+1})$

and is trained to satisfy a self-consistency loss on noisy versions of action trajectories. This one-step, non-iterative predictor yields fast inference, which is important for real-time or dynamic tasks (Yu et al., 2024, Xu et al., 17 Jan 2026). Actions can be joint targets, end-effector positions/orientations, or gripper motions, conditioning on both ego-motion history and the explicit keypose subgoal.

Coordination is enforced since both arms’ actions are jointly conditioned on a common (merged, if needed) keypose, so the output is globally consistent for dual-arm tasks (Yu et al., 2024, Zhao et al., 24 Jun 2025).

3. Keypose Generation, Anchoring, and Affordance Integration

In advanced frameworks such as AnchorDP3, keypose anchors are directly linked to geometric affordances computed by a simulator-supervised segmentation prior (Zhao et al., 24 Jun 2025). Each anchor contains both arms’ joint angles and end-effector 3D pose (using a compact SE(3) representation) concisely packaged into a $32$-D vector. Keyposes are anchored to detected affordance centroids, and their orientations are set by aligning to the local surface normal and tangent frame, which exploits environmental structure for geometric consistency.

Full-state supervision (from simulator rendering) is harnessed to accurately segment and label critical objects and affordances. Sparse anchor-based dataset generation yields compact data representations, while the task-conditioned feature encoder integrates downsampled point clouds, augmenting spatial perception. The resulting learned embeddings condition the entire diffusion or consistency process, enabling robust multi-task learning (Zhao et al., 24 Jun 2025).

4. Mathematical Formulation and Losses

The core modeling techniques—in both consistency- and diffusion-based policies—share the following elements:

• A forward process adds Gaussian noise to all keyposes over $T$ steps, yielding smoothed versions $x_t$ at every denoising stage.
• Denoising is accomplished via a learned conditional U-Net (or transformer), outputting noise estimates or next-step actions, modulated by FiLM layers on task embeddings.
• The overall losses consist of:
  • A denoising score-matching loss:

  $$\mathcal{L}_\mathrm{diff} = \mathbb{E}_{t, x_0, \epsilon}\left\|\epsilon - \epsilon_\theta(x_t, t, f_k)\right\|_2^2$$ - An L2 keypose reconstruction loss (if directly reconstructing keyposes):

  $$\mathcal{L}_\mathrm{recon} = \mathbb{E} \left[ \sum_{i=1}^H \|a_{A_{k+i}} - \hat a_{A_{k+i}}\|_2^2 \right]$$ - A forward-kinematics-based consistency loss to enforce pose alignment between predicted joint angles and end-effector pose:

  $$\mathcal{L}_{\mathrm{cons}} = \sum_{i \in \{\text{left},\text{right}\}} \left( \|\mathrm{FK}_i(\hat q^{(i)}) - \hat p^{(i)}\|_2^2 + \alpha \|\mathrm{RotFK}_i(\hat q^{(i)}) - \hat R^{(i)}\|_F^2 \right)$$

as detailed in (Zhao et al., 24 Jun 2025). The total loss weights these components, ensuring geometric, temporal, and task-level consistency.

For hierarchical imitation frameworks (BiKC, BiKC+), additional cross-entropy terms are used for mode prediction, and Pseudo-Huber losses are used for softly robust regression.

5. Empirical Results and Comparative Analysis

Keypose-conditioned, coordination-aware consistency policies achieve state-of-the-art performance in both simulation and real-world bimanual manipulation benchmarks. Success rates on complex multi-stage tasks (e.g., bimanual transfer, peg-in-hole insertion, deformable packing, conveyor pick-and-place) consistently outperform diffusion policies and conventional conditional policies. For example, BiKC+ achieves per-substage simulated success rates between $96$–$98$%, with real-world overall successes of $50$–$100$% depending on task (Xu et al., 17 Jan 2026). AnchorDP3 attains a $98.7$% average success rate on the RoboTwin benchmark under extreme randomized conditions (Zhao et al., 24 Jun 2025). Ablation studies confirm the criticality of keypose conditioning (improved subtask reliability and recovery), the impact of the coordination-aware merging rules (enabling both synchronized and independent bimanual behaviors), and the necessity of the forward-kinematics consistency loss (improved physical accuracy and convergence).

Comparative results indicate that non-hierarchical or non-keypose-based methods, such as classic diffusion policies, suffer latency or stage transition failures, while methods lacking the coordination-aware merger exhibit reduced flexibility and robustness, especially in multi-modal, real-world tasks (Yu et al., 2024, Xu et al., 17 Jan 2026).

Framework	Policy Style	Keypose Conditioning	Consistency Tool	Coordination Merging	Sim Success (%)	Real-World Latency (ms)
BiKC+	Hierarchical	Yes	Consistency Model	VLM-assisted, Stage	98 (transfer)	~35
AnchorDP3	Anchor-based	Yes	Diffusion Policy	Geometric FK + loss	98.7 (RoboTwin)	n/a
BiKC	Hierarchical	Yes	Consistency Model	Heuristic Merging	95–98	~26
Baseline DP	Flat	Optional	Diffusion Policy	None	0–96	$>$100

6. Connections to Related Methodologies

Several parallel research threads inform this policy paradigm. Methods such as the PPI (keyPose and Pointflow Interface) use transformer architectures to predict both turning-point gripper poses (keyframes) and object-centric pointflows, then employ diffusion to generate coordinated action sequences, attending across all bimanual tokens and scene geometry. Here, keyframe tokens serve as explicit temporal landmarks, and relative 3D attention encodes spatial and manipulator consistency, realizing similar objectives of stage segmentation, collision avoidance, and robust, curved trajectory planning (Yang et al., 24 Apr 2025).

Consistency models—used as fast, non-iterative samplers—are increasingly preferred over standard denoising diffusion for high-frequency robotic control, given their low-latency, high-fidelity inference capacity. Incorporating affordance or segmentation priors (as in AnchorDP3) further unifies spatial reasoning with keypose anchoring, building upon multi-modal fusion in robotic perception (Zhao et al., 24 Jun 2025, Xu et al., 17 Jan 2026).

7. Impact and Open Directions

Keypose-conditioned, coordination-aware consistency policies have established a new state-of-the-art for bimanual imitation learning, particularly where multi-stage, coordinated, or dynamic tasks are required. Their explicit segmentation, robust substage synchronization, real-time inference, and capacity for multi-modal behavior generalization prove effective under both simulated and real robotic platforms. These frameworks extend naturally to vision-based, object-centric, and language-conditioned tasks, and can be further enhanced by more powerful foundation models and better VLM-assisted scene understanding.

A plausible implication is that this paradigm may be generalized for general multi-agent or multi-effector robotics, as the hierarchical decomposition, merger rules, and fast trajectory consistency provide scalable solutions to complex, flexible manipulation. Future research is likely to focus on unsupervised or reinforcement-based stage segmentation, transfer across robot morphologies, and integration with LLM-prior task decomposition (Xu et al., 17 Jan 2026, Yang et al., 24 Apr 2025, Zhao et al., 24 Jun 2025).