GR-RL: Going Dexterous and Precise for Long-Horizon Robotic Manipulation

Abstract: We present GR-RL, a robotic learning framework that turns a generalist vision-language-action (VLA) policy into a highly capable specialist for long-horizon dexterous manipulation. Assuming the optimality of human demonstrations is core to existing VLA policies. However, we claim that in highly dexterous and precise manipulation tasks, human demonstrations are noisy and suboptimal. GR-RL proposes a multi-stage training pipeline that filters, augments, and reinforces the demonstrations by reinforcement learning. First, GR-RL learns a vision-language-conditioned task progress, filters the demonstration trajectories, and only keeps the transitions that contribute positively to the progress. Specifically, we show that by directly applying offline RL with sparse reward, the resulting $Q$-values can be treated as a robust progress function. Next, we introduce morphological symmetry augmentation that greatly improves the generalization and performance of GR-RL. Lastly, to better align the VLA policy with its deployment behaviors for high-precision control, we perform online RL by learning a latent space noise predictor. With this pipeline, GR-RL is, to our knowledge, the first learning-based policy that can autonomously lace up a shoe by threading shoelaces through multiple eyelets with an 83.3% success rate, a task requiring long-horizon reasoning, millimeter-level precision, and compliant soft-body interaction. We hope GR-RL provides a step toward enabling generalist robot foundations models to specialize into reliable real-world experts.

• The paper introduces GR-RL, a multi-stage framework that transforms general vision-language-action models into specialists for long-horizon, precise manipulation tasks.
• It combines offline RL-based data filtering, symmetry augmentation, and online latent-space reinforcement learning to boost performance from 45.7% to 83.3% success.
• The approach addresses challenges from noisy human demonstrations and dynamic mismatch, offering robust adaptation and scalability for complex robotic tasks.

GR-RL: Specialist Training for Long-Horizon, Dexterous Robotic Manipulation

Motivation and Context

The paper introduces GR-RL, a robotic learning framework designed to specialize large-scale generalist vision-language-action (VLA) models for challenging long-horizon and high-precision manipulation tasks, exemplified by autonomous shoe lacing. Existing VLA foundation models possess strong semantic generalization but fail to address dexterity, millimeter-level control, and error accumulation in extended manipulation. GR-RL directly confronts two interrelated bottlenecks: the inherent suboptimality and noise in human demonstrations, and the mismatch between the dynamics of demonstration data versus real-world inference conditions. The approach proposes a multi-stage pipeline that incrementally refines an initially generalist policy into a task-specialist capable of robust operation in precise, long-duration manipulation scenarios.

Figure 1: GR-RL performs long-horizon, dexterous, and high-precision manipulation, in the task of shoe lacing, by adopting a multi-stage training pipeline, consisting of 1) offline filtered behavior cloning with learned task progress, 2) simple yet effective action augmentation, 3) online reinforcement learning.

Architecture and Learning Paradigm

GR-RL employs a Mixture-of-Transformer (MoT) architecture comprising a vision-language-action policy $\pi_\theta$ and a multi-task critic $Q_\phi$, totaling 5B parameters.

• Policy: The robot is controlled via bi-manual chunked actions conditioned on vision, proprioception, and text instructions. The VLM backbone is Qwen2.5-VL-3B-Instruct, and the action diffusion transformer is trained with flow matching objectives for efficient inference and chunk generation.
• Critic: Chunked Q-values are predicted using distributional RL, framing value prediction as classification over a bounded discrete support which enhances uncertainty modeling in high-sparsity, long-horizon settings.

  Figure 2: The GR-RL Model. GR-RL adopts a Mixture-of-Transformer (MoT) architecture. It is co-trained on robot vision-language-action trajectories via a flow-matching objective, and Temporal-Difference (TD) errors via distributional reinforcement learning.

Multi-Stage Specialist Training Pipeline

1. Data Filtering via RL-based Progress Estimation: The critic $Q_\phi$ is trained offline (TD3+BC) to model sparse binary success over both successful and failed demonstrations, enabling an automated progress evaluator. Transitions with progress drops beyond a threshold are filtered, eliminating noisy, suboptimal fragments without manual labeling.
2. Morphological Symmetry Augmentation: Exploiting bi-manual task symmetry, vision, proprioception, actions, and instructions are mirrored. This augmentation markedly amplifies generalization and robustness, as validated by empirical gains across all evaluated manipulation stages.
3. Online RL Alignment with Latent Noise Exploration: To mitigate the training-inference mismatch caused by control-level optimizations (e.g., temporal ensembling), online RL is performed in model latent space. A noise predictor outputs structured action perturbations, penalized for excess divergence. Critic heads are distilled in noise space to stabilize closed-loop online adaptation. This strategy delivers rapid sample-efficient policy improvement.

Experimental Results and Quantitative Analysis

• Task: Autonomous shoe lacing with dual-arm ByteMini-v2 robot (specs in Figure 3).
• Main Metrics:
  • Baseline (GR-3, naive BC): 45.7% success rate
  • Filtered BC: 61.6%
  • Filtered BC + Augmentation: 72.7%
  • Online RL-tuned (GR-RL final): 83.3% success rate
  • 

  • Figure 4: Multi-stage training contributions to success rates; rapid improvement during offline-to-online RL adaptation.

• Stage-wise Analysis:

Filtering and online RL mainly reduce threading failures, while symmetry augmentation provides uniform improvements in picking, threading, handover, and tightening tasks.

Figure 5: Detailed success rates for intermediate manipulation stages, illustrating specific areas of improvement per training strategy.

• Progress Evaluator Ablation:

Distributional critics demonstrate superior sensitivity and stability respective to suboptimal actions in successful and near-failure cases, outperforming regression-based and non-distributional variants for long-horizon sparse reward signals.

Figure 6: Examples of learned task progress highlighting sudden drops during operator mistakes.

Figure 7: Distributional critics robustly reflect task progress; non-distributional variants exhibit poorly bounded predictions.

• Generalization Capabilities:

GR-RL adapts to variable shoe configurations, retries after failures, manipulates grasp strategy, and resolves ambiguous lacing scenarios without explicit programming.

Figure 8: GR-RL successfully threads a shoelace in a shoe with a novel color.

Implications and Future Directions

GR-RL demonstrates that offline RL-based data filtering, morphology-informed augmentation, and latent-space online RL can systematically transform a generalist foundation policy into a reliable specialist for dexterous manipulation. The claims that human demonstration is fundamentally noisy for high-precision tasks, and that distributional RL critics are essential for robust progress modeling in this setting, challenge conventional approaches in the VLA domain.

Practically, these results imply foundation models can be specialized for dexterity through autonomous data curation and efficient closed-loop RL, with only moderate requirements for additional interaction and no manual annotation. Theoretically, the pipeline establishes distributional progress evaluation and symmetry augmentation as nontrivial contributors to manipulation robustness.

Limitations persist regarding policy drift in noisy reward regimes and credit assignment in high-dimensional latent action spaces. Future directions include distilling online improvements into the base VLA and integrating higher-capacity exploration modules.

Conclusion

GR-RL provides a systematic, empirically validated framework for converting web-scale, generalist VLA policies into capable specialists for long-horizon, high-precision manipulation. By combining distributional RL-based progress filtering, symmetry augmentation, and latent-space online adaptation, GR-RL achieves state-of-the-art autonomous shoe lacing with 83.3% success. These methodological advances redefine the pathway from generalist pretraining to robust real-world specialist deployment, with ramifications for both the scaling laws of foundation models and the architecture of training pipelines in robotic manipulation (2512.01801).