RGBGrasp: RGB-Based 6-DoF Grasp Planning

• RGBGrasp is an RGB-based grasping system that reconstructs 3D scenes from a limited set of images for real-time 6-DoF grasp planning.
• It integrates motion planning, online image capture, and incremental NeRF training with depth ranking to generate actionable 3D point clouds.
• Experimental results demonstrate superior grasp success rates on challenging materials, including transparent and specular objects.

RGBGrasp is an RGB-based object grasping paradigm that leverages multi-view image acquisition, neural radiance field reconstruction, geometry constraints via monocular depth predictions, and accelerated NeRF optimization for precise 6-DoF grasp planning—even for transparent and specular objects. Unlike conventional approaches requiring dense visual data or specialized depth sensors, RGBGrasp uses a limited sequence of images captured during wrist-mounted camera trajectories to reconstruct the scene, extract actionable 3D information, and drive robust grasp execution, achieving real-time performance in both simulated and real-world contexts (Liu et al., 2023).

1. System Architecture and Pipeline

The RGBGrasp pipeline is architected for eye-on-hand setups, in which a robot arm equipped with a wrist-mounted RGB camera acquires a sparse multi-view image stream as it approaches the target object. The primary modules and their respective data flow are:

• Motion Planner: Programs a 90° arc trajectory in the table plane, optimizing the approach for partial, yet informative, scene coverage.
• Online Image Capture and ROS Streaming: RGB frames and camera-to-gripper extrinsics are streamed incrementally from a ROS node into the NeRF engine via NerfBridge.
• Incremental NeRF Training: As new views are acquired, rapid online NeRF optimization is performed, incorporating geometry regularization (see Section 2).
• Volumetric Rendering & Depth Extraction: The trained NeRF model volumetrically renders per-view depth maps, which are then converted into 3D point clouds.
• Grasp Planning & Execution: State-of-the-art grasp detectors (AnyGrasp or GraspNet) operate over this reconstructed point cloud, producing 6-DoF grasp candidates. The best-scoring grasp is transformed to the robot base frame and executed.

This architecture is specifically tuned for scenes with few views (typically <10), emphasizing adaptability to real-world robotic workflows with significant practical constraints (Liu et al., 2023).

2. 3D Geometry Estimation with Depth-Rank Regularization

RGBGrasp’s core novelty is its geometry constraint mechanism for NeRF learning under sparse-view conditions. The system leverages a pre-trained monocular depth predictor to provide relative ordering (not metric) of scene depths. For each pixel $(u,v)$, the frozen network predicts a scalar depth $\hat{Z}(u,v)$. NeRF then estimates continuous per-ray depth $Z(r) = \sum_{i=1}^N \alpha_i t_i$, with $\alpha_i = 1-\exp(-\sigma_i \delta_i)$ as compositing weights along the ray.

A custom “depth rank loss” enforces ordinal depth relationships: for each pair of pixels $(k_1,k_2)$ where $\hat{Z}_{k_1} < \hat{Z}_{k_2}$, the loss penalizes

$$L_R = \sum_{(k_1, k_2)} \max(Z_{k_1} - Z_{k_2} + \epsilon, 0)$$

with $\epsilon=10^{-3}$ for margin stability. This constraint stabilizes NeRF training and preserves boundary geometry, particularly beneficial for challenging object surfaces (specular, transparent) that confound both hand-tuned and learned photometric constraints. Ablation studies show that omitting $L_R$ reduces grasp success by 3–4% and increases artifacts on object boundaries (Liu et al., 2023).

3. Fast Sparse-View NeRF: Hash Encoding and Proposal Sampling

To meet real-time constraints, RGBGrasp incorporates two acceleration strategies:

• Hash Encoding: Rather than positional encoding, the pipeline uses multi-resolution hash tables. Each 3D point is scaled to grid resolution $r_\ell$, hashed via prime multipliers and modular arithmetic $h_\ell(v)$; features are interpolated and concatenated across levels, yielding $enc(x)$ for the NeRF MLP.
• Proposal Sampler: A coarse MLP $g_\phi$ produces preliminary densitities $\sigma_i^{(c)}$ and weights $w_i^{(c)}$ on each ray sample. $N_{fine}$ points are resampled from the coarse distribution and refined by the main NeRF MLP $F_\Theta$.

Together, these mechanisms enable efficient NeRF training with $\approx 10^5$ forward passes (around 1,200 optimization steps per scene, batch size 8,192 rays), such that full 3D reconstruction and grasp planning are achieved in $\sim 1.5$ minutes per scene (Liu et al., 2023).

4. Volumetric Rendering, Depth Extraction, and Grasp Candidate Generation

Scene representation follows the canonical NeRF formulation: $$F_\Theta : (\mathbf{x}, \mathbf{d}) \mapsto (\sigma, \mathbf{c})$$ Volume rendering along camera rays provides color $C(r)$ and depth $Z(r)$ estimates. Upon training completion, the system renders depth maps from a chosen viewpoint (top-down or wrist camera), projects these into 3D point clouds, and passes them to grasp planners (AnyGrasp/GraspNet):

• Grasp candidates $\{G_i\}$ are predicted and scored for antipodal stability and gripper geometry compatibility via a learned network.
• Selection and Execution proceeds by transforming the highest quality grasp into global coordinates and executing via a robot trajectory planner.

This modular integration of NeRF-based geometry and established grasp planners enables generalization across object shapes and materials with limited view coverage (Liu et al., 2023).

5. Experimental Evaluation and Quantitative Benchmarks

RGBGrasp has been rigorously validated in both simulation and real robot settings:

• Datasets: Simulated scenes (PyBullet + Blender) use 56 ShapeNet meshes with varying optical properties (“pile” and “packed” configurations); real-world tests employ a Franka Emika Panda with wrist-mounted RealSense D415, covering a $30\times30\times30$ cm workspace and various daily objects.
• Performance Metrics:
  • Success Rate (SR) / Detection Rate (DR): For a 90° view range, RGBGrasp achieves 86.7% SR / 81.5% DR (pile) and 84.8% / 86.0% (packed), versus GraspNeRF’s 17.4% / 9.4% and 28.1% / 23.6% respectively.
  • Depth RMSE: Comparable accuracy (RGBGrasp: $9.94 \times 10^{-3}$ m vs. GraspNeRF: $9.61 \times 10^{-3}$ m), but RGBGrasp requires only $\sim 62$ k reconstructed points versus GraspNeRF’s $262$ k.
  • Approach Trajectory Robustness: RGBGrasp maintains $83.7/82.2\%$ SR on diffuse/mixed material scenes, outperforming single-view RGB-D when transparent/specular objects are present.
  • Real Robot Trials: SR of $84.1\%$ (pile), $77.1\%$ (packed) vs. GraspNeRF’s $70.1\%/66.7\%$ and RGB-D methods $<72\%$ (Liu et al., 2023).

These quantitative results establish RGBGrasp’s superiority in handling diverse object properties and challenging view conditions with low data requirements and fast reconstruction.

6. Contributions, Limitations, and Future Directions

Key contributions:

• Depth-rank regularization stabilizes NeRF training with sparse multi-view RGB for transparent/specular objects.
• Hash encoding and proposal sampling enable real-time 3D scene reconstruction, meeting practical robotic manipulation rates.
• Demonstrated adaptability of grasp planning under narrow, flexible camera trajectories without reliance on dense point cloud data or reference CAD models.

Limitations:

• NeRF optimization for each scene entails $10^5$ MLP passes; further speedups may require transition to sparse voxel grids or distilled NeRFs for on-the-fly operation.
• Scene dynamics (moving objects) currently cannot be handled—integration of real-time object tracking and updating remains an open area.
• The system does not incorporate explicit object detection or semantic priors, which could increase both geometry accuracy and grasp reliability if merged into the pipeline (Liu et al., 2023).

Future work: Research directions include integrating fast online geometry update mechanisms, semantic-aware grasp planning, and architectural innovations in neural field learning to facilitate dynamic or unstructured environments.

---

RGBGrasp defines a contemporary standard for RGB-only, multi-view, geometry-aware robotic grasp planning, balancing real-time requirements, object diversity, and rigorous 3D reconstruction constraints—it is positioned as an advanced solution for both academic research and applied manipulation in real-world robotic contexts (Liu et al., 2023).