SCOPE: Semantic Conditioning for Sim2Real Category-Level Object Pose Estimation in Robotics

Abstract: Object manipulation requires accurate object pose estimation. In open environments, robots encounter unknown objects, which requires semantic understanding in order to generalize both to known categories and beyond. To resolve this challenge, we present SCOPE, a diffusion-based category-level object pose estimation model that eliminates the need for discrete category labels by leveraging DINOv2 features as continuous semantic priors. By combining these DINOv2 features with photorealistic training data and a noise model for point normals, we reduce the Sim2Real gap in category-level object pose estimation. Furthermore, injecting the continuous semantic priors via cross-attention enables SCOPE to learn canonicalized object coordinate systems across object instances beyond the distribution of known categories. SCOPE outperforms the current state of the art in synthetically trained category-level object pose estimation, achieving a relative improvement of 31.9\% on the 5$^\circ$5cm metric. Additional experiments on two instance-level datasets demonstrate generalization beyond known object categories, enabling grasping of unseen objects from unknown categories with a success rate of up to 100\%. Code available: https://github.com/hoenigpeter/scope.

• The paper introduces SCOPE, which uses semantic cross-attention with continuous DINOv2 embeddings to overcome discrete category limitations in pose estimation.
• The method leverages a diffusion-based U-Net to generate NOCS images, enabling the recovery of 6D poses and achieving a 31.9% improvement on the 5°5cm metric.
• Experiments on REAL275, YCB-V, and TYOL demonstrate robust Sim2Real performance, with up to 100% grasp success on unseen objects.

SCOPE: Semantic Conditioning for Sim2Real Category-Level Object Pose Estimation in Robotics

Introduction

"SCOPE: Semantic Conditioning for Sim2Real Category-Level Object Pose Estimation in Robotics" introduces SCOPE, a model aiming to enhance category-level object pose estimation by overcoming the limitations imposed by discrete category labels. This is achieved by incorporating continuous DINOv2 features into a diffusion-based U-Net model through semantic cross-attention conditioning, which allows the model to generalize beyond known categories.

Figure 1: SCOPE's Semantic Cross-Attention Conditioning Advantage.

Category-level pose estimation typically involves labeling objects into predefined semantic categories. However, these categories often do not accurately represent geometry or appearance, potentially hindering generalization to new and unlabelled object instances. SCOPE addresses this by leveraging continuous feature space from DINOv2 representations, allowing for natural transitions between object categories through semantic awareness. The model has demonstrated an improvement of 31.9% on the 5°5cm metric, compared to state-of-the-art synthetically trained models.

Methodology

SCOPE utilizes semantic conditioning to exploit continuous feature spaces rather than discrete labels. This conditioning is implemented using DINOv2 embeddings as semantic features injected into a U-Net diffusion model via cross-attention. Such an approach enables learning of geometric-semantic correspondences beyond human-defined category limitations.

Figure 2: Overview of SCOPE leveraging DINOv2 through cross-attention.

SCOPE generates estimated NOCS images through a conditional Denoising Diffusion Probabilistic Model (DDPM), which allows for recovering 6D poses and scaling using TEASER++, facilitating Sim2Real applications by minimizing the domain gap inherent in category-based methods.

Results

The model's effectiveness is evaluated on datasets such as REAL275, YCB-V, and TYOL, demonstrating superiority in category-level pose estimation without requiring real data. Whereas earlier models relied on concatenation of semantic features to inputs, SCOPE employs a cross-attention mechanism akin to modern image synthesis techniques such as Stable Diffusion, offering adaptive semantic prior estimation for robust performance in real-world tasks.

Figure 3: Exemplary Images of SCOPE on YCB-V and REAL275 Test Images.

SCOPE showed superior generalization capabilities across both known and unknown object categories. On the YCB-V and TYOL datasets, the model exceeded performance benchmarks for synthetic-only data, achieving grasp success rates up to 100% on unknown objects, indicating enhanced semantic reasoning and application potential beyond the training set.

Grasping Experiments

The experimentation extends to robotic grasping tasks, evaluating SCOPE's capacity for actionable robotics applications. Using a Toyota HSR, SCOPE's continuous semantic feature conditioning successfully guided the robot to grasp previously unseen objects from known and new categories alike, demonstrating adaptive real-time pose estimation capabilities.

Figure 4: Exemplary Images from Grasp Experiments.

Grasp success was validated across various objects with different scaling and orientation challenges, as shown by the grasp success rates achieved.

Figure 5: Grasp Success Rates (SR) (\%) for YCB-V Objects.

Conclusion

The presented work encapsulates advances in object pose estimation through the SCOPE model, wherein semantic cross-attention facilitates bridging the Sim2Real gap, achieving superior performance over existing state-of-the-art models. By integrating accessible semantic features, SCOPE not only improves 6D pose estimation but also exhibits potential for broad implementations in autonomous robotics. Future research will involve extending training datasets to encompass more diverse semantic and geometric categories to further scale SCOPE's applications in real-world robotics.