IEEE TPAMI 2024

CPPF++

Uncertainty-Aware Sim2Real Object Pose Estimation by Vote Aggregation

Yang You1,3 Wenhao He2 Jin Liu1 Hongkai Xiong1 Weiming Wang1 Cewu Lu1
1Shanghai Jiao Tong University2Flexiv Robotics Inc.3Stanford University
Paper IEEE Xplore Code DiversePose 300 BibTeX
Overview

A pose estimator you can train without a single real label.

Looking for a robust, general 6D pose estimator for RGB-D images from any sensor, for your own robotic setup? CPPF++ estimates the pose and size of everyday objects from a single RGB-D frame. It is trained only on synthetic CAD models, without real images, real pose labels or backgrounds, and still transfers to real scenes: a new voting mechanism keeps the model from overfitting to the training domain.

65.9mAP

on NOCS REAL275 at 10°/5 cm, the best among methods trained on synthetic objects only (CPPF: 45.2).

92.2AP

for bowls on DiversePose 300 at 60°/5 cm, versus 66.2 for the best prior method.

0real labels

used in training: only rendered CAD models, with no background or clutter.

Key ideas

What makes it work

Sim2Real by design

The only prior knowledge is a set of 3D CAD models. Training data is rendered online from these objects alone, so there is no dependence on real annotations, backgrounds or a particular camera.

Uncertainty-aware voting

Point pairs with similar local geometry can point to very different places on an object, a problem we call vote collision. CPPF++ predicts a probability distribution for each vote in canonical space instead of a single guess.

N-point tuples

Voting units of N points instead of pairs carry more context. Two pairs that look identical become distinguishable once a third point is added.

Filtering and refinement

Noisy tuples are filtered out before they vote on orientation, an online alignment step refines rotation and translation, and a tuple feature ensemble adds robustness.

Method

Pipeline

CPPF++ pipeline: sample point tuples, extract features, encode tuples, predict canonical coordinates and scale with uncertainty, vote for the centre, filter noisy tuples, vote for orientation, and refine with online alignment.
  1. Masked inputA masked point cloud from an off-the-shelf instance segmentation model.
  2. Point tuplesTuples are randomly sampled from the object and encoded into tuple embeddings.
  3. Coordinates & scaleEach tuple predicts its canonical coordinates and scale, with uncertainty.
  4. VotingTuples vote for the centre; only reliable tuples vote for the orientation.
  5. Online alignmentAn optimization step refines the predicted rotation and translation.
In the wild

A casual iPhone 15 video, straight from the model.

The pose of the mug is estimated independently on every frame of a handheld clip.

  • No temporal smoothing
  • No object mesh required
  • Trained on synthetic data only

CPPF++ also serves as the pose module of RPMArt (IROS 2024) for articulated objects such as microwaves and drawers.

Results

Strong on REAL275, stronger in new domains

On NOCS REAL275, CPPF++ is the best method trained on synthetic objects only, and is comparable to methods trained with real pose labels. When transferred to new data domains (Wild6D, PhoCAL and DiversePose 300), it generalizes better than previous methods. Pick a benchmark and a category below.

NOCS REAL275 · mAP (%) over all categories. Bold: best among methods using real training data, and best among synthetic-only methods.
MethodTraining data3D253D505° 5 cm10° 5 cm15° 5 cm
NOCSSyn. Obj + Real Obj & BG74.427.89.824.134.9
CASSSyn. Obj + Real Obj & BG––23.558.0–
SPDSyn. Obj + Real Obj & BG––21.454.1–
FS-NetSyn. Obj + Real Obj & BG––28.260.8–
CenterSnapSyn. Obj + Real Obj & BG––27.258.8–
ShAPOSyn. Obj + Real Obj & BG––48.866.8–
DualPoseNetSyn. Obj + Real Obj & BG82.357.336.167.876.3
HS-PoseSyn. Obj + Real Obj & BG82.671.656.184.192.8
RePoNetSyn. Obj + Real Obj & BG (w/o label)––33.963.0–
UDA-COPESyn. Obj + Real Obj & BG (w/o label)––34.866.0–
TTA-COPESyn. Obj + Real Obj & BG (w/o label)––35.973.2–
SAR-NetSyn. Obj + Real BG77.947.542.368.375.0
Chen et al.Syn. Obj16.70.70.83.17.2
Gao et al.Syn. Obj70.623.15.114.221.9
SAR-Net*Syn. Obj64.034.04.821.130.7
CPPFSyn. Obj78.225.018.045.251.6
CPPF++ (ours)Syn. Obj82.455.232.365.985.2
CPPF++ (ours) w/ GroundedSAM Det.Syn. Obj72.758.026.272.088.9

NOCS REAL275. In the per-category tables, NOCS, DualPoseNet and HS-Pose are trained with real-world pose annotations, while SAR-Net*, CPPF and CPPF++ are pure sim-to-real methods. Bold: best, underlined: second best, as marked in the paper.

Qualitative comparison on NOCS REAL275: ground truth; NOCS, DualPoseNet and HS-Pose trained with real-world data; CPPF and CPPF++ trained on synthetic data.
NOCS REAL275. NOCS, DualPoseNet and HS-Pose use real-world training data; CPPF and CPPF++ are trained on synthetic data. Though trained only on synthetic CAD models, CPPF++ is comparable to current state-of-the-art methods.
Qualitative comparison on DiversePose 300 for a bowl and a mug: ground truth, CPPF, NOCS, HS-Pose, DualPoseNet and CPPF++.
DiversePose 300. A bowl and a mug in non-upright poses: ground truth, CPPF, NOCS, HS-Pose, DualPoseNet and CPPF++ (right).
New benchmark

DiversePose 300

Existing benchmarks such as NOCS REAL275, Wild6D and PhoCAL are biased toward upright poses. DiversePose 300 contains 305 annotated real RGB-D frames of bottles, bowls and mugs in diverse, non-upright poses, captured in a crowded box, on a desk, and held in a hand.

Two bottles in a crowded box with their ground-truth 3D boxes
bottle · in a box
A bottle lying on a desk with its ground-truth 3D box
bottle · on a desk
A bottle held in a hand with its ground-truth 3D box
bottle · in hand
A bowl in a crowded box with its ground-truth 3D box
bowl · in a box
An upside-down bowl on a desk with its ground-truth 3D box
bowl · on a desk
A tilted bowl held in a hand with its ground-truth 3D box
bowl · in hand
A mug in a crowded box with its ground-truth 3D box
mug · in a box
A mug lying on its side on a desk with its ground-truth 3D box
mug · on a desk
A mug held in a hand with its ground-truth 3D box
mug · in hand

Orange boxes: ground-truth 3D annotations. The release provides RGB images, colored point clouds, 3D box labels and camera intrinsics.

Download on Hugging Face Code on GitHub
Abstract
Read the full abstract

Object pose estimation constitutes a critical area within the domain of 3D vision. While contemporary state-of-the-art methods that leverage real-world pose annotations have demonstrated commendable performance, the procurement of such real training data incurs substantial costs. This paper focuses on a specific setting wherein only 3D CAD models are utilized as a priori knowledge, devoid of any background or clutter information. We introduce a novel method, CPPF++, designed for sim-to-real category-level pose estimation. This method builds upon the foundational point-pair voting scheme of CPPF, reformulating it through a probabilistic view. To address the challenge posed by vote collision, we propose a novel approach that involves modeling the voting uncertainty by estimating the probabilistic distribution of each point pair within the canonical space. Furthermore, we augment the contextual information provided by each voting unit through the introduction of N-point tuples. To enhance the robustness and accuracy of the model, we incorporate several innovative modules, including noisy pair filtering, online alignment optimization, and a tuple feature ensemble. Alongside these methodological advancements, we introduce a new category-level pose estimation dataset, named DiversePose 300. Empirical evidence demonstrates that our method significantly surpasses previous sim-to-real approaches and achieves comparable or superior performance on novel datasets.

Cite

BibTeX

@article{You_2024,
  title={CPPF++: Uncertainty-Aware Sim2Real Object Pose Estimation by Vote Aggregation},
  ISSN={1939-3539},
  url={http://dx.doi.org/10.1109/TPAMI.2024.3419038},
  DOI={10.1109/tpami.2024.3419038},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  publisher={Institute of Electrical and Electronics Engineers (IEEE)},
  author={You, Yang and He, Wenhao and Liu, Jin and Xiong, Hongkai and Wang, Weiming and Lu, Cewu},
  year={2024},
  pages={1–16}
}