UniPose9D

One model, any object, a full 9D pose

Give UniPose9D an object's mask in one RGB-D image, or in one RGB image with predicted depth, and it returns the object's rotation, translation and metric size. It needs no category label, no CAD model, no shape prior and no reference views. Instead of regressing a canonical coordinate at every pixel, it predicts them for pairs of points, calibrates metric scale from each pair, and recovers the pose with an N-hop Kabsch–Umeyama solver; flow matching handles symmetric objects. A single set of weights, trained on a mixture of public datasets, is used for every result on this page.

Yellow cuboids are the predicted metric boxes; the red, green and blue axes show the predicted orientation.

Just the image and a mask

Most pose estimators need something about the object beyond the picture. UniPose9D is category-agnostic: at inference it is not given a category name, a category template, a mean shape, an instance mesh or reference views. It learns canonical orientations from aligned pose annotations and shares one prediction head across all categories.

Point pairs instead of pixels

1. FeaturesThe masked depth is back-projected to a point cloud. Each point gets a visual descriptor from DINOv2 (enhanced by 3DCorrEnhance) and a geometric one from a small multi-radius PointNet.
2. Pairwise predictionPairs of points are sampled, each with three context points. A residual MLP and a transformer encode them, and a joint flow-matching head predicts the pair's two NOCS coordinates and the metric box size.
3. Pose recoveryEach pair calibrates the metric scale by itself. N-hop Kabsch–Umeyama with an adaptive inlier threshold gives rotation and translation; averaging the box sizes of inlier pairs gives the size.

Why point pairs

From noisy pairs to a box, step by step

A toy version of the recovery stage in 2D, seen from above. The “network” here is simulated: predicted NOCS coordinates are the true ones plus noise, and a share of the pairs are pure outliers. Everything after that, the per-pair scale, the RANSAC hops with the adaptive threshold and the size average, is computed live as in the paper.

Four benchmarks, the same weights

The training mixture is the training splits of PACE, Omni6DPose, NOCS and HouseCat6D; the model is evaluated on their held-out test sets with each benchmark's own metrics and no fine-tuning. On PACE it leads every reported metric, and on Omni6DPose it leads or ties every one. On NOCS REAL275 and HouseCat6D it is competitive with category-level specialists that are trained for each benchmark's fixed categories: close to SecondPose on REAL275 and ahead of it on HouseCat6D, while the recent specialist CleanPose scores higher on both.

Cat: a category-level specialist, trained for a fixed set of categories with category-specific inputs or branches. Gen: generalizable, needing no category label, mean shape, CAD model or reference view at inference.

Datasets and categories it never trained on

YCB-Video and HOPE are not in the training mixture: none of their object instances or scenes were seen. With RGB-D input, more than 91% of poses are within 2 cm and the median rotation error is about 4°. With RGB alone, depth comes from MoGe-2. Rotation stays usable (median 8.9° and 7.6°), but metric translation mostly fails (3.4% and 1.5% within 2 cm), because any error in the predicted depth scale goes straight into the translation; on HOPE, GenPose++ is slightly better on that one number.

Categories absent from training

A stricter test keeps only categories with no example in the four training datasets, such as the power drill and pitcher of YCB-Video or the ladle and whisk of HANDAL. (This says nothing about what the pretrained DINOv2 or MoGe-2 saw.)

Omni6D is a different dataset from Omni6DPose. Its baselines train on Omni6D and are tested on its 13 held-out categories; UniPose9D is evaluated zero-shot on the same split. It is ahead on box IoU (50.7 vs 36.9) and on every rotation and joint metric, while the trained baselines are more accurate on translation alone. On HANDAL, 25 examples from five unseen categories serve as a small diagnostic.

In the wild

What matters, what it costs, where it fails

On REAL275, removing the geometric PointNet features or replacing flow matching with plain regression costs the most. Predicting NOCS per point instead of per pair, dropping the pairwise scale calibration, or using plain RANSAC instead of the N-hop loop each lose a few points.

Limitations

The model targets rigid indoor objects of moderate size and needs an instance mask. Its accuracy depends on the mask and the depth; with RGB input, errors of the predicted depth mainly hurt the metric translation. Flow sampling dominates the runtime, which limits uses that need frequent pose updates.

Two typical failures are shown on the right: under severe occlusion too few reliable correspondences remain, and very thin objects leave the box size and orientation ambiguous.

Try it on your own image

The demo runs on Hugging Face Spaces: upload an image, mark the object with point prompts, and estimate its 9D pose. Depth and camera intrinsics are predicted when you only have RGB.

See also

BibTeX

@misc{you2026unipose9duniversalcategoryagnosticobject,
  title={UniPose9D: Universal Category-Agnostic Object Pose Estimation},
  author={Yang You and Yi Du and Cole Harrison and Leonidas Guibas},
  year={2026},
  eprint={2607.09985},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2607.09985}
}