KeypointNet

Keypoints that many people agree on

KeypointNet is a large-scale, diverse 3D keypoint dataset built on ShapeNetCore. Instead of one expert's template, many annotators clicked the points they found important, and a new aggregation method turned their noisy, independent clicks into keypoints with semantic ids that are consistent across every model of a category. We also benchmark deep networks and hand-crafted detectors on two tasks: keypoint saliency and keypoint correspondence.

Same colour, same keypoint

Every keypoint has a semantic id shared by all models of its category, so the same colour marks the same part on every object. Drag to rotate all nine models together, and hover a keypoint to find its counterparts.

Large, template-free and in correspondence

How KeypointNet compares

From noisy clicks to consistent keypoints

Different people click different points, and their clicks are never exact. KeypointNet aggregates them by minimizing a fidelity loss: it learns an embedding of every point together with a set of ground-truth keypoints that stay close, in that embedding, to what people annotated.

1. AnnotateIn a web tool, each annotator clicks up to 24 points per model. Keypoints must be shared by the category, spread over the object and distinct in meaning.
2. EmbedA PointConv network maps every point to an embedding. Each click is modelled as a Gaussian, because people make small mistakes.
3. Fidelity errorFor every point, the embedding distances to the matching clicks on the other models of the category are summed into an error map.
4. Non-minimum suppressionLocal minima of the error map become candidate keypoints on each model.
5. ClusterCandidates are projected to 2D with t-SNE and grouped by embedding, which gives the semantic ids shared across models.
6. VerifyExperts check the result with simple priors such as rotational symmetry and centrosymmetry, and fix keypoints the method missed.

Why not cluster the clicks directly? Clustering needs a distance threshold, cannot separate closely spaced keypoints such as the four on an airplane's tail, and gives no labels that match across models. Distances in the learned embedding are also less sensitive to mis-clicks than geodesic distances.

The annotation tool

Annotators work in a browser. The model to label is on the right; models they have already labelled stay on the left, which helps them keep their own keypoint indices consistent.

Keypoints are clicked on meshes and then transferred to point clouds of 2,048 points, so both forms are released.

What's inside

Counts per category, from the released annotations. Splits are train, validation and test at 7:1:2. Click a category to open it in the explorer.

Semantic keypoints are hard to detect

We benchmark eight deep networks and three hand-crafted detectors on two tasks. Keypoint saliency asks which points are keypoints. Keypoint correspondence also asks which semantic id each one has. All numbers use a strict distance threshold of 0.01.

The same metrics as the distance threshold grows from 0 to 0.1. RSNet leads on correspondence at every threshold.

Bold: best in each row, as marked in the paper. Hand-crafted detectors output no probabilities, so they have no mAP.

See also

BibTeX

@inproceedings{you2020keypointnet,
  title={KeypointNet: A Large-Scale 3D Keypoint Dataset Aggregated From Numerous Human Annotations},
  author={You, Yang and Lou, Yujing and Li, Chengkun and Cheng, Zhoujun and Li, Liangwei and Ma, Lizhuang and Lu, Cewu and Wang, Weiming},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  pages={13647--13656},
  year={2020}
}