UKPGAN

Keypoints are the points that rebuild the shape

What makes a point a keypoint? UKPGAN treats it as information compression: a good set of keypoints should be enough to reconstruct the whole object. The detector scores every point, a GAN keeps the scores sparse, and a decoder has to rebuild the point cloud from what the high-scoring points carry. No labels are used, yet the keypoints match human annotations, follow human bodies through poses, survive arbitrary rotations, and carry over from synthetic shapes to real scans.

Look at the keypoints yourself

UKPGAN's keypoints on ShapeNet objects, SMPL human bodies and real 3DMatch scans. Each object has 2,048 points; the detector gives each one a keypoint probability. Switch between the keypoints left after non-maximum suppression and every point with probability above 0.5, and drag to turn the object.

Compress the shape into a few points

A rectangle can be rebuilt from its four corners; the points between them add little. UKPGAN learns this trade-off with an encoder that scores points and a decoder that reconstructs the shape from the scored points.

1. Rotation-invariant local featuresEach point's neighbourhood is aligned to a local reference frame and voxelized into a smoothed density grid, then refined by 3D convolutions.
2. Probability and embeddingTwo heads give every point a keypoint probability Φ(x) and a 128-d embedding h(x).
3. GAN sparsity controlA WGAN-GP critic pushes each object's probabilities toward a U-shaped Beta prior: most near 0, a few near 1.
4. Salient information distillationA channel-wise max over Φ(x)·h(x) feeds a TopNet decoder that must reconstruct the point cloud (Chamfer loss).

Why the max, and why a GAN

The decoder only sees the channel-wise maximum of Φ(x)·h(x), so a point contributes only if it is both probable and distinctive; points with ordinary, repeated context get suppressed. An L1 penalty on Φ would just push all scores down; matching a Beta prior through the GAN instead lets scores pile up at both 0 and 1, and its two parameters set how many keypoints come out.

Close to what people mark

On ShapeNet chairs with expert keypoints and on airplanes, chairs and tables from KeypointNet, a detected keypoint counts as correct if it lies within a geodesic distance of a human-marked one. Above the smallest thresholds, UKPGAN's keypoints overlap human annotations more than those of the learned detectors USIP and D3Feat and the hand-crafted Harris-3D, ISS and SIFT-3D.

A single model for 11 more categories

The same keypoints as the body moves

SMPL gives point-to-point correspondences between human bodies in different poses, so we can check whether a keypoint on one pose is also detected at the same place on another (IoU), and how far apart the corresponding detections are (consistency loss). UKPGAN finds hands, feet and the head in every pose.

Fixed budgets of 10, 20 and 40 keypoints

SMPL bodies with a fixed budget of 10, 20 or 40 keypoints after NMS (supplementary). UKPGAN is best in every column.

Detector	IoU (%) ↑			Consistency loss (×10⁻²) ↓
	10	20	40	10	20	40
USIP	2.0	2.4	2.1	13.9	12.5	12.7
D3Feat	1.6	1.8	1.9	15.8	14.1	11.6
HARRIS-3D	0.4	0.5	0.9	21.5	15.1	10.4
ISS	1.0	0.9	1.2	20.8	14.8	10.3
SIFT-3D	0.1	0.4	1.0	21.9	14.9	10.4
UKPGAN (ours)	7.2	7.5	9.2	9.2	7.9	5.9

Trained on shapes, used on real scans

Registration first picks keypoints, then matches their descriptors. We keep two state-of-the-art descriptors, PerfectMatch and D3Feat, and swap the detector, giving each a budget from 2,500 down to 100 points per fragment. UKPGAN is trained only on synthetic ShapeNet models; the scans come from 3DMatch (indoor) and ETH (outdoor).

With few keypoints, where a detector matters most, UKPGAN with PerfectMatch beats ISS, SIFT-3D and random sampling by a wide margin. With the D3Feat descriptor, D3Feat's own detector, trained jointly with the descriptor on real scans, stays best, and UKPGAN is second.

Rotate the object, keep the keypoints

Orientations are usually unknown, so a detector should find the same points however the object is turned. Thanks to the local reference frames, UKPGAN's keypoints are repeatable under arbitrary rotations: a keypoint counts as repeated if another rotated copy has one within 0.1 of it. With just four keypoints, more than 90% repeat; from 16 on, nearly all do.

What each piece does

Without the local rotation-invariant features, keypoints stop being repeatable under rotation. Without GAN sparsity control, scores no longer single out meaningful points and agreement with human labels drops, even though the remaining keypoints repeat slightly more. Salient information distillation and symmetric regularization both raise agreement with human labels.

Beyond 3D

The same idea works on 2D binary images. Treating each MNIST pixel as a point, UKPGAN finds the strokes' skeleton points and, without class labels, learns embeddings that are consistent within each digit.

See also

BibTeX

@inproceedings{you2022ukpgan,
  title={UKPGAN: A General Self-Supervised Keypoint Detector},
  author={You, Yang and Liu, Wenhai and Ze, Yanjie and Li, Yong-Lu and Wang, Weiming and Lu, Cewu},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  pages={17042--17051},
  year={2022}
}