The same features, however the object is turned
A point cloud arrives in whatever pose the object happened to be in. Networks that read coordinates, such as PointNet++, are trained on upright shapes and break when a shape is turned, and teaching them every rotation by augmentation is expensive. PRIN and SPRIN take another route: they compute a feature for every point that provably does not change when the input is rotated. PRIN builds it with a convolution on spherical voxels. SPRIN, its sparse successor, works on the points directly, which is more efficient and more accurate. Trained on upright shapes only, SPRIN segments randomly rotated shapes as well as upright ones.
part segmentation on randomly rotated ShapeNet shapes for SPRIN, trained on upright shapes only; the best other method reaches 82.17
between SPRIN's scores on upright and on rotated test shapes (82.59 and 82.67). PointNet++ falls from 84.63 to 36.66
of points matched to the right part across objects and rotations with PRIN's features; the best baseline descriptor, CGF, reaches 52.5%
largest deviation from exact invariance of the spherical voxel convolution on a test signal: floating-point precision
A convolution on spherical voxels
PRIN, the Point-wise Rotation Invariant Network, turns the point cloud into a signal on a grid of spherical voxels, convolves it there, and reads the result back at the points.
- InDensity Aware Adaptive SamplingThe unit ball is divided into voxels by two angles and the distance to the centre. Voxels that span equal angles are not equal in size: they shrink towards the poles, which would bias the signal. DAAS scales its sampling filter by sin β, the factor by which spherical coordinates distort volume.
- FeaturesSpherical Voxel ConvolutionVoxel coordinates are mapped one to one onto 3D rotations, which lifts the signal to the rotation group SO(3). There, a group convolution is computed with fast Fourier transforms and averaged over the last rotation angle.
- OutPoint re-samplingEach input point reads its feature from the eight voxels around it by trilinear interpolation. Fully connected layers turn point features into part labels; max-pooling over all voxels gives a global feature for classification.
Why the output cannot depend on the rotation
Rotate the point cloud by any rotation Q. The signal f turns into the rotated signal LQ f, and a point p moves to Qp. The theorem says that the convolution, evaluated at the moved point of the rotated signal, returns exactly the value it had before:
It holds under one condition on the filter ψ: written as a function of the three Euler angles (α, β, γ) of SO(3), it must not depend on γ. In the implementation the filter is a 3D tensor that is constant along its third axis. This condition and the full proof are what the journal paper adds to the theory of the AAAI version.
Why the sampling must be density aware
The convolution needs the signal on a regular grid of angles, because that is what the fast Fourier transforms on the rotation group work with. In space, however, such a grid is far from regular. The volume of a spherical voxel is proportional to h² sin β: the factor h² is the same in all directions, but sin β makes voxels near the poles much smaller than those at the equator.
A filter of fixed width would therefore see the same surface differently depending on where the pole happens to be, which changes with every rotation of the object. DAAS corrects the filter by sin β. The value written into a voxel also records how far the nearby points are from it along the radius, a quantity that no rotation changes.
The same idea, directly on the points
PRIN's voxel grid is dense. GPU memory limits it to 64³ voxels, and most of them contain no point. SPRIN, the sparse version, drops the grid: the point cloud is treated as a sum of Dirac deltas, and the convolution becomes a sum over the points themselves.
T(xj) is the rotation that belongs to the point xj. Applying its inverse turns the cloud so that xj lies on the z axis, and each neighbour xi is seen from there. What is left undetermined is a spin about that axis, and the output is point-wise rotation invariant when the filter ignores it: ψ(x) = ψ(Z(θ)x) for every angle θ.
In practice ψ is a small fully connected network, and the sum runs over a random subset of the k nearest neighbours. The filter also receives six more rotation-invariant numbers: the sides and angles of the triangle formed by xi, xj and the centroid. No sampling into voxels and no re-sampling back are needed.
The network
Sparse spherical correlation layers are stacked as in PointNet++. Two layers run on all 2,048 points; farthest point sampling then keeps 128 points, and later 32, with three layers at each level.
For classification, max- and average-pooled features go through fully connected layers. For part segmentation, two up-sampling stages run the same correlation layer on the finer point sets, without any interpolation weights.
Because the correlation is sparse, 2,048 points fit on one GPU with a large batch. The paper notes that tensor field networks and SE(3)-Transformers do not scale to clouds of this size.
Two classifiers, one rotating object
A toy, not the paper's networks. The shapes are procedural, and nothing is trained: each point gets the part label of its nearest neighbour, in feature space, among the points of five upright example shapes. On the left the features are the raw coordinates x, y, z. On the right they come from one hand-made layer in the spirit of SPRIN: for every pair of points it keeps only what a rotation cannot change, namely where the neighbour lies along and around the axis through the point, and the triangle the two form with the centroid. Drag a view to turn the object. Both sides recompute everything from the rotated coordinates.
Shape
Colour the points by
Rotation
Right now
- Turned away from upright by
- 0°
- mIoU, reading coordinates
- –
- mIoU, invariant features
- –
- Largest change of any feature value since upright
- –
White rings mark points with the wrong part. The classifier on the left has learned a partition of space, so it is right only when the object stands the way the examples did.
The invariant features change only by floating-point rounding, so the labels on the right never change. That they are not perfect has nothing to do with rotation: a hand-made layer and a nearest-neighbour rule are a weak classifier. PRIN and SPRIN learn their filters, stack several layers and are trained on ShapeNet.
Trained upright, tested rotated
Every method is trained on shapes in their canonical orientation, without rotation augmentation, and tested twice: on shapes in the canonical orientation (“upright” below) and on randomly rotated shapes. RIConv, Kim et al. and Li et al. originally reported results with rotation augmentation about the z axis during training; for a like-for-like comparison they were re-trained without it.
Part segmentation, ShapeNet Part
The eight networks that read coordinates score 83 to 85 mIoU on upright shapes and 26 to 50 on rotated ones. Hand-crafted rotation-invariant descriptors fed to a PointNet (SHOT, CGF) are stable but weak. SPRIN has the best instance mIoU and the best class mIoU on rotated shapes (82.67 and 79.50; Li et al.: 82.17 and 78.78), and the best score in 12 of the 16 categories. On upright shapes it stays within 3 points of the best coordinate-reading network (82.59 against 85.33). PRIN, limited by the resolution of its voxel grid, reaches 71.20.
Classification, ModelNet40 and ScanObjectNN
A class label needs one global feature, not point-wise ones, and here the picture is less one-sided. All coordinate-reading networks collapse under rotation, as before. Among the rotation-invariant networks SPRIN is competitive but not the best: 86.13% on rotated ModelNet40 against 89.32% for Li et al., and 69.83% against 73.44% on rotated ScanObjectNN, which consists of real scans with background clutter. SPRIN's accuracy is practically the same with and without rotation on both datasets (86.01% and 86.13% on ModelNet40). PRIN loses some accuracy under rotation (79.76% to 72.43% on ModelNet40): its voxel grid is discrete, and a rotation does not move voxel centres onto voxel centres, an effect the AAAI paper already describes.
Finding the same spot on another object
A feature that ignores rotation is a point descriptor: the nearest feature on a second object tells where a point of the first one lies there, whatever the two poses are. In the toy below, each line joins a point of the upright shape on the left to the point of the rotated shape on the right whose feature is closest. Green lines end on the same part, red ones on a different part. Drag to rotate the shape on the right, and switch the features to raw coordinates to see the matches fall apart.
Features
Shape
Rotation
On ShapeNet
The point features of 80% of the test objects, in canonical orientation, are stored as a database. The remaining 20% are rotated at random and used as queries: every query point retrieves its nearest feature, and the match counts as correct when both points belong to the same part. PRIN's features match 86.1% of the points correctly and SPRIN's 78.9%, against 52.5% for the hand-crafted CGF descriptor.
Aligning a shape from a label
Point-wise labels that follow the object can also be used to pose it. Given a requirement such as “the back of the chair is at the top”, virtual points carrying that label are placed where the part should be, and the pose is optimized until the predicted labels of these points agree with it, measured by KL divergence.
In the example of the AAAI paper the divergence drops from 1.34 for the initial pose to 0.12 for the optimized one.
What the pieces contribute
All numbers are on the randomly rotated ShapeNet Part test set: mean IoU, or the share of correctly labelled points.
PRIN: resolution and sampling
The bandwidth is the resolution of the spherical voxel grid. Going from 4 to 32 raises the mIoU from 52.98 to 71.20, at a higher memory cost. With plain uniform sampling instead of DAAS, the mIoU at bandwidth 32 falls to 67.47.
SPRIN: what the filter sees
The filter receives the neighbour's position in the frame of the point (the correlation feature) and the six triangle features. Each alone already works; together they give the best result.
Fewer points
The test clouds are sub-sampled without retraining. Both networks keep most of their accuracy at half the points; at 512 points SPRIN still reaches 72.04 mIoU, and at 256 both drop sharply.
From PRIN to PRIN/SPRIN
AAAI 2020 introduced PRIN: density-aware sampling, the spherical voxel convolution and point re-sampling. On rotated shapes it was already ahead of every method it was compared with, without any augmentation.
TPAMI 2022 extends it in two ways. The first is the theory: a complete proof of point-wise rotation invariance, which shows what the filter must satisfy. With the convolution revised accordingly, PRIN rises from 58.76 to 71.20 mIoU on rotated shapes, and the gap between its scores on upright and rotated shapes, 12.8 points in the first version, shrinks to 0.8. The second is SPRIN, which carries the idea to sparse points and adds another 11.5 points.
The AAAI paper lists Yang You and Yujing Lou as equal contributors, with Qi Liu, Yu-Wing Tai, Lizhuang Ma, Cewu Lu and Weiming Wang.
| PRIN AAAI 2020 | PRIN TPAMI 2022 | SPRIN TPAMI 2022 | |
|---|---|---|---|
| ShapeNet Part, rotated: instance mIoU | 58.76 | 71.20 | 82.67 |
| ShapeNet Part, rotated: class mIoU | 54.22 | 66.75 | 79.50 |
| ShapeNet Part, upright: instance mIoU | 71.52 | 72.04 | 82.59 |
| ShapeNet Part, upright: class mIoU | 70.33 | 68.39 | 79.31 |
| ModelNet40, rotated: accuracy | 70.35 | 72.43 | 86.13 |
| ModelNet40, upright: accuracy | 80.13 | 79.76 | 86.01 |
| Point matching on rotated shapes: accuracy | 64.23 | 86.12 | 78.92 |
Limits
PRIN's dense grid is the main one: its resolution is bounded by memory (bandwidth 32 in the experiments), and shapes with complex internal structure were failure cases of the first version. Its filters are aware of density but not of curvature. SPRIN removes the grid, but for classification, where a single global feature is enough, it is competitive rather than the best.
Code and weights
qq456cvb/SPRIN holds PyTorch code for both PRIN and SPRIN in their journal versions; SPRIN needs only PyTorch and hydra. Trained checkpoints for ShapeNet part segmentation are on Hugging Face. The original AAAI code is at qq456cvb/PRIN.
PRIN/SPRIN: On Extracting Point-wise Rotation Invariant Features (TPAMI 2022)
Point cloud analysis without pose priors is very challenging in real applications, as the orientations of point clouds are often unknown. In this paper, we propose a brand new point-set learning framework PRIN, namely, Point-wise Rotation Invariant Network, focusing on rotation invariant feature extraction in point clouds analysis. We construct spherical signals by Density Aware Adaptive Sampling to deal with distorted point distributions in spherical space. Spherical Voxel Convolution and Point Re-sampling are proposed to extract rotation invariant features for each point. In addition, we extend PRIN to a sparse version called SPRIN, which directly operates on sparse point clouds. Both PRIN and SPRIN can be applied to tasks ranging from object classification, part segmentation, to 3D feature matching and label alignment. Results show that, on the dataset with randomly rotated point clouds, SPRIN demonstrates better performance than state-of-the-art methods without any data augmentation. We also provide thorough theoretical proof and analysis for point-wise rotation invariance achieved by our methods.
Pointwise Rotation-Invariant Network with Adaptive Sampling and 3D Spherical Voxel Convolution (AAAI 2020)
Point cloud analysis without pose priors is very challenging in real applications, as the orientations of point clouds are often unknown. In this paper, we propose a brand new point-set learning framework PRIN, namely, Pointwise Rotation-Invariant Network, focusing on rotation-invariant feature extraction in point clouds analysis. We construct spherical signals by Density Aware Adaptive Sampling to deal with distorted point distributions in spherical space. In addition, we propose Spherical Voxel Convolution and Point Re-sampling to extract rotation-invariant features for each point. Our network can be applied to tasks ranging from object classification, part segmentation, to 3D feature matching and label alignment. We show that, on the dataset with randomly rotated point clouds, PRIN demonstrates better performance than state-of-the-art methods without any data augmentation. We also provide theoretical analysis for the rotation-invariance achieved by our methods.
BibTeX
@article{you2022prin,
title={PRIN/SPRIN: On Extracting Point-wise Rotation Invariant Features},
author={You, Yang and Lou, Yujing and Shi, Ruoxi and Liu, Qi and Tai, Yu-Wing and Ma, Lizhuang and Wang, Weiming and Lu, Cewu},
journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
volume={44},
number={12},
pages={9489--9502},
year={2022},
doi={10.1109/TPAMI.2021.3130590}
}
@inproceedings{you2020pointwise,
title={Pointwise Rotation-Invariant Network with Adaptive Sampling and 3D Spherical Voxel Convolution},
author={You, Yang and Lou, Yujing and Liu, Qi and Tai, Yu-Wing and Ma, Lizhuang and Lu, Cewu and Wang, Weiming},
booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
volume={34},
number={07},
pages={12717--12724},
year={2020}
}