Regress what does not turn, vote for what does
To find oriented 3D boxes, voting detectors such as VoteNet ask every point for the offset to its object's centre and for the box heading. Both change whenever the object turns, and per point they are barely better than a guess. Canonical Voting regresses only what stays fixed: each point's Local Canonical Coordinates (where it sits in its object's own frame) and the box size. Every point then votes for the centre under every possible heading; votes pile up at real centres, and an LCC back-projection check removes the peaks that do not agree with the predicted coordinates.
heading error of the boxes detected on ScanNet, vs 62.5° to 79.1° for the other detectors
on ScanNet with one model per category: 21.7 vs 11.9 for the next best method (AP at IoU 0.5)
on SceneNN with the ScanNet models, no retraining, vs 21.7 for the next best
at IoU 0.5 on single-view SUN RGB-D, vs 43.7 for BRNet
Predicted boxes in real scans
Boxes predicted by Canonical Voting in two ScanNet scans, and in two SceneNN scans with the model trained on ScanNet. Drag to orbit, use the buttons or pinch to zoom, and click a class to hide or show its boxes.
Scan
Classes
Boxes are coloured by predicted class. The few classes we could not name with confidence from these scans are grouped as “other”.
Offsets and headings turn with the object
A VoteNet-style detector regresses, for every point, the offset to its object's centre and the box heading. Both depend on how the object is turned: the same backrest has to point somewhere else for every orientation of the chair, so the network must memorise each local pattern under every rotation. On ScanNet, these per-point predictions are barely better than a random guess, and the heading is even slightly worse:
Offset to the centre (mean abs. error)
Heading vector (mean abs. error)
Local Canonical Coordinates (LCCs) place each point in its object's own box, scaled to [−1, 1]: the backrest is always at the back, whatever the heading. Turn the chair below. Each colour is the value the network has to output for that point.
Direct offset to the centre
The target of each part changes as the chair turns: point 1 must output a different offset for every heading.
Local Canonical Coordinates
The target stays with the part, so the network only has to recognise parts, much like part segmentation.
Vote under every heading, keep what checks out
A point knows its LCC p̃ and box size s, but not the heading. So it votes for every heading r it could have: the centre would be at p − R(r)(s ∘ p̃). These candidate centres trace a circle; circles from the points of one object cross at its centre, each weighted by the point's objectness. A peak in the vote map proposes a box, with the heading and size averaged from the votes there. The box is kept only if its points, back-projected into the box's own frame, land where the network said they would.
A toy room seen from above, not network output: the “predicted” LCCs and sizes are the true ones plus noise, and floor and wall points get random predictions with low objectness. Click any point to see its votes.
Headings tried, K
Noise in predicted LCCs, σ
Three stages
- Regress LCCs, sizes and objectnessA sparse-convolution U-Net predicts, for every point, its Local Canonical Coordinates, its object's box size, an objectness score and a class. For symmetric objects the loss takes the minimum over the symmetric versions.
- Canonical votingEach point votes for its box centre under K = 120 headings, weighted by objectness, into a 3D grid that also collects the voted heading and size. The voting is linear in the number of points and runs in about 30 ms on a GPU.
- LCC check by back projectionPeaks become box candidates one at a time, highest first. The points inside a candidate are mapped back to LCCs; a candidate stays only if enough of them are object points and they agree with the predicted LCCs.
Why the check matters
Trying every heading also makes circles from different objects cross by chance, and some of these crossings become peaks. A false box has the wrong centre or heading, so the coordinates it implies for its points disagree with the network's LCCs, and the check removes it. In the paper's run on the right, iteration 3 accepts three chairs; iteration 4 rejects a box on the edge of a table.
The thresholds come from 5-fold cross-validation on the training set: a candidate needs more than 20% object points among those inside it (β = 0.2) and an objectness-weighted LCC error below 0.3 (γ = 0.3), and the search stops when the highest peak drops below δ = 60 votes.
Three indoor benchmarks
ScanNet with the oriented boxes of Scan2CAD (14K+ boxes in 1,506 scans, 9 categories). Canonical Voting has the best mAP with one model per category (21.7 vs 11.9) and with one joint model (15.4 vs 12.1 for H3DNet). Per-category training helps it much more than the baselines, since LCCs are consistent within a category; it is not the best in every category.
SceneNN, 76 scans with oriented boxes, evaluated with the models trained on ScanNet and no retraining. With per-category models Canonical Voting leads (26.8 vs 21.7); with one joint model, VoteNet is slightly ahead (21.5 vs 21.1).
SUN RGB-D
Single RGB-D frames rather than full scans, with about 5K training images. Here the vote map is sampled for 512 box proposals that a BRNet-style module refines; proposals from the vote map work better than farthest point sampling of votes.
Heading, position and speed
Errors of the boxes detected at IoU 0.5 on ScanNet. The heading error drops by a factor of six; centres are as accurate as the other methods'; one scan takes 0.32 s.
The SUN RGB-D variant
What each part does
Replacing LCCs with direct offsets, with the same voting, gives no correct boxes at all: the offsets do not agree, so votes never pile up. Without the back-projection check, false peaks become boxes; without objectness, every floor and wall point votes too.
Every point votes
VoteNet clusters or subsamples its votes into proposals, which can miss objects with few visible points. Here every point casts its own votes. Grouping objects by a partial index, the number of points per unit of box volume, Canonical Voting has a higher recall for partial and occluded objects (indices 1 to 8); for the most complete ones (9 and 10), VoteNet is ahead.
Beyond headings: full 3D rotations
The same scheme handles full 3D rotations: each pixel of an RGB-D frame votes over directions on the sphere instead of headings on a circle, and each object's rotation is then solved in closed form from its points' LCCs with the Umeyama algorithm. On the NOCS REAL275 pose benchmark this reaches 17.0, 40.7 and 45.8 mAP at (5°, 5 cm), (10°, 5 cm) and (10°, 10 cm), well above the NOCS baseline (supplementary). The idea continues in CPPF, which votes with point pairs for category-level 9D poses.
Run it
The repository has the training and evaluation scripts for per-category and joint models on ScanNet with Scan2CAD labels, the CUDA kernel for canonical voting, and the SUN RGB-D variant.
At a glance
Input: a coloured point cloud, voxelized at 3 cm. Backbone: a ResNet34-like U-Net of 3D Minkowski convolutions. Training: Adam, learning rate 0.001, batch size 3, scenes rotated about the vertical axis in 90° steps with 20° jitter. Inference: canonical voting with K = 120, the back-projection check with β = 0.2, γ = 0.3, δ = 60, then NMS at IoU 0.3.
Read the full abstract
3D object detection has attracted much attention thanks to the advances in sensors and deep learning methods for point clouds. Current state-of-the-art methods like VoteNet regress direct offset towards object centers and box orientations with an additional Multi-Layer-Perceptron network. Both their offset and orientation predictions are not accurate due to the fundamental difficulty in rotation classification. In the work, we disentangle the direct offset into Local Canonical Coordinates (LCC), box scales and box orientations. Only LCC and box scales are regressed, while box orientations are generated by a canonical voting scheme. Finally, an LCC-aware back-projection checking algorithm iteratively cuts out bounding boxes from the generated vote maps, with the elimination of false positives. Our model achieves state-of-the-art performance on three standard real-world benchmarks: ScanNet, SceneNN and SUN RGB-D.
BibTeX
@inproceedings{you2022canonical,
title={Canonical Voting: Towards Robust Oriented Bounding Box Detection in 3D Scenes},
author={You, Yang and Ye, Zelin and Lou, Yujing and Li, Chengkun and Li, Yong-Lu and Ma, Lizhuang and Wang, Weiming and Lu, Cewu},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
pages={1193--1202},
year={2022}
}