3DCorrEnhance

Do vision transformers understand 3D?

Foundation models such as DINOv2 are trained on 2D images, yet 3D tasks increasingly rely on their features. We measure how view-equivariant those features are, meaning whether the same 3D point gets the same feature from every viewpoint. Equivariance turns out to predict performance on pose estimation, tracking and semantic correspondence, and a light finetuning on 3D correspondences improves it for every model we tried.

The same 3D point should get the same feature

We render 1,000 Objaverse objects from 42 views (42,000 images and 1.8 billion dense correspondences) and sample 1,000 real MVImgNet objects (33.3 million correspondences). For each point in one view, we find its nearest neighbour in feature space in another view, and score how far it lands from the true match: APE, the average pixel error, and PCDP, the percentage of correct dense points. We compare DINOv2, DINOv2 with registers, MAE, CLIP and DeiT.

Equivariance predicts 3D task performance

We test three tasks that rely on correspondences, from easy to hard. Across models, lower APE (more equivariant features) goes with better scores on all of them.

One-shot pose estimation

Same object, rigid motion. Match a query image to stored views, then solve RANSAC-PnP. OnePose-LowTex and YCB-Video.

Video tracking

Same object, non-rigid motion. Follow points through a video by feature similarity. TAP-Vid-DAVIS.

Semantic correspondence

Different objects of a category, any viewpoint. Transfer keypoints by feature similarity. PF-PASCAL.

The points line up from the top left to the bottom right: more equivariant features, better 3D task performance. DINOv2 leads on both axes.

Pull corresponding pixels together

If equivariance helps, can we train for it? We finetune each model so that pixels showing the same 3D point in two views get similar features.

1. Two viewsPick an Objaverse object and two of its rendered views, and sample pixels that see the same 3D points.
2. Light adaptersLoRA on the last four transformer blocks, plus one 3×3 convolution that mixes neighbouring patches before upsampling.
3. Ranking lossSmoothAP makes each pixel's true match rank first among candidates. It beats a contrastive loss with a fixed margin.
4. Short training10K iterations with AdamW at learning rate 1e-5, on a 10K-object subset of Objaverse.

Before and after, on unseen objects and scenes

SpeakerTurtleStreet scene

BeforeAfter finetuning

DINOv2 features as PCA colours, before and after finetuning. Drag across the image: after finetuning, features are smoother, less noisy and more consistent between views. Objects from MVImgNet; scene from TAP-Vid-DAVIS.

Every model gets better at every 3D task

The same finetuning recipe improves DINOv2, DINOv2 with registers, MAE, CLIP and DeiT, and it is trained only on synthetic objects, yet it also improves equivariance on real MVImgNet images. Pick a benchmark and a metric.

All numbers from the paper and its appendix. The main metric of every benchmark improves for every model; a few secondary ones barely move (for example tracking occlusion accuracy for MAE and CLIP). "More backbones" covers DINOv2 Small to Giant, with and without registers, and ConvNeXt, on the three main benchmarks.

A little 3D goes a long way

Finetuning on a single object already brings large improvements, and which object hardly matters: six random Objaverse objects all work, even an untextured hemisphere. Training on one multiview pair for a single iteration improves DINOv2 by 4.85 (3cm-3deg pose), 3.55 (tracking AJ) and 3.47 (PCK@0.05).

Drop-in better features

Methods that use DINO features can simply swap in the finetuned ones. In Wild Gaussians, which uses DINOv2 to handle occluders in novel view synthesis, the finetuned features improve PSNR, SSIM and LPIPS on almost every scene. In LERF, they sharpen the language-embedded relevancy maps.

Finetuning targets 3D correspondence. On tasks that need other things, features stay close to the original: Paris-H instance retrieval 75.92 → 76.23, VOC2012 segmentation mIoU 83.60 → 82.65, NYUv2 depth δ₁ 86.88 → 85.48 (DINOv2-Base, from the appendix).

What matters in the recipe

One convolution layer works best; SmoothAP beats contrastive and Procrustes losses; object-centric Objaverse beats real MVImgNet and scene-centric data overall; results are stable across learning rates.

See also

BibTeX

@inproceedings{you2025multiview,
  title={Multiview Equivariance Improves 3D Correspondence Understanding with Minimal Feature Finetuning},
  author={You, Yang and Li, Yixin and Deng, Congyue and Wang, Yue and Guibas, Leonidas},
  booktitle={International Conference on Learning Representations (ICLR)},
  year={2025},
  url={https://openreview.net/forum?id=CNO4rbSV6v}
}