ShapeClipper: Scalable 3D Shape Learning from Single-View Images via Geometric and CLIP-based Consistency

Zixuan Huang1, Varun Jampani2, Anh Thai1, Yuanzhen Li2, Stefan Stojanov1, James M. Rehg1
1Georgia Institute of Technology, 2Google Research
CVPR 2023
Paper Video Code

ShapeClipper learns to infer 3D shapes from images, only supervised by single-view masked images.

Abstract

We present ShapeClipper, a novel method that reconstructs 3D object shapes from real-world single-view RGB images. Instead of relying on laborious 3D, multi-view or camera pose annotation, ShapeClipper learns shape reconstruction from a set of single-view segmented images. The key idea is to facilitate shape learning via CLIP-based shape consistency, where we encourage objects with similar CLIP encodings to share similar shapes. We also leverage off-the-shelf normals as an additional geometric constraint so the model can learn better bottom-up reasoning of detailed surface geometry. These two novel consistency constraints, when used to regularize our model, improve its ability to learn both global shape structure and local geometric details. We evaluate our method over three challenging real-world datasets, Pix3D, Pascal3D+, and OpenImages, where we achieve superior performance over state-of-the-art methods.

Video

BibTeX

@inproceedings{huang2023shapeclipper,
  author    = {Huang, Zixuan and Jampani, Varun and Thai, Anh and Li, Yuanzhen and Stojanov, Stefan and Rehg, James M},
  title     = {ShapeClipper: Scalable 3D Shape Learning from Single-View Images via Geometric and CLIP-based Consistency},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year      = {2023},
}