Object reconstruction from a single image — in the wild — is a problem where we can make progress and get meaningful results today. In this paper, we tackle the problem of reconstructing the 3D shape of objects from a single image in realistic scenes. Our approach leverages deformable 3D models that can be learned from 2D annotations available in existing object detection datasets. We use automatic object segmentations to drive the models, and include a bottom-up module for recovering high-frequency shape details that the top-down deformable model may miss. We evaluate our method on the PASCAL 3D+ dataset and demonstrate results on PASCAL VOC imagery, showing that category-level 3D reconstruction from a single image is achievable with current data and techniques.

Recovering the 3D structure of objects from images is one of the oldest problems in computer vision. While significant progress has been made in multi-view stereo and structure from motion, these methods require multiple views of the same scene. Single-image 3D reconstruction is fundamentally more challenging because it is inherently ill-posed — infinitely many 3D shapes can project to the same 2D image. However, by exploiting category-level shape priors, we can constrain the solution space and produce plausible reconstructions even from a single view.

Our approach consists of two complementary components. The top-down component uses deformable 3D models learned from category-level training data. These models capture the typical shape distribution of an object category and are deformed to fit the observed image evidence. The bottom-up component recovers fine-grained shape details — such as the individual legs of a chair or the side mirrors of a car — that the smooth deformable model cannot capture. Crucially, our models are learned from 2D annotations already available in existing detection benchmarks (bounding boxes, segmentation masks, keypoints), avoiding the need for expensive 3D ground truth.

Shape-from-X methods — including shape-from-shading, shape-from-texture, and shape-from-contour — have been studied extensively but typically recover only 2.5D surface representations rather than complete 3D shapes. Category-specific reconstruction approaches such as those using morphable models (e.g., for faces) achieve impressive results but are limited to specific categories with strong prior models. Non-rigid structure from motion (NRSfM) can recover deformable shapes but requires temporal correspondences across frames. Our work uniquely combines category-level deformable models with bottom-up refinement, operating on single images without temporal information.

For each object category, we learn a deformable 3D model consisting of a mean shape and a set of deformation bases. The mean shape is computed by aligning and averaging 3D CAD models from the category. The deformation bases are obtained via Principal Component Analysis (PCA) on the aligned shapes, capturing the dominant modes of shape variation within the category. Given a new image, the model is fit by optimizing deformation coefficients, camera parameters, and pose to minimize the reprojection error between the projected 3D model and image evidence (silhouettes, keypoints). The objective function combines silhouette consistency, keypoint reprojection error, and a shape regularization term that penalizes deviation from the mean shape.

The deformable model provides a smooth, low-frequency approximation of the object shape, but many important details are lost. The bottom-up module addresses this by using the object segmentation mask to carve the initial 3D shape. Specifically, we project the current 3D shape estimate onto the image plane and compare it with the predicted figure-ground segmentation. Regions where the projected shape extends beyond the segmentation mask are carved away (removed), while regions within the mask but outside the projection indicate areas where shape should be added (extruded). This iterative carving and extrusion process recovers fine-grained geometric details that the parametric model cannot represent.

We evaluate on the PASCAL 3D+ dataset, which provides 3D CAD model annotations for 12 rigid object categories in PASCAL VOC images. We consider two evaluation settings: fully annotated (using ground-truth bounding boxes, keypoints, and segmentations) and fully automatic (using predicted detections and segmentations). Under the fully annotated setting, our method achieves high-quality reconstructions with accurate pose estimation across categories including cars, chairs, bicycles, and aeroplanes. Under the fully automatic setting, the system produces reasonable reconstructions despite noise in the automated predictions. Qualitative results on PASCAL VOC imagery demonstrate that the method generalizes to challenging real-world images with clutter, occlusion, and varying viewpoints.

Ablation analysis confirms the complementary value of both components: the deformable model alone produces smooth but detail-lacking shapes, while the bottom-up refinement alone (without global shape initialization) often fails to converge. The combination consistently outperforms either component in isolation. We also demonstrate that the system can produce 3D reconstructions for objects detected by standard object detectors (R-CNN), enabling a fully automatic pipeline from image to 3D shape.

We have demonstrated that category-specific 3D object reconstruction from a single image is a problem where meaningful progress can be made today. By combining top-down deformable shape models with bottom-up visual refinement, and training from readily available 2D annotations, our system produces 3D reconstructions in challenging real-world settings. The key insight is that category-level shape priors, learned from existing datasets, provide sufficient constraints to overcome the fundamental ambiguity of single-view reconstruction. We believe this work opens a path toward richer scene understanding where objects are not just detected and segmented but also understood in their full 3D extent.