This paper presents 3D-INN (3D Interpreter Network), an end-to-end framework that sequentially estimates 2D keypoint heatmaps and 3D object structure from a single image. The key innovation is a Projection Layer that projects estimated 3D structure to 2D space, enabling training with real 2D-annotated images without requiring 3D ground truth. The method uses keypoint heatmaps as an intermediate representation bridging real and synthetic data. By combining real images with 2D annotations and synthetic images with 3D annotations, the network learns to recover 3D skeleton structure and viewpoint from a single image.

Recovering 3D object structure from a single 2D image is a fundamental yet ill-posed problem in computer vision. Humans can effortlessly infer the 3D shape and pose of objects from a single view, yet teaching machines to do so remains challenging. Existing approaches either rely on optimization-based methods that are sensitive to initialization, or require 3D annotations during training that are expensive to obtain for real images. A key challenge is the domain gap between synthetic 3D training data and real test images — models trained on synthetic data alone generalize poorly to real photographs.

3D pose and shape estimation methods can be categorized into optimization-based and learning-based approaches. Classical methods like deformable models fit parameterized shape models to image observations through iterative optimization that is slow and prone to local minima. Recent deep learning approaches train CNNs to directly regress 3D parameters but typically require 3D-annotated real images for training. Methods using only synthetic training data suffer from the domain gap problem. Keypoint-based methods detect 2D keypoints first and then solve for 3D structure, but treat the two stages independently without end-to-end optimization.

The 3D-INN architecture consists of three components. The Keypoint Estimator is a multi-scale CNN that produces 2D keypoint heatmaps with an information bottleneck for refinement. The 3D Interpreter uses fully connected layers to infer structural and viewpoint parameters from the estimated keypoints. Objects are represented as 3D skeletons, where 3D keypoints are modeled as weighted combinations of deformation basis shapes: Y = sum(alpha_k * B_k), with alpha_k as internal shape parameters and B_k as basis shapes.

The Projection Layer is the key enabler for training with 2D supervision. It maps 3D coordinates to 2D using: X = P(RY + T), where R is the rotation matrix, T is the translation vector, and P is perspective projection. This layer is fully differentiable, enabling backpropagation through the 3D-to-2D mapping. During training on real images (which have only 2D keypoint annotations), the loss is computed in 2D space after projection, and gradients flow backward through the projection to update both the 3D interpreter and the keypoint estimator. This design enables a three-stage training strategy: (1) train keypoint estimator on real 2D-annotated images; (2) train 3D interpreter on synthetic data with known 3D structure; (3) end-to-end fine-tuning on real images using the projection layer.

Experiments evaluate both 2D keypoint estimation and 3D structure recovery. For 2D keypoints, 3D-INN achieves state-of-the-art performance on FLIC (human pose), comparable results on CUB-200-2011 (birds), and superior results on Keypoint-5 (furniture). For 3D structure recovery, the method achieves 88.03% average recall on the IKEA dataset, significantly outperforming optimization-based methods, especially on noisy inputs. On PASCAL 3D+ for viewpoint estimation, 3D-INN achieves comparable performance to state-of-the-art methods. The framework also enables applications such as 3D-aware image retrieval and object graph visualization, demonstrating the practical utility of the recovered 3D structure.

3D-INN demonstrates that a differentiable Projection Layer enables end-to-end learning of 3D object structure from single images using only 2D annotations on real data. By bridging real and synthetic domains through keypoint heatmaps and leveraging the Projection Layer for gradient flow from 2D to 3D, the method achieves strong performance on both 2D keypoint estimation and 3D structure recovery. The approach demonstrates that combining the complementary strengths of real and synthetic data through a carefully designed differentiable pipeline is a viable path for single-image 3D understanding.