Predicting the 3D shape of objects from a single image is a fundamental challenge in computer vision. This paper presents a system that detects objects in real-world images and produces a triangle mesh representing the full 3D shape of each detected object. The approach augments Mask R-CNN with a mesh prediction branch that uses voxel representations converted to meshes and refined through graph convolution networks operating on mesh vertices and edges. This unified system handles multiple objects with varying topology, occlusion, and diverse real-world conditions.

While current systems excel at recognizing and localizing objects in 2D, they ignore the 3D structure of the world. Unifying 2D object detection with 3D shape prediction in a single system is an important but largely unsolved problem, especially for real-world images with multiple objects, occlusion, and varying lighting. Prior approaches to single-image 3D reconstruction typically operate on synthetic data with clean backgrounds and known object categories, limiting their practical applicability.

The novelty of this work lies in predicting meshes with varying topology by leveraging voxel predictions as initialization, which are then refined to capture fine object structures like chair legs and table edges. The key insight is that voxels provide a robust coarse shape representation that can be converted to a mesh via a "cubify" operation and subsequently refined through learned graph convolutions. This two-stage approach avoids the fixed-topology limitation of deformation-based methods while still producing high-quality mesh outputs.

Existing approaches to single-image 3D reconstruction can be categorized by their output representation. Voxel-based methods predict 3D occupancy grids but suffer from cubic memory scaling that limits resolution. Point cloud methods generate unordered sets of 3D points but lack surface connectivity information. Mesh deformation approaches start from a template mesh (typically a sphere) and deform vertices to match the target shape, but are restricted to objects with the same topology as the template. This work combines the topological flexibility of voxels with the geometric precision of meshes.

The voxel branch predicts 3D occupancy grids representing coarse object shapes. Given aligned features from the RoI head, a small network predicts per-voxel occupancy probabilities. A "cubify" operation then converts the voxel predictions into triangle meshes by replacing each occupied voxel with a cuboid structure, merging shared vertices and eliminating interior faces. This initial mesh captures the coarse topology of the object and serves as the starting point for mesh refinement.

The mesh refinement branch refines initial meshes through three operations in sequence. First, vertex alignment extracts image features for each mesh vertex by projecting it onto the feature map. Second, graph convolution propagates information along mesh edges, allowing vertices to aggregate neighborhood context. Third, vertex refinement updates vertex positions using learned transformations. Training employs chamfer distance, normal consistency, and edge length regularization losses operating on point clouds sampled from mesh surfaces. These three stages are applied iteratively, progressively improving mesh quality.

Experiments are conducted on ShapeNet and Pix3D datasets. On ShapeNet, the method significantly outperforms prior work across multiple shape categories. On the more challenging Pix3D benchmark with real-world images, Mesh R-CNN achieves 51.1% AP^mesh on the S1 split, substantially surpassing baseline approaches. The key advantage appears on objects with complex topologies like chairs with holes, where template-deformation methods fundamentally cannot represent the correct topology. Ablation studies confirm that combining voxel prediction with mesh refinement substantially outperforms using either component alone.

Mesh R-CNN demonstrates that end-to-end 3D shape prediction from real-world images is achievable by augmenting a state-of-the-art 2D detection system with a voxel-to-mesh prediction pipeline. The cubify operation bridges the gap between voxel and mesh representations, enabling topologically diverse predictions. The graph convolution-based refinement captures fine geometric details that coarse voxels miss. This work suggests that the rich feature representations learned by modern detection systems contain sufficient information for 3D reasoning, opening the door to more holistic scene understanding systems.