We address the problem of amodal 3D object detection in RGB-D images, aiming to produce a 3D bounding box of an object in metric form at its full extent, even for partially occluded objects. We introduce a 3D ConvNet approach that takes a volumetric representation of the scene constructed from RGB-D input. Our method features the first 3D Region Proposal Network (RPN) that learns objectness from geometric shapes rather than relying solely on 2D features. A joint Object Recognition Network (ORN) extracts geometric features in 3D space and color features in 2D simultaneously. Our approach achieves 13.8 mAP improvement over prior methods and operates 200 times faster than the original Sliding Shapes.

While 2D object detection has made remarkable progress with deep learning, most methods produce 2D bounding boxes that lack depth information and physical scale. For applications such as robotics, autonomous driving, and augmented reality, understanding the full 3D extent of objects is crucial. The availability of RGB-D sensors (e.g., Microsoft Kinect) provides depth information that can be leveraged for 3D scene understanding. Previous approaches to 3D detection, including the original Sliding Shapes method, used hand-crafted features and exhaustive sliding window search in 3D space, resulting in extremely slow inference. We propose to bring the power of deep learning to 3D object detection through a fully 3D convolutional neural network pipeline that operates on volumetric scene representations.

3D object detection methods can be categorized by their input representation. Image-based methods estimate 3D properties from monocular images but suffer from depth ambiguity. Point cloud methods directly process raw 3D data but face challenges with irregular data structures and varying point density. Voxel-based methods discretize 3D space into regular grids, enabling the use of 3D convolutions, but prior work relied on hand-crafted features. The original Sliding Shapes demonstrated the potential of voxel representations but was limited by SVM classifiers and exhaustive search. Recent success of Region Proposal Networks (RPNs) in 2D detection (Faster R-CNN) motivates our extension to 3D, replacing exhaustive search with learned proposals.

We propose the first 3D Region Proposal Network (3D RPN) that operates directly on a volumetric representation of the scene. The RGB-D input is first converted into a 3D Truncated Signed Distance Function (TSDF) volume that encodes both geometry and free space. The 3D RPN applies 3D convolutional layers on the TSDF volume, producing objectness scores and 3D bounding box regression offsets at each voxel location. To handle objects at different scales, we employ a multi-scale amodal RPN training strategy with two sets of anchors. The 3D RPN learns to detect objectness from geometric shapes — distinguishing the regular geometry of man-made objects from the irregular structure of walls and floors — without relying on color information.

For each 3D proposal from the RPN, the Object Recognition Network (ORN) performs joint feature extraction from both 3D geometry and 2D color. The 3D branch extracts geometric features by applying 3D convolutions on the proposal's TSDF volume. The 2D branch projects the 3D proposal onto the color image and extracts appearance features using a 2D CNN (VGGNet). These two feature streams are concatenated and fed into fully-connected layers for object classification and 3D bounding box regression. This dual-stream architecture leverages the complementary strengths of geometric and appearance information: geometry provides scale and shape cues while color provides texture and semantic cues.

Experiments are conducted on the NYU Depth v2 dataset for indoor 3D object detection. Our method achieves a 13.8 mAP improvement over the original Sliding Shapes baseline, demonstrating the effectiveness of learned features over hand-crafted ones. The 3D RPN alone generates high-recall proposals while being significantly faster than exhaustive search. The joint ORN with both geometric and color features consistently outperforms using either modality alone, validating the dual-stream design. Importantly, the entire pipeline operates 200 times faster than the original Sliding Shapes, making it practical for real-world deployment. Ablation studies confirm that the TSDF representation outperforms binary occupancy grids, as the signed distance encodes richer geometric information about surface proximity.

We have presented Deep Sliding Shapes, a complete 3D convolutional neural network pipeline for amodal 3D object detection in RGB-D images. The 3D Region Proposal Network learns to generate object proposals from geometric shape priors in volumetric space, while the joint Object Recognition Network fuses 3D geometric and 2D color features for classification and localization. Our approach demonstrates that deep learning can be effectively extended from 2D to 3D object detection, achieving significant improvements in both accuracy and speed over prior methods.