This paper addresses the problem of 3D object detection in indoor environments using RGBD cameras. Unlike prior approaches that treat object detection in isolation, we propose a holistic approach that jointly reasons about the 3D scene layout, 3D object hypotheses, and their mutual interactions. We extend the Constrained Parametric Min-Cuts (CPMC) framework to 3D in order to generate candidate 3D object cuboids from depth data. We then formulate the problem as inference in a conditional random field (CRF) that couples scene classification with 3D object recognition, leveraging both 2D appearance features and 3D geometric cues. Experiments on the NYU Depth V2 dataset demonstrate that our holistic model achieves substantial improvements over state-of-the-art methods that reason about objects independently.

The recent availability of consumer-grade RGBD sensors such as the Microsoft Kinect has opened new possibilities for indoor scene understanding. Depth information provides strong cues about the 3D geometry of a scene, complementing the appearance information in RGB images. However, most existing methods for 3D object detection from RGBD data follow a pipeline approach: they first detect objects independently and then optionally apply contextual reasoning as a post-processing step. This sequential design fails to capture the rich mutual dependencies between scene layout, object categories, and spatial relationships.

We argue that scene understanding and object detection should be addressed jointly. Knowing the scene type (e.g., kitchen vs. bedroom) provides strong priors on which objects are likely to be present, and conversely, detected objects inform the scene type. Moreover, spatial relationships between objects (e.g., a monitor typically sits on a desk) provide additional constraints. Our model captures all these dependencies through a unified CRF framework that performs joint inference over scene class, object categories, and 3D layouts.

RGBD-based scene understanding has attracted increasing attention with the availability of depth sensors. Silberman et al. introduced the NYU Depth dataset and proposed methods for semantic segmentation from RGBD data. Gupta et al. focused on contour detection and hierarchical segmentation in RGBD images. For 3D object detection, Song and Xiao proposed sliding shapes that search for objects in 3D space. However, these approaches treat detection as an isolated task without leveraging holistic scene context. Our work is also related to contextual models in 2D detection, such as Desai et al.'s work on modeling object co-occurrence and spatial layouts, which we extend to the 3D domain.

We extend the Constrained Parametric Min-Cuts (CPMC) framework from 2D to 3D. Given an RGBD image, we first compute a 3D point cloud from the depth map and extract planar surfaces using RANSAC-based plane fitting. We then generate 3D cuboid proposals by placing axis-aligned bounding boxes at multiple locations and scales in the 3D space. Each cuboid is scored based on how well it encloses a coherent group of 3D points, using features such as point density, surface normal consistency, and color homogeneity. This generates a manageable set of high-quality 3D object candidates that cover the scene.

We formulate the joint reasoning as inference in a Conditional Random Field. The CRF includes three types of variables: a scene class variable (kitchen, bedroom, etc.), object detection variables for each cuboid proposal, and support relationship variables encoding which objects support others. The unary potentials capture individual evidence: scene type is predicted from global features (GIST, spatial pyramid), and each object cuboid is classified using 3D shape descriptors and 2D appearance features. The pairwise potentials encode contextual interactions: scene-object compatibility, object-object co-occurrence statistics, and spatial support relationships. Inference is performed via message passing, iterating until convergence.

We evaluate our approach on the NYU Depth V2 dataset, which contains 1449 RGBD images of indoor scenes with dense per-pixel labels. We follow the standard split with 795 training and 654 testing images. For 3D object detection, we evaluate using average precision (AP) at IoU threshold of 0.25 in 3D space. Our holistic model achieves significant improvements across most object categories compared to baselines that detect objects independently. Specifically, the joint model improves AP by an average of 4.2 points over the strongest baseline. Scene classification accuracy also improves when jointly optimized, reaching 72.1% compared to 67.8% for the independent classifier. Ablation studies confirm that both the scene-object coupling and the support relationship modeling contribute to the overall performance gain.

We have presented a holistic approach to 3D scene understanding from RGBD data that jointly reasons about scene layout, object detection, and support relationships. By extending CPMC to 3D for proposal generation and formulating the joint reasoning as CRF inference, our method effectively leverages the mutual dependencies between scene-level and object-level understanding. Experiments on the NYU Depth V2 dataset demonstrate substantial improvements over methods that treat detection in isolation. Our work suggests that holistic scene reasoning is a fruitful direction for indoor 3D understanding, and future work could incorporate richer relationships such as functional affordances and temporal consistency.