We address the problems of contour detection, bottom-up grouping and semantic segmentation of RGB-D images from indoor environments. We propose algorithms for object boundary detection and hierarchical segmentation that generalize the gPb-ucm approach by leveraging depth information. We label contours by type: depth, normal, or albedo discontinuities, and introduce a long-range amodal completion mechanism for grouping. For semantic segmentation, we classify superpixels into 40 object categories on the NYUD2 dataset using both generic and class-specific features capturing appearance and geometry. We demonstrate the effectiveness of our approach for scene understanding and show that contextual information further enhances object recognition, achieving significant improvements over the state-of-the-art.

The availability of affordable RGB-D sensors such as the Microsoft Kinect has opened up new possibilities for indoor scene understanding. Unlike traditional RGB images, depth maps provide explicit geometric information about the scene, enabling algorithms to reason about 3D structure, surface orientations, and spatial relationships between objects. However, most existing approaches for scene recognition and object detection have been developed primarily for RGB data and do not fully exploit the rich geometric cues available in depth images.

In this work, we take a principled approach to indoor scene understanding from RGB-D images, addressing three interrelated tasks. First, we extend the gPb contour detector and UCM hierarchical segmentation framework to incorporate depth-derived features including height above ground, angle with gravity, and surface normal differences. Second, we introduce an amodal completion mechanism that uses long-range depth cues to infer occluded surface boundaries. Third, we build a semantic segmentation system that uses region-level features combining appearance, shape, and 3D geometric context to achieve state-of-the-art recognition performance on the NYUD2 benchmark.

Contour detection has a rich history in computer vision, from early edge detectors to the more recent gPb (globalized probability of boundary) framework which combines multiscale local brightness, color, and texture gradients with spectral clustering. The UCM (ultrametric contour map) provides a natural hierarchy of regions. However, these methods were designed for RGB images and do not leverage depth information. Recent works on RGB-D segmentation have used depth in various ways, but typically treat it as an additional channel rather than exploiting its geometric meaning.

For semantic segmentation of indoor scenes, the seminal NYU Depth V2 dataset by Silberman et al. provides 1449 densely labeled RGB-D images with 40 semantic classes. Prior approaches have used hand-crafted features with random forest classifiers or kernel descriptors on RGB-D patches. Scene classification methods typically rely on global scene descriptors such as GIST. Our work differs by building a unified pipeline that integrates low-level perceptual organization with high-level semantic recognition, leveraging the geometric structure revealed by depth at every stage.

We extend the gPb framework to RGB-D images by incorporating depth-derived gradient channels. Specifically, we compute oriented gradients of depth values, surface normals, and height above the estimated ground plane. Each gradient channel captures different types of boundaries: depth gradients detect occlusion boundaries, normal gradients reveal orientation discontinuities at surface folds, and height gradients help separate objects from the support surface. These are combined with the standard brightness, color, and texture gradients using a logistic regression classifier trained to predict true boundary probability.

Furthermore, we introduce a contour classification scheme that labels each detected boundary as arising from a depth discontinuity, a surface normal discontinuity, or an albedo/texture change. This classification is performed by training a multi-class classifier on the relative contributions of each gradient channel. Knowing the type of boundary provides valuable semantic information: depth boundaries indicate occlusion relationships, normal boundaries indicate object parts or surface curvature, and albedo boundaries suggest material or illumination changes.

Building on the extended contour detector, we construct ultrametric contour maps (UCMs) that provide a hierarchical segmentation of the RGB-D image. At each level of the hierarchy, regions are merged based on the minimum boundary strength along their shared contour. We further enhance grouping with an amodal completion mechanism: using depth-based reasoning about occlusion relationships, we infer the full extent of surfaces that are partially hidden behind other objects. This allows the algorithm to group together disconnected image regions that belong to the same physical surface, such as a table visible on both sides of a chair.

For semantic segmentation, we classify each superpixel in the hierarchical segmentation into one of 40 object categories defined in the NYUD2 dataset. Our feature set includes generic features (color histograms, texture descriptors, shape statistics, 3D bounding box properties, surface normal distributions) as well as class-specific features that capture the geometric context of each region relative to the scene structure — e.g., height above ground, distance to walls, and local support relationships. A linear SVM classifier is trained on these features, followed by contextual refinement using a CRF that encourages spatial consistency.

We evaluate our approach on the NYU Depth V2 dataset, which contains 1449 RGB-D images of indoor scenes with dense semantic labels across 40 categories. For contour detection, our RGB-D gPb achieves an ODS F-score of 0.72, compared to 0.65 for the RGB-only baseline, demonstrating the significant benefit of depth features. For semantic segmentation, our method achieves a pixel-wise accuracy of 60.3% and a mean class accuracy of 35.1%, significantly outperforming prior methods. The addition of contextual features and CRF-based refinement provides a further 3-4% improvement in pixel accuracy. We also demonstrate strong performance on scene classification, correctly categorizing 73.7% of test scenes into room types.

We have presented a comprehensive framework for perceptual organization and recognition of indoor scenes from RGB-D images. By extending the gPb-ucm pipeline with depth-derived features, introducing amodal completion for better grouping, and building a rich feature-based semantic segmentation system, we achieve significant improvements over prior work across multiple tasks. Our results demonstrate that depth information, when properly exploited through geometric reasoning rather than treated as a simple additional channel, provides substantial benefits for scene understanding. Future directions include incorporating learned feature representations and extending to full 3D scene parsing.