Both image segmentation and dense 3D modeling from images are intrinsically ill-posed problems that require strong regularization. In this paper, we argue that these two tasks can mutually benefit each other. Segmentations provide geometric cues about which surface orientations are more likely at given spatial locations, while 3D reconstruction provides suitable regularization for the segmentation problem by lifting the labeling from 2D images to 3D space. We propose a joint formulation that incorporates learned appearance-based cues and 3D surface orientation priors for class-specific regularization. We demonstrate through experiments on real datasets that our joint approach improves results compared to treating each problem separately.

Multi-view 3D reconstruction from images has made remarkable progress in recent years, producing dense surface models of complex environments. Separately, semantic image segmentation has advanced significantly, with methods that can assign pixel-level class labels to images. However, these two tasks have largely been studied in isolation. 3D reconstruction methods typically produce geometric models without semantic meaning, while segmentation methods operate in 2D without exploiting the underlying 3D scene structure. We propose to solve both problems jointly, allowing each task to inform and constrain the other.

The key intuition is that different object classes have characteristic 3D geometric properties. For example, ground surfaces are predominantly horizontal, walls are vertical, and cars have smooth curved surfaces. If we know the class of a region, we can impose class-specific surface orientation priors that guide the 3D reconstruction. Conversely, the 3D volumetric representation provides a natural regularization for segmentation: labeling in 3D space ensures view-consistency across multiple images and leverages the full geometric context, something that 2D-only methods cannot achieve.

Dense 3D reconstruction from multiple views has been extensively studied, from volumetric methods using truncated signed distance functions (TSDF) to variational approaches that minimize photoconsistency-based energy functionals. These methods produce high-quality geometry but lack semantic understanding. On the segmentation side, conditional random fields (CRFs) have become standard for incorporating spatial context into pixel-wise classification. Recent works have begun to bridge these two domains: Semantic SLAM approaches incorporate class labels into mapping, but typically treat segmentation and reconstruction as sequential rather than truly joint processes.

We formulate the problem on a 3D volumetric grid where each voxel is assigned both a binary occupancy label (inside/outside the surface) and a semantic class label. The joint energy function consists of three terms: (1) a data term derived from multi-view photoconsistency and appearance-based class likelihoods; (2) a surface regularization term that penalizes surface area with class-dependent anisotropic weights favoring expected surface orientations for each class; and (3) a semantic smoothness term that encourages spatially coherent labeling in 3D. The total energy is minimized over both the surface geometry and the semantic labels simultaneously.

The class-specific surface regularization is the key innovation connecting segmentation and reconstruction. For each semantic class c, we learn a surface orientation distribution from training data: ground planes concentrate probability mass on horizontal normals, walls on vertical normals, and vegetation shows a more uniform distribution. These distributions are encoded as anisotropic weights in the surface area regularizer, so that when a voxel is labeled as "ground," horizontal surfaces are penalized less than vertical ones, effectively guiding the reconstruction toward class-appropriate geometry. The appearance term uses boosted classifiers on texton and color features projected from the images to provide per-voxel class likelihoods.

The joint energy function is non-convex due to the coupling between geometry and semantics, making global optimization intractable. We adopt an alternating minimization strategy: in one step, we fix the semantic labels and optimize the 3D surface using convex relaxation of the binary labeling problem; in the other, we fix the surface geometry and optimize the semantic labels using graph cuts. The convex relaxation of the surface reconstruction subproblem can be solved efficiently using primal-dual algorithms. Convergence is typically achieved within 5-10 iterations of the alternating scheme. While we cannot guarantee global optimality, experiments show consistent improvement over independent optimization of each task.

We evaluate our approach on outdoor urban scenes with semantic classes including ground, building, vegetation, car, and sky. Input consists of calibrated multi-view image sequences. For 3D reconstruction quality, we compare against standard volumetric reconstruction without semantic priors and show that class-specific regularization produces cleaner surfaces with fewer artifacts — ground planes are flatter, building facades are smoother. For segmentation accuracy, lifting labels to 3D improves consistency across views, reducing the per-pixel error rate by approximately 5-8% compared to independent 2D segmentation. The mutual benefit is confirmed: joint optimization improves both reconstruction and segmentation.

We have presented a joint formulation for simultaneous 3D scene reconstruction and semantic class segmentation. By incorporating class-specific surface orientation priors into the reconstruction energy and leveraging 3D consistency for segmentation regularization, our approach demonstrates that these two traditionally separate tasks benefit significantly from being solved together. The improvements in both geometric quality and segmentation accuracy confirm the value of this joint perspective. Future work includes scaling to larger scenes with more diverse object classes and incorporating dynamic objects into the framework.