Semantic segmentation on LiDAR point clouds is crucial for autonomous driving perception. Most existing methods rely solely on 3D point cloud features, ignoring the rich semantic information available from 2D images. In this paper, we propose 2DPASS (2D Priors Assisted Semantic Segmentation), a general training scheme that leverages 2D image priors to boost 3D LiDAR semantic segmentation. Our framework introduces a multi-scale fusion-to-single knowledge distillation approach, which first fuses multi-modal features at multiple scales and then distills the fused knowledge back into the 3D network. Extensive experiments on SemanticKITTI and NuScenes benchmarks demonstrate that 2DPASS achieves state-of-the-art performance, significantly outperforming existing methods on both datasets.

LiDAR-based 3D semantic segmentation assigns a semantic label to each point in the point cloud and serves as a fundamental component for autonomous driving systems. Recent advances have achieved impressive results using either point-based methods that process raw points directly or projection-based methods that convert point clouds into 2D representations such as range images or bird's-eye-view maps. However, these approaches typically operate on geometric features alone and do not exploit the complementary texture and color information available from camera images, which limits their ability to distinguish between visually similar but semantically different objects.

Several recent works attempt to incorporate multi-modal information by fusing 2D image features with 3D point cloud features at inference time. While effective, these methods require both LiDAR and camera inputs during deployment, increasing system complexity and latency. Moreover, the misalignment between 2D and 3D representations and the difficulty of handling sensor failures present additional challenges. We argue that a more practical approach is to leverage 2D priors only during training, producing a 3D-only network at inference that benefits from multi-modal knowledge without the overhead of multi-sensor fusion.

Point cloud semantic segmentation methods can be broadly categorized into three families. Point-based methods, exemplified by PointNet and PointNet++, directly process unordered point sets using shared MLPs and local aggregation. Voxel-based methods discretize the space into regular grids and apply 3D sparse convolutions, with MinkowskiNet and Cylinder3D being representative works. Projection-based methods convert point clouds to 2D representations such as range views or multi-view images and leverage mature 2D convolutional neural networks. Each family has distinct trade-offs between spatial resolution, computational cost, and ability to capture local geometric structures.

Multi-modal fusion for 3D understanding has gained increasing attention. Early approaches perform late fusion by concatenating predictions from separate 2D and 3D branches. More recent works explore point-level fusion, where 2D features are projected onto 3D points using camera-LiDAR calibration. Knowledge distillation from 2D to 3D has also been explored, but existing methods typically distill from a single scale and do not fully exploit the hierarchical structure of multi-modal features. Our 2DPASS addresses this gap by introducing multi-scale fusion followed by knowledge distillation, enabling richer transfer of 2D priors.

The 2DPASS framework consists of three main components. First, a multi-modal feature extraction module that processes both point clouds and their corresponding 2D images using separate backbone networks. The 3D branch employs a sparse convolutional U-Net to extract multi-scale voxel features, while the 2D branch uses a standard CNN encoder to produce multi-scale image feature maps. Second, a Multi-scale Fusion-to-Single (MSFTS) module that fuses 2D and 3D features at multiple scales through point-to-pixel projection using calibration matrices, creating enriched multi-modal representations. Third, a knowledge distillation objective that transfers the multi-modal knowledge from the fused representation back to the pure 3D branch.

The MSFTS module operates at each scale of the encoder hierarchy. For a given scale, 3D point features are projected onto the 2D image plane using known camera intrinsics and extrinsics, and the corresponding 2D features are retrieved via bilinear interpolation. The retrieved 2D features are concatenated with the 3D features and passed through an attention-based fusion layer that learns adaptive weights for each modality. The fused features at all scales are then aggregated into a single enriched representation. During training, the distillation loss minimizes the KL divergence between the logits of the fused branch and the pure 3D branch, ensuring that the 3D network internalizes the multi-modal knowledge. At inference, only the 3D branch is deployed, requiring no 2D input.

To further enhance segmentation accuracy, we introduce a scale-aware auxiliary loss that applies semantic supervision at each scale of the 3D encoder, encouraging the network to produce discriminative features early in the hierarchy. Additionally, we employ class-balanced sampling to address the severe class imbalance common in outdoor LiDAR datasets, where categories such as bicyclists and motorcyclists are significantly underrepresented compared to road surfaces and buildings. These design choices collectively contribute to robust performance across both common and rare object categories.

We evaluate 2DPASS on two major benchmarks: SemanticKITTI (19 classes, 22 sequences) and NuScenes (16 classes, 1000 scenes). On SemanticKITTI, 2DPASS achieves a mean IoU of 72.9% on the test set, surpassing previous state-of-the-art methods including Cylinder3D (68.9%), RPVNet (70.3%), and SPVCNN (67.4%). Notably, 2DPASS demonstrates particularly strong improvements on under-represented categories such as bicyclist (+5.2 IoU), motorcycle (+4.8 IoU), and person (+3.6 IoU), confirming the value of 2D texture priors for distinguishing fine-grained semantic classes. On NuScenes, 2DPASS achieves 79.4% mean IoU on the validation set, establishing a new state of the art.

Ablation studies reveal the contribution of each component. Removing the multi-scale fusion and using only single-scale fusion decreases performance by 1.8 mIoU on SemanticKITTI, confirming the importance of hierarchical feature fusion. Replacing the attention-based fusion with simple concatenation leads to a 1.2 mIoU drop, showing that adaptive modality weighting is beneficial. Disabling the knowledge distillation objective entirely and relying only on joint training results in a 2.5 mIoU decrease, underscoring that explicit distillation is critical for effective transfer of 2D priors to the 3D network.

We have presented 2DPASS, a novel training framework that leverages 2D image priors to enhance 3D LiDAR semantic segmentation through multi-scale fusion and knowledge distillation. By incorporating rich texture and color information from camera images during training, the resulting 3D-only network achieves state-of-the-art performance on SemanticKITTI and NuScenes without requiring 2D input at inference. Our approach demonstrates that the complementary nature of 2D and 3D modalities can be effectively exploited through a training-time fusion-and-distill paradigm, offering a practical path toward deploying high-accuracy point cloud segmentation in resource-constrained autonomous driving systems. Future directions include extending 2DPASS to temporal sequences and exploring self-supervised 2D pretraining for domains with limited labeled image data.