We present a network architecture for processing point clouds that directly operates on a collection of points represented as a sparse set of samples in a high-dimensional lattice. Our core building block is a Bilateral Convolutional Layer (BCL) that performs convolutions only on the occupied parts of the lattice, maintaining computational efficiency through hash table-based indexing. The architecture enables hierarchical feature learning and joint 2D-3D reasoning within an end-to-end trainable framework. We demonstrate state-of-the-art performance on 3D segmentation benchmarks including ShapeNet part segmentation and RueMonge2014 facade labeling.

Processing 3D point clouds with neural networks presents unique challenges. Unlike images, point clouds are irregular, unordered, and of varying density. Common approaches either voxelize the space, which introduces discretization artifacts and scales cubically with resolution, or use point-based networks like PointNet, which process points independently and may miss local geometric structure. We propose SPLATNet, which projects points onto a high-dimensional permutohedral lattice and performs sparse convolutions, combining the regularity of grid-based processing with the efficiency of sparse representations.

PointNet and PointNet++ pioneered direct point cloud processing using shared MLPs and set abstraction layers. Voxel-based methods such as VoxNet and OctNet apply 3D convolutions on discretized grids, with OctNet using octree structures to reduce memory. Bilateral filtering has been used in image processing for edge-preserving smoothing, and the permutohedral lattice provides an efficient implementation. Our work extends this by making the lattice-based bilateral operations learnable and composable into deep networks, rather than using them as fixed filtering steps.

SPLATNet is built on Bilateral Convolutional Layers (BCL). Each BCL operates through three steps: Splat — project input point features onto the vertices of a permutohedral lattice using barycentric interpolation; Convolve — apply learnable filters on the lattice vertices; Slice — interpolate the filtered features back to the original point positions. The lattice structure allows convolutions to be performed on a regular grid while respecting the sparse, irregular nature of the input point cloud. Hash tables index the occupied lattice vertices, ensuring that computation scales with the number of points rather than the volume of the space.

SPLATNet_3D stacks T successive BCL layers with progressively coarser lattice scales. Starting from a fine lattice at scale lambda_0, each subsequent layer operates at half the spatial resolution (lambda_0/2, lambda_0/4, ...), enabling hierarchical feature extraction with increasing receptive fields. This is analogous to the pooling layers in standard CNNs but operates on the irregular point cloud via the lattice representation. The lattice dimensions are typically based on 3D position coordinates (XYZ), but can flexibly incorporate additional dimensions such as color or normal vectors.

SPLATNet_2D-3D extends the framework to jointly reason over 2D image features and 3D point cloud features. The pipeline consists of: (1) a 2D CNN (DeepLab) extracts image features; (2) a BCL_2D-3D projects 2D features onto the 3D lattice using known camera parameters; (3) 3D features and projected 2D features are concatenated and refined via 3D BCL layers; (4) a BCL_3D-2D back-projects refined features to the 2D image domain; (5) a second 2D CNN produces final predictions. This bidirectional information flow allows 3D geometric context to enhance 2D predictions and vice versa.

We evaluate on multiple benchmarks. On RueMonge2014 facade segmentation, SPLATNet_3D achieves 65.4 IoU (vs. OctNet 59.2) and SPLATNet_2D-3D reaches 69.8 IoU (vs. Autocontext 62.9) with dramatically better efficiency (0.06 min vs. 16 min). On ShapeNet part segmentation, SPLATNet_3D achieves 82.0 class average mIoU (84.6 instance) and SPLATNet_2D-3D reaches 83.7 class average (85.4 instance), outperforming PointNet++ (81.9 class average). Processing speed is 9.4 shapes/sec for SPLATNet_3D compared to 2.7 for PointNet++, demonstrating the computational advantages of the lattice representation.

We have presented SPLATNet, a neural network architecture for point cloud processing using sparse bilateral convolutions on permutohedral lattices. The Bilateral Convolutional Layer provides an efficient and flexible building block that enables hierarchical feature learning, arbitrary lattice dimensions, and seamless 2D-3D integration. Our results on RueMonge2014 and ShapeNet demonstrate state-of-the-art accuracy with superior computational efficiency. The lattice-based approach opens new possibilities for processing irregular data structures with the regularity and efficiency of convolutional networks.