Point cloud is an important type of geometric data structure. Due to its irregular format, most researchers transform such data to regular 3D voxel grids or collections of images. This, however, renders data unnecessarily voluminous and causes issues. In this paper, we design a novel type of neural network that directly consumes point clouds, which well respects the permutation invariance of points in the input. Our network, named PointNet, provides a unified architecture for applications ranging from object classification, part segmentation, to scene semantic parsing. Though simple, PointNet is highly efficient and effective. Empirically, it shows strong performance on par or even better than state of the art. Theoretically, we provide analysis towards understanding what the network has learned and why the network is robust with respect to input perturbation and corruption.

Typical convolutional architectures require highly regular input data formats, like those of image grids or 3D voxels, in order to perform weight sharing and other kernel optimizations. Since point clouds or meshes are not in a regular format, most researchers typically transform such data to regular 3D voxel grids or collections of images before feeding it to a deep net. This data representation transformation, however, renders the data unnecessarily voluminous — while also introducing quantization artifacts that can obscure natural invariances of the data. For this reason, we focus on a different input representation and design a deep net that directly takes point clouds as input while respecting the permutation invariance of the point set.

Key to our approach is the use of a single symmetric function, max pooling. The network effectively learns a set of optimization functions/criteria that select interesting or informative points of the point cloud and encode the reason why they are informative. The final fully connected layers of the network aggregate these learned optimal values into the global descriptor for the entire shape (for classification) or combined with per-point features for per-point labeling (for segmentation). Our input format is easy to apply rigid or affine transformations to, as each point transforms independently. We provide a theoretical analysis demonstrating the network learns to approximate any continuous set function and showing its robustness to perturbation and corruption via a bounded critical point set.

Prior approaches for 3D deep learning fall into three categories. Volumetric CNNs apply 3D convolutions on voxelized shapes, but are constrained by data sparsity and computation cost of 3D convolutions, with resolution typically limited to 30x30x30. Multi-view CNNs render 3D shapes into 2D images and apply image CNNs, achieving dominant performance on shape classification, but are difficult to extend to 3D tasks like point classification and shape completion. Spectral CNNs on meshes are limited to manifold meshes and are not straightforward to apply to non-isometric shapes. A fundamental challenge across these methods is that none directly operates on the raw point cloud as an unordered set.

Our input is a subset of points from an Euclidean space. It has three main properties. Unordered: Unlike pixel arrays in images or voxel arrays in volumetric grids, a point cloud is a set of points without specific ordering. A network that consumes N 3D point sets needs to be invariant to N! permutations of the input set. Interaction among points: The points come from a space with a distance metric. It means that neighboring points form a meaningful subset and the model needs to be able to capture local structures from nearby points and the combinatorial interactions among local structures. Invariance under transformations: As a geometric object, the learned representation of the point set should be invariant under certain transformations such as rotation and translation.

To make a model invariant to input permutation, three strategies exist: sort input into a canonical order; treat the input as a sequence to train an RNN, but augment training data by all kinds of permutations; use a simple symmetric function to aggregate the information from each point. We argue that sorting is not a natural solution — there does not exist an ordering in high dimensional space that is stable with respect to point perturbations. RNNs also fail to scale to thousands of input elements. We approximate a general function defined on a point set by applying a symmetric function on transformed elements: f({x_1, ..., x_n}) ~ g(h(x_1), ..., h(x_n)) where h is a multi-layer perceptron and g is a composition of a single variable function and a max pooling function. This is our key insight and forms the basis of our PointNet architecture.

The semantic labeling of a point cloud has to be invariant if the point cloud is subject to certain geometric transformations such as rigid transformation. We therefore expect that the learnt representation by our point set is invariant to these transformations. A natural solution is to align all input set to a canonical space before feature extraction. We predict an affine transformation matrix by a mini-network (T-net) and directly apply this transformation to the coordinates of input points. This idea can be further extended to the alignment of feature space, with a regularization loss L_reg = ||I - AA^T||^2 to constrain the feature transformation matrix to be close to orthogonal.

On ModelNet40 3D classification, PointNet achieved 89.2% overall accuracy, representing state-of-the-art performance among methods based on 3D input, outperforming volumetric approaches while being significantly faster than multi-view methods. On ShapeNet part segmentation, PointNet achieved 83.7% mean IoU, improving 2.3% over prior methods. On Stanford 3D semantic parsing, PointNet achieved 47.71% mean IoU versus 20.12% for handcrafted feature baselines. Ablation studies validated that max pooling substantially outperformed alternatives, input and feature transformations provided incremental gains, and robustness tests confirmed stability against missing points, outliers, and perturbations. PointNet demonstrated 141x more efficient than MVCNN and 8x more efficient than volumetric approaches in FLOPs/sample, with linear O(N) complexity in point count.

We have proposed PointNet, a novel deep net that directly consumes point clouds. Our network provides a unified approach to a number of 3D recognition tasks including object classification, part segmentation, and semantic segmentation while obtaining on par or better results than state of the art on standard benchmarks. We also provide theoretical analysis and visualizations towards understanding the network. The key contributions are the novel architecture design, comprehensive empirical validation, theoretical analysis revealing learned representations, and interpretable visualizations showing that the network learns to summarize a shape by a sparse set of key points. The general applicability extends beyond 3D point sets to any unordered set processing domain.