We present VoteNet, an end-to-end 3D object detection network based on a synergy of deep point set networks and Hough voting. Current 3D object detection methods either rely heavily on 2D detectors in projection-based approaches or convert point clouds to regular grids (voxels), sacrificing the sparsity and geometric detail of raw point data. Our method directly processes raw point cloud data without 2D or grid dependencies. The key challenge is that object centroids are often far from any surface point in sparse 3D scans. VoteNet addresses this by "generating votes from learned features to reach object centers, creating clusters that enable accurate bounding box prediction." The approach achieves state-of-the-art results on ScanNet and SUN RGB-D using only geometric information.

The goal is estimating oriented 3D bounding boxes and semantic classes from point clouds. Point clouds offer accurate geometry and robustness to illumination changes but are irregular, making standard CNNs unsuitable. Most prior methods rely on 2D detectors or voxelization, which sacrifice sparsity advantages or geometric detail. VoteNet introduces a point cloud-focused framework operating directly on raw data. The core challenge involves predicting bounding box parameters when object centroids are distant from surface points. The solution employs Hough voting: "generating new points that lie close to object centers, which can be grouped and aggregated to generate box proposals."

The related work spans three domains. 3D Object Detection: voxel-based methods (e.g., VoxNet) discretize point clouds but suffer resolution limitations due to cubic memory growth. Projection-based methods (e.g., Frustum PointNets) depend on 2D detectors and are constrained by their accuracy. Point-based methods like PointNet++ process raw points but have not been applied to direct 3D detection without 2D assistance. Hough Voting: Classical Hough voting has long been used in object recognition, but traditional implementations use hand-crafted features. VoteNet reformulates classical Hough voting within modern deep learning frameworks for direct point cloud processing.

VoteNet comprises two stages: voting generation and proposal generation. The backbone uses PointNet++ for hierarchical feature learning, producing seed points with associated features. Each seed point independently generates a vote via an MLP that outputs 3D coordinate offsets and feature offsets. The coordinate offset predicts how far and in which direction the seed point should "move" to reach its parent object's centroid. Ideally, votes from surface points of the same object cluster near its center, creating concentrated vote clusters that indicate object locations even when the center itself has no point cloud data.

The proposal module uses farthest point sampling on the vote points to select cluster centers, then groups nearby votes by spatial proximity. Grouped vote features are aggregated through learned PointNet layers to predict objectness scores, 3D bounding box parameters (center, size, orientation), and semantic class labels. The loss function combines voting regression loss, objectness classification loss, box estimation loss, and semantic classification loss with weighted contributions. This end-to-end differentiable pipeline allows the voting and proposal stages to be jointly optimized, unlike classical Hough voting which uses separate, non-differentiable feature extraction and voting stages.

Evaluation on SUN RGB-D and ScanNetV2 shows VoteNet surpasses prior methods by 3.7 and 18.4 mAP respectively, using geometry only while competitors used RGB data. Analysis experiments demonstrate: voting provides substantial gains (5-13 mAP), with benefits strongest when object points are distant from centroids; learned PointNet aggregation outperforms manual feature pooling; and the model is 4x more compact and 20x faster than prior art. Qualitative results show robust performance despite clutter, partiality, and scanning artifacts, though thin objects remain challenging without RGB information.

VoteNet successfully combines Hough voting with deep learning for direct 3D object detection in point clouds. The synergy between classical geometric voting and modern deep feature learning achieves strong results using only geometry, demonstrating that raw point cloud data, when properly leveraged through voting mechanisms, contains sufficient information for accurate 3D detection. The approach suggests potential for extending to additional applications like 6-DoF pose estimation and 3D instance segmentation. The success of this classical-meets-modern design philosophy opens new directions for revisiting other established geometric algorithms within deep learning frameworks.