Volumetric neural rendering methods like NeRF generate high-quality view synthesis results but are optimized per-scene leading to prohibitive reconstruction time. In this paper, we propose Point-NeRF, a novel point-based neural radiance field representation that uses neural 3D point clouds, with associated neural features, to model a volumetric radiance field. Point-NeRF can be rendered efficiently by aggregating neural point features near scene surfaces, in a ray marching-based rendering pipeline. Point-NeRF can be initialized via direct deep network inference, yielding a reasonable radiance field for novel view synthesis, which can be further fine-tuned to surpass the neural radiance field quality of NeRF, with 30x faster training time. Point-NeRF further introduces novel pruning and growing mechanisms for the neural point cloud, to handle errors and outliers in the reconstructed point cloud, achieving state-of-the-art results on the DTU, NeRF Synthetics, ScanNet, and Tanks and Temples datasets.

Modeling real scenes from image data for photo-realistic novel view synthesis is a long-standing and fundamental problem in computer vision and graphics. While NeRF and its extensions have achieved great success, these methods often reconstruct radiance fields using global MLPs for the entire space through ray marching. This leads to long reconstruction times due to the slow per-scene network fitting and the unnecessary sampling of vast empty space. These limitations make NeRF-based approaches impractical for real-world applications that require efficient scene reconstruction.

We address these challenges by introducing Point-NeRF, a point-based neural radiance field representation that uses 3D neural points. Unlike NeRF that purely depends on per-scene fitting, Point-NeRF can be effectively initialized via a feed-forward deep neural network, pre-trained across scenes. Each neural point encodes the local 3D scene geometry and appearance around it. By leveraging point clouds approximating scene geometry, our approach avoids ray sampling in empty space and can be rendered efficiently. For any 3D location, we propose to use an MLP network to aggregate the neural points in its neighborhood to regress the volume density and view-dependent radiance at that location.

Our learning framework leverages deep multi-view stereo techniques for initial field generation: a cost-volume-based network predicts depth and a CNN extracts 2D feature maps. These neural points from multiple views are combined as a neural point cloud, which forms a point-based radiance field of the scene. To further handle the inevitable holes and outliers in the initial point cloud, we introduce novel point growing and pruning mechanisms. The growing mechanism progressively grows new points near the point cloud boundary based on the local scene geometry modeled by our representation. The pruning mechanism utilizes point confidence values to remove unnecessary outlier points. These mechanisms effectively improve our final reconstruction and rendering quality.

Traditional and neural methods study various 3D representations including volumes, point clouds, meshes, depth maps, and implicit functions. Recently, various neural scene representations have been presented, advancing the state of the art in novel view synthesis and realistic rendering, with volumetric neural radiance fields (NeRFs) producing high fidelity results. Point-NeRF combines volumetric radiance fields with point clouds. We distribute fine-grained neural points to model complex local scene geometry and appearance, leading to better rendering quality than NeRF. Voxel grids with per-voxel neural features offer local neural radiance representation, yet our point-based representation adapts better to actual surfaces, leading to better quality. Also, we directly predict good initial neural point features, bypassing the per-scene optimization that is required by most voxel-based methods.

Multi-view 3D reconstruction uses structure-from-motion and multi-view stereo (MVS) techniques. Point clouds are often the direct output from MVS or depth sensors, though they are usually converted to meshes for rendering and visualization. Point-based rendering traditionally uses rasterization-based point splatting. However, reconstructed point clouds often have holes and outliers that lead to artifacts in rendering. Point-based neural rendering methods address this by splatting neural features and using 2D CNNs to render them. In contrast, our point-based approach utilizes 3D volume rendering, leading to significantly better results than previous point-based methods.

NeRFs have demonstrated remarkably high-quality results for novel view synthesis. They have been extended to achieve dynamic scene capture, relighting, appearance editing, fast rendering, and generative models. Most methods follow the original NeRF framework, training per-scene MLPs. Point-NeRF uses spatially varying neural features in scene points. This localized representation can model more complex scene content than pure MLPs that have limited network capacity. Regarding generalizable methods, PixelNeRF and IBRNet aggregate multi-view 2D image features at every sampled ray point. In contrast, we leverage features in 3D neural points around the scene surface to model radiance fields. MVSNeRF can achieve very fast voxel-based radiance field reconstruction, however, its prediction network requires a fixed number of three small-baseline images as input and thus can only efficiently reconstruct local radiance fields.

Physically-based volume rendering can be numerically evaluated via differentiable ray marching. Specifically, a pixel's radiance can be computed by marching a ray through the pixel, sampling M shading points, and accumulating radiance using volume density. A radiance field represents volume density and view-dependent radiance at any 3D location. NeRF proposes to use a multi-layer perceptron (MLP) to regress such radiance fields. We propose Point-NeRF that instead utilizes a neural point cloud to compute the volume properties, allowing for faster and higher-quality rendering.

The neural point cloud is denoted as P = {(p_i, f_i, γ_i)}, where each point i is located at position p_i and associated with a neural feature vector f_i encoding local scene content. We also assign each point a confidence value γ_i ∈ [0,1] that represents how likely that point is being located near an actual scene surface. Given any 3D location x, K neighboring neural points within radius R are queried. Our point-based radiance field can be abstracted as a neural module that regresses volume density σ and view-dependent radiance r at any shading location x from its neighboring neural points.

For per-point processing, we use an MLP F to process each neighboring neural point to predict a new feature vector for the shading location x. This expresses a local 3D function that outputs the specific neural scene description at x, modeled by the neural point in its local frame. The usage of relative position (x − p) makes the network invariant to point translation for better generalization. For view-dependent radiance regression, we use standard inverse distance weighting to aggregate the neural features regressed from K neighboring points to obtain a single feature. Then an MLP R regresses the view-dependent radiance from this feature given a viewing direction. The inverse-distance weight makes closer neural points contribute more. In addition, we use the per-point confidence γ in this process, giving the network the flexibility of rejecting unnecessary points through optimization with a sparsity loss.

To compute volume density σ at x, we follow a similar multi-point aggregation. However, we first regress a density σ_i per point using an MLP T and then do inverse distance-based weighting. Thus, each neural point directly contributes to the volume density, and point confidence γ_i is explicitly associated with this contribution. Unlike previous neural point-based methods that rasterize point features and then render them with 2D CNNs, our representation and rendering are entirely in 3D. By using a point cloud that approximates the scene geometry, our representation naturally and efficiently adapts to scene surfaces and avoids sampling shading locations in empty scene space.

The initial point cloud, features, and confidence values are predicted via feed-forward neural networks for efficient reconstruction. We leverage deep MVS methods to generate 3D point locations using cost volume-based 3D CNNs. For each input image at viewpoint q, a plane-swept cost volume is built by warping 2D features from neighboring viewpoints, then depth probability is regressed using deep 3D CNNs. Since the depth probabilities describe the likelihood of the point being on the surface, we tri-linearly sample the depth probability volume to obtain the point confidence γ_i at each point. We use a 2D CNN G_f to extract neural 2D image feature maps, with a VGG network architecture with three downsampling layers. We combine intermediate features at different resolutions as f_i, providing a meaningful point description that models multi-scale scene appearance.

Per-scene optimization improves the radiance field, but initial point clouds often contain holes and outliers that degrade the rendering quality. For point pruning, we utilize the confidence values γ_i that describe whether a neural point is near a scene surface. The point confidence is directly related to the per-point contribution in volume density regression; as a result, low confidence reflects low volume density in a point's local region indicating that it is empty. Points with γ_i < 0.1 are pruned every 10K iterations. Additionally, a sparsity loss on point confidence is imposed, which forces the confidence value to be close to either zero or one, facilitating clear-cut pruning decisions.

We also propose a novel technique to grow new points to cover missing scene geometry in the original point cloud. Unlike point pruning that directly utilizes information from existing points, growing points requires recovering information in empty regions where no point exists. Our method progressively grows points near the point cloud boundary based on the local scene geometry modeled by our Point-NeRF representation. New points are identified using per-ray shading locations with highest opacity along each ray. We identify the shading location with the highest opacity along the ray and compute its distance to the closest neural point. For a marching ray, we grow a neural point at the high-opacity location if both opacity and distance thresholds are exceeded. By repeating this growing strategy, our radiance field can be expanded to cover missing regions in the initial point cloud.

We combine point clouds from multiple viewpoints to obtain our final neural point cloud. We train the point generation networks along with the representation networks, from end to end with a rendering loss. This allows our generation modules to produce reasonable initial radiance fields. It also initializes the MLPs in our Point-NeRF representation with reasonable weights, significantly saving the per-scene fitting time. Our pipeline also supports using point clouds from other approaches like COLMAP, where our model (excluding the MVS network) can still provide meaningful initial neural features for each point. Per-scene optimization then combines rendering loss and sparsity loss, with point growing and pruning performed every 10K iterations to achieve the final high-quality reconstruction.

Quantitative results on the DTU testing set compare Point-NeRF with PixelNeRF, IBRNet, MVSNeRF, and NeRF using PSNR, SSIM, and LPIPS metrics. Fine-tuning results after 10K iterations achieve the best SSIM and LPIPS, two out of the three metrics, significantly better than MVSNeRF and NeRF. While IBRNet produces slightly better PSNRs, our final renderings in fact recover more accurate texture details and highlights. However, IBRNet is also more expensive to fine-tune, taking 1 hour — 5x longer than ours for the same iterations. This is because IBRNet utilizes a large global CNN, whereas Point-NeRF leverages local point features with small MLPs that are easier to optimize.

On the NeRF Synthetic dataset, though trained solely on DTU, our network generalizes well to novel datasets that have completely different camera distributions. Our results at 20K iterations already outperformed IBRNet's converged results with better PSNR, SSIM, and LPIPS. Results at 20K iterations are quantitatively very close to NeRF's results trained with 200K iterations. Point-NeRF at 20K is optimized for only 40 minutes, which is at least 30x faster than the 20+ hours optimization time taken by NeRF. Converging to 200K iterations produces significantly better results than NeRF, NSVF, and all other comparison methods. As shown in visualizations, our 200K results contain the most geometry and texture details. Attributed to the point growing technique, our method is the only one that can fully recover details like the thin rope structure.

On Tanks and Temples and ScanNet datasets, quantitative comparisons show Point-NeRF outperforms NSVF on both datasets across PSNR, SSIM, and LPIPS metrics. Point-NeRF can also be used to convert standard point clouds reconstructed by other techniques to point-based radiance fields. Testing on the full NeRF Synthetic dataset using COLMAP-reconstructed point clouds shows that even from this low-quality point cloud, our final results are still of very high quality with very high SSIM and LPIPS numbers compared to all other methods. This demonstrates that Point-NeRF is robust to the quality of the initial point cloud and its growing and pruning mechanisms can effectively compensate for imperfect initialization.

Ablation studies demonstrate the effectiveness of the proposed mechanisms. Point growing and pruning techniques are very effective, significantly improving the reconstruction results in both cases. An extreme example demonstrates that starting from a very sparse point cloud with only 1000 points sampled from our original point reconstruction, our approach can progressively grow new points from the point cloud boundary until filling the entire scene surface through iterations. Compared with point-based neural rendering (NPBG), our results are significantly better than the previous state-of-the-art point-based rendering methods. NPBG can only produce blurry rendering results with their rasterization and 2D CNN framework. In contrast, we leverage volumetric rendering technique with neural radiance fields, leading to photo-realistic results.

We present Point-NeRF, a novel point-based volumetric radiance field representation that combines the classical point cloud representation with neural radiance fields. We reconstruct a good initialization of Point-NeRF directly from input images via direct network inference and show that we can efficiently fine-tune this initialization for a scene. This enables highly efficient Point-NeRF reconstruction with only 20-40 min per-scene optimization, leading to rendering quality comparable to and even surpassing NeRF that requires substantially longer training time (20+ hours).

Novel growing and pruning techniques significantly improve results and robustness. Our Point-NeRF successfully combines the advantages from both classical point cloud representation and neural radiance field representation, making an important step towards a practical scene reconstruction solution with high efficiency and realism. The point-based design enables natural handling of scene surfaces at varying scales and the learned neural features provide expressive local appearance modeling beyond what global MLPs can achieve. We believe this direction of combining explicit geometric structures with neural implicit representations holds great promise for future 3D vision and graphics applications.