"Neural Radiance Fields (NeRF) are a rapidly growing area of research with wide-ranging applications in computer vision, graphics, robotics, and more." The paper presents Nerfstudio, described as "a modular PyTorch framework" that includes "plug-and-play components for implementing NeRF-based methods." The framework features real-time visualization tools, streamlined pipelines for importing captured in-the-wild data, and export capabilities. The authors developed Nerfacto, their method combining "components from recent papers to achieve a balance between speed and quality," while maintaining flexibility for future modifications.

Since NeRF's introduction in 2020, research has advanced through "few-image training," "explicit features for editing," "surface representations for high-quality 3D mesh exports," and "speed improvements for real-time rendering and training." The core problem is that "despite the growing use of NeRFs, support for development is still rudimentary" because "many papers implement features in their own siloed repository," complicating feature transfer across implementations. Additionally, "while NeRFs solve an inherently visual task, there is a lack of comprehensive and extensible tools for visualizing and interacting with NeRFs." Three design goals guide Nerfstudio: (1) consolidating various NeRF techniques into reusable, modular components; (2) enabling real-time visualization; and (3) providing an end-to-end, easy-to-use workflow for creating NeRFs from user-captured data.

Multiple existing implementations exist, including "the original NeRF codebase, nerf-pytorch, Instant NGP, torch-ngp, and MultiNeRF." The fundamental problem remains: "Due to the lack of consolidation, there exists a significant number of NeRF repositories that focus on improving specific components of specific algorithms." Concurrent efforts include "NeRF-Factory, NerfAcc, MultiNeRF, and Kaolin-Wisp." The critical distinction is that "none of these repositories are as comprehensive as Nerfstudio in delivering our three design goals: modularity, real-time visualization, and end-to-end usability." Nerfstudio is released under an Apache2 license, allowing use by both researchers and companies.

The framework's philosophy reflects specific priorities: "we prefer an implementation that allows for a modularized pythonic non-CUDA method over one that supports a faster, non-modularized CUDA method." This enables simpler interfacing with an extensive visualization ecosystem supporting real-time rendering during test and train with custom camera paths. The framework "focuses on delivering results for real-world data rather than synthetic scenes" to address audiences outside research including those in industry and non-technical users. The design promotes collaboration by "providing a consolidated platform on which people can request for or contribute to new features."

Nerfstudio takes a set of posed images and optimizes for a 3D representation defined by radiance (color), density (structure), and possibly other quantities (semantics, normals, features). The pipeline comprises a DataManager and a Model, where the DataManager handles (1) parsing image formats via a DataParser and (2) generating rays as RayBundles. These rays are passed into a Model which queries Fields and renders quantities. The DataParser is designed for compatibility with arbitrary data formats, supporting mobile apps (Record3D, Polycam, KIRI Engine) and 3D tools (Metashape, Reality Capture) beyond traditional COLMAP, making the framework accessible to scientists, artists, photographers, hobbyists and journalists.

Nerfacto leverages the modular design to integrate ideas from multiple research papers, heavily influenced by MipNeRF-360 with components from NeRF--, Instant-NGP, NeRF-W, and Ref-NeRF. The method employs a piece-wise sampler that samples uniformly up to a fixed distance, then with increasing step sizes, followed by a proposal network sampler from MipNeRF-360 that consolidates samples into contributing regions. Configuration uses 256 initial samples reduced to 96, then 48 through two proposal iterations. For unbounded scenes, L-infinity norm scene contraction (cube-based, aligning with hash encodings) replaces MipNeRF-360's L2 norm. Per-image appearance embeddings handle exposure differences, and Ref-NeRF techniques predict normals.

"In as little as 5K iterations (~2 minutes), our Nerfacto method achieves reasonable quality in contrast to MipNeRF-360 which takes several hours on a TPU with 32 cores." Training for up to 70K iterations (~30 minutes) further improves quality. While Nerfacto falls short of metric results obtained by MipNeRF-360, the authors prioritize efficiency and general usability over optimizing quantitative metrics. A critical finding from ablation studies: "disabling the appearance embeddings leads to an improvement in PSNR and SSIM. However, qualitative results show that the 'w/o app' method results in the production of blurry 'floater' artifacts." This demonstrates that standard metrics can be misleading for real-world NeRF evaluation.

Nerfstudio draws upon existing techniques and proposes a framework that supports a more modularized approach to NeRF development, allows for real-time visualization, and is readily usable with real-world data. The authors "emphasize the importance of utilizing the interactive real-time viewer during training to compensate for imperfect quantitative metrics." The open-source repository has grown to include over 60 contributors and over 3K stars, with extensions like SDFStudio and ArcNerf built upon the framework. Future research directions include development of more appropriate evaluation metrics and integration with other fields.