"Humans describe the physical world using natural language to refer to specific 3D locations based on a vast range of properties: visual appearance, semantics, abstract associations, or actionable affordances." In this work the authors propose Language Embedded Radiance Fields (LERF), a method for grounding language embeddings from off-the-shelf models like CLIP into NeRF, which enable open-ended language queries in 3D. LERF learns a dense, multi-scale language field by volume rendering CLIP embeddings along training rays with multi-view consistency. After optimization, LERF generates 3D relevancy maps for language prompts in real-time without requiring region proposals or masks.

"Neural Radiance Fields (NeRFs) have emerged as a powerful technique for capturing photorealistic digital representations of intricate real-world 3D scenes." However, "the immediate output of NeRFs is nothing but a colorful density field, devoid of meaning or context." The authors propose enabling natural language interaction with 3D scenes by grounding CLIP embeddings within NeRF structures. LERF supports diverse query types — from visual properties to abstract concepts — without fine-tuning CLIP or relying on region proposals. This bridges the gap between photorealistic 3D reconstruction and semantic scene understanding.

Prior work covers open-vocabulary detection in 2D, 2D feature distillation into NeRF, and 3D language grounding approaches. Methods like Distilled Feature Fields (DFF) distill features from 2D vision models into NeRF but operate at a single scale, missing the multi-scale nature of language concepts. 3D-SLR and similar approaches use region proposals or pre-extracted bounding boxes, creating a bottleneck. LERF is positioned as offering a "dense, volumetric interface for 3D text queries" distinct from prior sparse or single-scale methods, without requiring any pre-segmentation or region proposal stages.

"To render language embeddings into an image, we adopt a similar technique as prior work to render language embeddings along a ray." Since LERF is a field over volumes, not points, the method must also define a scale parameter. The scale is defined proportionally: s(t) = s_img * f_xy / t, representing a frustum along each ray where s_img is the physical scale in image space and f_xy is the focal length. This enables multi-scale language supervision — the same 3D point can hold different language meanings at different scales, mirroring how humans describe objects at varying levels of granularity.

LERF supervises the language field using image pyramids at multiple scales. For each training ray, CLIP embeddings are computed from image crops at three scales centered on the pixel corresponding to that ray. The rendered language embedding is trained to match all three simultaneously. Additionally, DINO features are used as a regularizer to enforce spatial smoothness in the language field. The DINO regularization helps because DINO features capture local visual similarity that CLIP's global pooling may miss, providing complementary geometric grounding for the language embeddings.

At inference time, LERF generates 3D relevancy maps for arbitrary text queries. The query process uses cosine similarity between the rendered CLIP embedding and the text query embedding. To normalize relevancy scores, the method employs canonical phrase comparisons — computing similarity against a set of generic text prompts (e.g., "object," "things," "texture") and normalizing the target query's score relative to these baselines. This produces a per-point relevancy score that can be volume-rendered into 2D relevancy maps or queried directly in 3D, enabling applications in robotics, scene interaction, and spatial reasoning.

LERF is evaluated on 13 real-world scenes spanning indoor and outdoor environments. For localization tasks, LERF demonstrates superior performance over LSeg (3D) and competitive results against OWL-ViT, a 2D open-vocabulary detector. Ablation studies confirm the importance of both DINO regularization and multi-scale training — removing either component degrades localization accuracy significantly. The method handles diverse query types including object names ("mug"), visual attributes ("shiny"), abstract concepts ("something to write with"), and even text recognition ("SUNKIST"), demonstrating the breadth of open-vocabulary 3D querying.

"We present LERF, a novel method of fusing raw CLIP embeddings into a NeRF in a dense, multi-scale fashion without requiring region proposals or fine-tuning." The method demonstrates that language understanding can be seamlessly integrated into 3D scene representations, enabling real-time open-ended queries. Acknowledged limitations include CLIP's bag-of-words behavior (difficulty with compositional queries like "the red cup to the left of the book"), spatial reasoning difficulties, and dependency on NeRF reconstruction quality — poorly reconstructed regions yield unreliable language embeddings.