Neural Radiance Fields (NeRF) is a popular view synthesis technique that represents a scene as a continuous volumetric function, parameterized by multilayer perceptrons. While NeRF has been demonstrated to generate photorealistic renderings of complex scenes, it is only capable of producing plausible renditions for scenes with fine geometric structures that exhibit smooth appearance variation. NeRF often fails to accurately capture and reproduce the appearance of glossy surfaces, because the scene's true radiance function varies quickly with view direction, especially around specular highlights. We address this limitation by replacing NeRF's parameterization of view-dependent appearance with a representation of reflected radiance and structuring this function using a collection of spatially-varying scene properties. We additionally propose a regularization on normal vectors that improves the realism of specular reflections. Our model, which we call Ref-NeRF, achieves state-of-the-art rendering quality with an interpretable internal representation that enables scene editing.

Neural Radiance Fields (NeRF) have emerged as a powerful representation for novel view synthesis, encoding a scene as a continuous function that maps a 3D position and 2D viewing direction to a volume density and emitted color. Recent extensions such as mip-NeRF have further improved quality by reasoning about the volume of each sampled conical frustum. Despite these advances, NeRF and its variants often produce poor renderings of glossy and reflective objects. The scene's true radiance function varies quickly with view direction around specular highlights, and NeRF is only able to accurately render the appearance of scene points from the specific viewing directions observed in the training images — its interpolation of glossy appearance from novel viewpoints is poor.

This poor interpolation leads to a secondary problem: NeRF tends to "fake" specular reflections using isotropic emitters inside the object instead of view-dependent radiance emitted by points at the surface. The resulting geometry is "foggy" — the density field becomes diffuse and spread out rather than concentrated at true surfaces. This not only degrades rendering quality but also prevents extraction of accurate surface normals, which limits applications such as relighting and material editing. The root cause is that NeRF's parameterization of view-dependent color as a function of the viewing direction produces a function that is poorly suited for interpolation, since the same specular highlight appears at different viewing directions for surface points with different orientations.

Our key insight is that structuring NeRF's representation of view-dependent appearance can make the underlying function simpler and easier to interpolate. We observe that for a surface with a rotationally-symmetric BRDF, the view-dependent radiance can be expressed as a function of the reflection of the viewing direction about the surface normal. In this reflected radiance parameterization, the function is constant across the scene for a flat mirror — because it is unaffected by changes in surface orientation. We therefore propose Ref-NeRF, which restructures NeRF's outgoing radiance as a function of the reflection direction, and further decomposes appearance into diffuse and specular components controlled by spatially-varying material properties including a roughness parameter.

Neural scene representations have gained significant attention for their ability to encode complex 3D scenes. NeRF represents a scene using a multilayer perceptron (MLP) that maps 3D coordinates and viewing directions to density and color. Subsequent works have addressed various limitations: mip-NeRF mitigates aliasing by casting cones instead of rays and reasoning about the volume of each sample; other methods improve training speed, rendering efficiency, and handling of unbounded scenes. However, none of these works specifically address the fundamental issue of representing view-dependent appearance for specular and glossy surfaces.

Classical approaches to view-dependent appearance in computer graphics rely on explicit BRDF models such as Phong, Cook-Torrance, and microfacet models that describe how light interacts with surfaces. These models parameterize reflectance as a function of surface normal, incoming light direction, and outgoing view direction. Recent neural rendering methods attempt to learn such reflectance properties implicitly, but most conflate geometry and appearance in a single representation, making it difficult to achieve physically plausible rendering of specular surfaces. Some works decompose appearance into intrinsic properties (albedo, normal, roughness), but these typically require known lighting conditions or multi-illumination captures. Our approach bridges these two lines of research by structuring the neural representation using insights from physically-based rendering without requiring explicit BRDF fitting or known illumination.

Accurate surface normal estimation is critical for realistic rendering of specular objects. In volumetric representations, normals can be computed as the negative gradient of the density field, but NeRF's density fields are often noisy and poorly concentrated at true surfaces, yielding inaccurate normals. Some methods directly predict normals using an auxiliary MLP head or enforce surface-like density distributions through regularization. Our work combines both strategies: we predict normals via the MLP and regularize them to be consistent with density-derived normals, while additionally introducing an orientation loss that encourages density to concentrate at surfaces rather than forming diffuse clouds.

Our approach builds upon mip-NeRF and modifies its representation of view-dependent appearance. The standard NeRF model queries a directional MLP with the viewing direction to produce the output color. We make three key modifications: (1) we replace the viewing direction with the reflection direction as input to the directional MLP; (2) we introduce Integrated Directional Encoding (IDE) that encodes the reflection direction along with a roughness parameter; and (3) we decompose the output color into diffuse and specular components, where the diffuse color depends only on position while the specular color depends on the encoded reflection direction.

In standard NeRF, the directional MLP takes the viewing direction as input. For a surface with a rotationally-symmetric BRDF, the outgoing radiance at a surface point is a function of the reflection of the viewing direction about the local surface normal. We compute the reflection direction as: ω̂_r = 2(ω̂_o · n̂)n̂ − ω̂_o, where ω̂_o is the outgoing (viewing) direction and n̂ is the predicted surface normal. This reparameterization has a crucial advantage: for a perfectly specular flat surface, the reflected radiance function is constant across the entire surface, since all surface points with the same normal reflect the same environment. In contrast, with the viewing direction parameterization, the same specular highlight requires the MLP to learn a complex mapping that varies with surface orientation.

To obtain the surface normal n̂ needed for computing the reflection direction, we add an auxiliary output head to the spatial MLP that directly predicts a normal vector n̂' at each 3D point. This predicted normal is used for computing the reflection direction, rather than the density gradient normal n̂ = −∇σ/||∇σ||, because the gradient-derived normal is noisy for NeRF's diffuse density fields. However, we add a regularization loss that encourages the predicted normal to be consistent with the density gradient, thereby coupling the two and improving the overall quality of both the geometry and the appearance.

Standard NeRF encodes the viewing direction using sinusoidal positional encoding, which maps the direction to a fixed set of frequency bands. This encoding does not account for material roughness — a rough surface and a smooth surface with the same reflection direction receive identical encodings. We propose Integrated Directional Encoding (IDE), which encodes directions using spherical harmonics {Y_l^m} and represents the distribution of reflected directions using a von Mises-Fisher (vMF) distribution centered at the reflection direction with concentration parameter κ = 1/ρ, where ρ is a spatially-varying roughness parameter predicted by the MLP.

The IDE computes the expected value of a set of spherical harmonics under this vMF distribution. The key mathematical result is that the expected value of each spherical harmonic of degree l under a vMF distribution is simply the spherical harmonic evaluated at the mean direction, multiplied by an attenuation factor: A_l(κ) ≈ exp(−l(l+1) / 2κ). Intuitively, increasing the roughness of a material by lowering κ corresponds to attenuating the encoding's spherical harmonics with high orders. This means that a perfectly smooth surface (κ → ∞) retains all frequency components, while a very rough surface (κ → 0) only retains low-frequency components. This directly mirrors how glossy BRDFs act as low-pass filters on incident illumination.

The final output color is decomposed as c = γ(c_d + s ⊙ c_s), where c_d is the diffuse color produced by the spatial MLP (depending only on position), s is a specular tint that modulates the color of specular reflections, c_s is the specular color produced by the directional MLP (depending on the IDE-encoded reflection direction), and γ is a tone mapping function. This decomposition separates position-dependent diffuse appearance from view-dependent specular appearance, yielding an interpretable internal representation where each component has a clear physical meaning.

We introduce two complementary regularization losses that address NeRF's tendency to produce foggy geometries. The first is the orientation loss: R_o = Σ_i w_i max(0, n̂_i' · d̂)², which penalizes back-facing normals that contribute to the rendered color. Specifically, it penalizes sample points whose density-gradient normals point away from the camera while simultaneously having non-negligible rendering weights. This loss prevents the model from explaining specular highlights as emitters hidden beneath a semi-transparent surface, forcing density to concentrate at true surfaces where normals face the camera.

The second is the predicted normal loss: R_p = Σ_i w_i ||n̂_i − n̂_i'||², which encourages the MLP-predicted normals to be consistent with the density-gradient normals. While we use the predicted normals n̂ for computing reflection directions (because they are smoother), we want them to agree with the gradient-derived normals n̂' that are geometrically grounded in the density field. This loss creates a bidirectional coupling: the predicted normals guide the density field toward better geometry, while the density gradients anchor the predicted normals to the actual volumetric structure. Together, these two losses significantly improve both the quality of rendered normals and the sharpness of the density field.

The total training loss combines the photometric reconstruction loss with the two regularization terms: L = L_recon + λ_oR_o + λ_pR_p, where λ_o and λ_p are hyperparameters controlling the relative weight of each regularization term. The reconstruction loss L_recon measures the difference between rendered and ground-truth pixel colors, following the standard mip-NeRF formulation. In practice, we find that moderate values of λ_o and λ_p work well across a range of scenes, and our ablation studies demonstrate that both losses are necessary for achieving the best rendering quality.

We evaluate Ref-NeRF on three types of datasets. First, we introduce the Shiny Blender dataset containing six glossy objects — car, ball, helmet, teapot, toaster, and coffee — rendered with 100 training and 200 test views under realistic lighting with complex specular reflections. Second, we evaluate on the standard Blender dataset from the original NeRF paper, which contains eight synthetic scenes with varying degrees of specularity. Third, we test on real captured scenes including a sedan, garden spheres, and a toy car, demonstrating generalization to real-world conditions. We compare against NeRF, mip-NeRF, NeRF++, and several variants of our method.

On the Shiny Blender dataset, Ref-NeRF achieves a PSNR of 35.96 dB compared to mip-NeRF's 29.76 dB, representing a substantial improvement of over 6 dB. The SSIM improves from 0.928 to 0.966, and LPIPS decreases significantly. Critically, normal accuracy improves dramatically: Ref-NeRF achieves a mean angular error (MAE) of 18.38 degrees versus mip-NeRF's 60.38 degrees — a 69% improvement. On the standard Blender dataset, Ref-NeRF reaches 33.99 dB PSNR with a 35% improvement in normal accuracy over mip-NeRF, demonstrating that the structured representation benefits even scenes that are not predominantly specular.

Ablation studies demonstrate the importance of each component. Removing the reflection direction parameterization and reverting to viewing direction drops PSNR from 35.96 to 29.47 dB on Shiny Blender, confirming that the reparameterization is the most impactful change. Omitting the orientation loss severely degrades normal quality, with MAE increasing from 18.38 to 52.56 degrees, showing that the orientation loss is critical for preventing foggy geometry. Removing the predicted normal loss modestly reduces performance, while each structural component — roughness, diffuse color, and specular tint — contributes measurably to the overall rendering quality. These results validate that the structured representation, the IDE, and the regularization losses are all necessary components of the full system.

Beyond quantitative improvements, Ref-NeRF's interpretable internal representation enables practical scene editing applications. Because the model explicitly represents diffuse color, specular tint, roughness, and surface normals as separate components, users can modify material properties after training — for example, changing the roughness of a surface to make it more or less glossy, or altering the diffuse color while preserving specular reflections. These edits produce physically plausible results because each component corresponds to a meaningful physical quantity. This capability distinguishes Ref-NeRF from methods that encode appearance as an opaque, entangled feature vector.

We have shown that prior neural representations for view synthesis fail to accurately represent and render scenes with specularities and reflections. We proposed Ref-NeRF, which addresses this by reparameterizing the outgoing radiance as a function of the reflection direction rather than the viewing direction, encoding this direction with an Integrated Directional Encoding that accounts for material roughness, and regularizing surface normals to concentrate density at true surfaces. These modifications yield state-of-the-art rendering quality on scenes with complex specular appearance while producing an interpretable representation that enables practical scene editing.

While Ref-NeRF significantly improves upon previous top-performing neural scene representations for view synthesis, it requires increased computation — making our model roughly 25% slower than mip-NeRF. The method also does not explicitly model interreflections or non-distant illumination, so our improvement upon mip-NeRF is reduced in scenes with strong near-field lighting or complex light transport. Future work could integrate more sophisticated light transport models and extend the approach to handle dynamic scenes and relighting. The structured, physically-motivated representation of Ref-NeRF provides a solid foundation for such extensions, as each component can be independently refined or replaced with more physically accurate models.