We introduce the first system for physically-based neural inverse rendering from multi-viewpoint videos of propagating light. Our approach extends neural radiance caching through time-resolved techniques, storing infinite-bounce radiance at any point from any direction. The model accounts for direct and indirect light transport, enabling state-of-the-art 3D reconstruction with strong indirect light from flash lidar systems. Additional capabilities include view synthesis of propagating light, decomposition of measurements into direct and indirect components, and multi-view time-resolved relighting.

Ultrafast imaging systems like lidar illuminate scenes with light pulses and capture backscattered echoes from propagating wavefronts. Precise measurement of speed-of-light time delays enables 3D reconstruction, making lidar systems popular in autonomous driving, augmented reality, and remote sensing applications. Conventional lidar relies on time-resolved measurements of direct light transport — light reflecting directly from surfaces back to sensors. Measurements of indirect light transport, involving multiple scattering events before sensor arrival, are typically ignored or discarded because modeling indirect light requires computationally expensive inverse rendering using recursive path tracing procedures. However, indirect light measurements provide rich information about material properties, appearance, and geometry.

Conventional lidar systems pre-process captured time-resolved measurements into 3D point clouds representing geometry estimates based on direct light transport. Recent work leverages multi-viewpoint lidar measurements for 3D reconstruction and novel view synthesis, but existing methods use point cloud representations or direct-only time-resolved measurements, thus ignoring indirect light. The work relates closely to Malik et al., which uses lidar measurements and neural radiance fields representations to render videos of propagating light from novel viewpoints. However, while their representation is effective for view synthesis, it lacks a physically-based model, preventing accurate geometry reconstruction or scene rendering under novel illumination.

This paper proposes a method for inverse rendering from multi-viewpoint, time-resolved measurements of propagating light from flash lidar systems. The approach uses a hybrid neural representation, modeling geometry through volume rendering and appearance through a physically-based model simulating global illumination using a radiance cache. Instead of integrating light paths via path tracing, the radiance cache stores time-resolved radiance arriving at any point in a volume from any direction. The representation is optimized in an amortized fashion, eliminating recursive rendering integral evaluation. Our contributions include: (1) a method for neural inverse rendering of propagating light using physically-based models with time-resolved radiance caching; (2) a new multi-viewpoint, time-resolved flash lidar measurement dataset; and (3) demonstrations showing state-of-the-art geometry reconstruction under strong indirect light, and novel-view propagating light rendering in scenes with varying reflectance.

Time-of-flight systems such as lidar measure flight times by marking backscattered pulse arrival times. These systems typically combine nanosecond or picosecond pulsed lasers with fast photodiodes or single-photon avalanche diodes (SPADs) to measure ultrafast light variations. Resulting time-resolved measurements capture direct and indirect light transport, recording videos of propagating light at ultrafast timescales. This work uses lidar systems with picosecond pulsed lasers and SPADs to capture multi-view propagating light videos. This work is the first to capture multi-view photon count histogram datasets where both flash lidar light sources and sensors vary in position, and develops the first physically-based time-resolved inverse rendering technique using multi-viewpoint propagating light videos.

Time-resolved path-tracing renderers simulate propagating light wavefronts and account for birefringence, refraction, and volumetric scattering effects. Recently, differentiable time-resolved renderers have been developed; however, robust analysis-by-synthesis scene reconstruction using these methods remains an open problem due to computational complexity and sensitivity to initialization and noise. In the area of non-line-of-sight (NLOS) imaging, techniques model two-bounce, three-bounce, or higher-order scattering events to recover occluded geometry. This work performs physically-based modeling of multiply scattered light without restrictive assumptions on scene geometry or material properties, integrating the approach into inverse rendering frameworks for captured multi-viewpoint experiments.

Inverse rendering aims to recover scene attributes like materials, lighting, and geometry from images. Recent physically-based rendering techniques using NeRFs have made inverse rendering considerably more robust, but either consider only direct illumination or require explicitly simulating multiple light bounces to model indirect light, which is computationally expensive. Another approach uses radiance caches — data structures storing hemispheric incoming radiance at every point. Combining radiance caches with NeRFs leads to more efficient indirect light modeling. However, no previous technique performs physically-based inverse rendering from multi-viewpoint time-resolved measurements. This work develops a new time-resolved radiance cache enabling neural inverse rendering from propagating light videos.

The rendering equation models outgoing radiance in direction ω_o at point x along ray x(t) = o − tω_o: L_o(x(t), ω_o) = ∫_Ω f(x, ω_i, ω_o) L_i(x, ω_i) (n · ω_i) dω_i. The equation integrates incident radiance L_i arriving to x from direction ω_i, weighted by the BRDF f, over the positive hemisphere with respect to normal n. Naive evaluation leads to exponential computation increases since the equation must be evaluated recursively to compute incident radiance.

To avoid this computational penalty, radiance caching removes the problematic recursion by replacing incident radiance L_i in the rendering equation with cache lookups L_i^cache. The integral can be efficiently approximated through cache sampling and BRDF sampling using multiple importance sampling. Recent work demonstrates that NeRFs provide accurate radiance cache modeling. Specifically, L_i^cache(x, ω_i) can be computed by volume rendering the NeRF along a secondary ray, where outgoing radiance at each secondary ray point is predicted by the NeRF and combined using quadrature weights accounting for transmittance and absorption along the ray.

A lidar measurement is modeled by casting a primary ray x(t) = o − tω_o into the scene. The time-resolved rendering equation is a modified version of the standard rendering equation, adding time of flight τ: L_o(x(t), ω_o, τ) = ∫_Ω f(x(t), ω_i, ω_o) L_i(x(t), ω_i, τ) (n · ω_i) dω_i. The reflectance f is modeled using the Disney-GGX BRDF, depending on scene material properties. Incident radiance is decomposed into two components: L_i = L_i^dir + L_i^cache, comprising a direct component and an indirect component evaluated using the radiance cache.

The direct component models light emitted from the lidar source at x_l, propagating to scene point x, and scattering directly back to the sensor. It is given as: L_i^dir(x(t), ω_i, τ) = δ(ω_l − ω_i) L_i^l(ω_l, τ − ||x(t) − x_l|| / c) / ||x(t) − x_l||². The model accounts for inverse-square law intensity falloff and time delay to position x(t) based on light speed c. The time-resolved radiance cache L_i^cache is evaluated using secondary rays cast from primary ray points, applying time-resolved volume rendering where light incident at a point is the sum of delayed copies of light leaving each point along the secondary ray, with delays depending on distance.

The cache is parameterized using multi-resolution hash encoding H^app to learn position-dependent appearance feature f^app. Similarly, hash encoding-based neural network N^geom represents scene geometry through density and normals. Density values used for volume rendering are shared across both the physically-based model and radiance cache, ensuring geometric consistency. The radiance cache output is decomposed into direct and indirect components: L_o^cache = L_o^cache,dir + L_o^cache,indir. The direct component uses a neural network learning the BRDF, while the indirect component employs a split-sum approximation that efficiently factorizes the integral into a product of an integrated BRDF term and an integrated incident radiance term, both predicted by neural networks.

A key design choice is that the indirect radiance component is conditioned on the light source position x_l, because indirect light depends on where the light source illuminates the scene. Following Malik et al., L_i,Ω^indir predicts a vector representing radiance over discretized time intervals. The network f_Ω^indir predicts the integrated BRDF and L_i,Ω^indir predicts the integrated incident radiance, both conditioned on appearance features, normals, outgoing direction, and light source position. This conditioning on x_l enables relighting capabilities — by changing the light source position at inference time, the system can synthesize time-resolved measurements under novel illumination conditions.

The representation is optimized by minimizing differences between lidar measurements and renderer output: L_data = ∑ α(L_i^cache) ||L_i − L_i^meas||². The function α is chosen to more strongly penalize darker region errors, improving perceptual quality similar to tonemapping curves. A radiometric prior constrains cache-rendered direct and indirect light consistency with the full physically-based model, by supervising the cache with the physically-based model's outputs. The complete photometric loss function is: L_data + λ_cache L_cache + λ_dir L_dir + λ_indir L_indir. By minimizing this loss, the method recovers scene material models (parameterized using Disney-GGX), scene geometry, normals, and appearance parameters.

Beyond the photometric loss, the method includes a regularizer L_normals tying predicted normals to analytic density field normals; a smoothness penalty L_geom on analytic normals; a smoothness penalty L_mat on predicted BRDF parameters; proposal resampling and distortion losses from Zip-NeRF; and a mask loss L_mask. Multiple importance sampling is used for secondary rays based on BRDF and a learnable importance sampler for incident illumination. The learnable importance sampler is supervised with loss L_vMF. Time-resolved direct outgoing light is represented as a one-hot vector where each bin corresponds to a discrete time interval, and indirect light as a dense vector of the same size.

The system is evaluated on three tasks: (1) time-resolved lidar measurement view synthesis, (2) integrated steady-state lidar image view synthesis, and (3) geometry reconstruction. Rendered integrated lidar images are assessed using PSNR, SSIM, and LPIPS. Time-resolved measurement accuracy is evaluated using transient intersection-over-union (T-IOU). Mean absolute error and L1 error measure normal and depth accuracy, respectively. The method is compared to state-of-the-art baselines: T-NeRF, which accounts only for direct light; and Flying with Photons (FWP++), which predicts time-resolved radiance at every spatial point but lacks a physically-based model. Both baselines are implemented using the same hash-encoding-based neural representation, with identical regularizers and hyperparameters for fairness.

On simulated scenes (Cornell box, pots, peppers, kitchen), the method achieves PSNR of 30.99 dB compared to T-NeRF's 22.44 dB and FWP++'s 29.00 dB. More critically, normal estimation MAE is 8.45 degrees versus T-NeRF's 28.00 and FWP++'s 22.80, demonstrating substantially better geometry recovery. Since T-NeRF models only direct light, it fails to recover accurate geometry under strong indirect light from specular reflections and diffuse inter-reflections, introducing floating artifacts. Conversely, FWP++ models both direct and indirect radiance but lacks a physically-accurate rendering model, causing it to overfit — it uses mirror scene copies to explain specular reflections and incorrect depths for diffuse inter-reflections.

On captured scenes (globe, house, spheres, statue), the method demonstrates more accurate geometry recovery than FWP++, especially for areas with indirect light presence, such as wall corners and candle bottoms. However, FWP++ shows slight view synthesis improvements in captured data — the physically-grounded approach may be more sensitive to system calibration and model mismatch. The approach further enables time-resolved view synthesis and relighting, which has not previously been demonstrated from captured multi-view, time-resolved measurements. The system also demonstrates time-resolved imaging without lidar supervision — the model can be trained using continuous-wave time-of-flight measurements or even intensity images, recovering direct and indirect light transport effects from each input type.

The method relies on a more constrained physical model than other approaches, including FWP++. Some captured data performance degradation compared to baselines is observed, where model mismatch is a potential issue. This might be mitigated through improved physical setup calibration. Additionally, the method requires more than one day of optimization on a single GPU due to time-consuming physical light transport simulation, I/O penalties from loading large time-resolved measurement vectors, and GPU memory bandwidth requirements. Addressing this might involve using neural representations predicting time-resolved signals at single time instants rather than complete vectors, or using faster neural representations like 3D Gaussian Ray Tracing or EVER.

Although non-line-of-sight imaging is not tackled, the framework could principally extend to this application in unconstrained conditions, like non-line-of-sight imaging with non-planar relay surfaces. Due to widespread lidar technology usage, the work has potential impact in autonomous navigation or remote sensing — especially in scenarios with strong indirect lighting effects. The physically-based decomposition of direct and indirect light transport opens avenues for improved depth estimation in challenging multi-bounce environments, where conventional lidar processing pipelines fail due to indirect light contamination.