We address the problem of synthesizing novel views from a monocular video depicting a complex dynamic scene. State-of-the-art methods based on dynamic Neural Radiance Fields (NeRF) have demonstrated impressive results on this task. However, they struggle to scale to long videos featuring complex object motions and uncontrolled camera trajectories, producing blurry or inaccurate renderings that limit their real-world applications.

Rather than encoding the entire dynamic scene within the weights of MLPs, we present an approach that adopts a volumetric image-based rendering (IBR) framework that synthesizes new viewpoints by aggregating features from nearby views in a scene-motion-aware manner. Our system retains the advantages of prior methods in its ability to model complex scenes and view-dependent effects, but also enables synthesizing photo-realistic novel views from long videos featuring complex scene dynamics with unconstrained camera trajectories. We demonstrate significant improvements over state-of-the-art methods on dynamic scene datasets and show successful application to in-the-wild videos with challenging camera and object motion where prior methods fail.

Computer vision has achieved high-quality free-viewpoint renderings of static 3D scenes. However, novel view synthesis from monocular video of dynamic scenes — featuring moving people or objects — presents significantly greater challenges. Recent progress has emerged through time-varying neural volumetric representations like HyperNeRF and Neural Scene Flow Fields (NSFF), which encode spatiotemporally varying scene content within coordinate-based MLPs.

Despite these achievements, dynamic NeRF methods face fundamental limitations preventing their application to casual, in-the-wild videos. Local scene flow-based methods like NSFF struggle to scale to longer input videos captured with unconstrained camera motions: the NSFF paper only claims good performance for 1-second, forward-facing videos. Methods like HyperNeRF that construct canonical models remain mostly constrained to object-centric scenes with relatively small object motion and controlled camera paths, and fail on scenes exhibiting complex object motion.

We present a new approach that is scalable to dynamic videos captured with (1) long time duration, (2) unbounded scenes, (3) uncontrolled camera trajectories, and (4) fast and complex object motion. Our approach retains the advantages of volumetric scene representations that can model intricate scene geometry with view-dependent effects, while significantly improving rendering fidelity for both static and dynamic scene content. Our inspiration comes from recent static scene rendering methods that synthesize novel images by aggregating local image features from nearby views along epipolar lines. However, moving scene elements violate epipolar constraints. We propose aggregating multi-view image features in "scene motion-adjusted" ray space, enabling proper reasoning about spatio-temporally varying geometry and appearance.

We encountered efficiency and robustness challenges in scaling aggregation-based methods to dynamic scenes. We represent scene motion using motion trajectory fields described in terms of learned basis functions that efficiently model motion across multiple frames. We introduce a new temporal photometric loss operating in motion-adjusted ray space that achieves temporal coherence. We also propose a scene factorization into static and dynamic components through IBR-based motion segmentation within a Bayesian framework that improves novel view quality. On two dynamic scene benchmarks, our approach renders highly detailed scene content with average LPIPS error reductions exceeding 50% across entire scenes and dynamic object regions. We hope that our work advances the applicability of dynamic view synthesis methods to real-world videos.

Classic image-based rendering (IBR) methods synthesize novel views by integrating pixel information from input images, categorized by their dependence on explicit geometry. Light field or lumigraph rendering methods generate new views by filtering and interpolating sampled rays without explicit geometric models. To handle sparser input views, numerous approaches leverage pre-computed proxy geometry such as depth maps or meshes. Recently, neural representations have demonstrated high-quality novel view synthesis, with Neural Radiance Fields (NeRF) achieving unprecedented fidelity by encoding continuous scene radiance fields within MLPs.

Among NeRF-based methods, IBRNet proves most relevant to this work. IBRNet combines classical IBR techniques with volume rendering to produce a generalized IBR module that can render high-quality views without per-scene optimization. We extend volumetric IBR frameworks designed for static scenes to more challenging dynamic scenes, focusing on higher-quality novel view synthesis for long videos with complex camera and object motion rather than generalization across scenes.

Most prior work on novel view synthesis for dynamic scenes requires multiple synchronized input videos, limiting real-world applicability. More recently, many works propose synthesizing novel views of dynamic scenes from single cameras. With neural rendering advances, NeRF-based dynamic view synthesis methods show state-of-the-art results. Approaches like Nerfies and HyperNeRF represent scenes using deformation fields mapping local observations to canonical scene representations. These methods can handle long videos, but are mostly limited to object-centric scenes with relatively small object motion and controlled camera paths. Other methods represent scenes as time-varying NeRFs. NSFF particularly uses neural scene flow fields capturing fast and complex 3D scene motion for in-the-wild videos. However, this method only works well for short (1-2 second), forward-facing videos.

Given monocular video frames (I_1, I_2, ..., I_N) with known camera parameters (P_1, P_2, ..., P_N), our goal is to synthesize novel viewpoints at any desired time within the video. Like many approaches, we first optimize a model via per-video training that reconstructs input frames, then use it for rendering novel views. Rather than encoding 3D color and density directly in MLP weights as recent dynamic NeRF methods do, we integrate classical IBR ideas into a volumetric rendering framework. Compared to explicit surfaces, volumetric representations can more readily model complex scene geometry with view-dependent effects.

We synthesize new views by aggregating features extracted from temporally nearby source views. To render an image at time i, source views I_j within temporal radius r frames are identified as j in N(i) = [i-r, i+r]. For each source view, a shared convolutional encoder extracts 2D feature map F_i, forming input tuple {I_j, P_j, F_j}. For static scenes, points along target rays lie along corresponding epipolar lines in neighboring source views, allowing potential correspondence aggregation. However, moving scene elements violate epipolar constraints, leading to inconsistent feature aggregation without motion accounting.

We perform "motion-adjusted" feature aggregation. Determining correspondence in dynamic scenes could straightforwardly estimate scene flow fields via MLP to determine motion-adjusted 3D locations. However, this strategy is computationally infeasible in a volumetric IBR framework due to recursive unrolling of the MLPs. Instead, we represent scene motion using motion trajectory fields described through learned basis functions. For a 3D point x along target ray r at time i, trajectory coefficients are encoded with MLP G_MT: {phi_i^l(x)} = G_MT(gamma(x), gamma(i)), where phi_i^l are basis coefficients and gamma denotes positional encoding. We choose L=6 bases and 16 linearly increasing frequencies, based on the assumption that scene motion tends toward low frequency.

A global learnable motion basis {h_i^l} spanning every input video time step is optimized jointly with the MLP. The motion trajectory is defined as Gamma_{x,i}(j) = sum_l h_j^l * phi_i^l(x), and the relative displacement between x and its 3D correspondence x_{i->j} at time j is: Delta_{x,i}(j) = Gamma_{x,i}(j) - Gamma_{x,i}(i). With this representation, finding 3D correspondences for a query point in neighboring views requires just a single MLP query, enabling efficient multi-view feature aggregation. We initialize the basis with DCT basis but fine-tune it during optimization, as fixed DCT basis fails to model wide-ranging real-world motions. The resulting source features across neighbor views are aggregated through weighted average pooling, then processed by a ray transformer network with time embedding, predicting per-sample colors and densities.

Optimizing the dynamic scene representation by comparing the rendered image C_hat_i with ground truth C_i alone risks overfitting: the representation might perfectly reconstruct input views but fail at rendering correct novel views. The representation has the capacity to reconstruct completely separate models for each time instance without utilizing or accurately reconstructing scene motion. To recover consistent scenes with physically plausible motion, temporal coherence enforcement becomes necessary. One way to define temporal coherence is that scenes at two neighboring times i and j should remain consistent after accounting for scene motion.

We specifically enforce temporal photometric consistency through "cross-time rendering in motion-adjusted ray space." The idea is to render a view at time i but "via" some nearby time j, called cross-time rendering. For each nearby time j in N(i), rather than directly using points x along ray r, we consider points x_{i->j} along motion-adjusted ray r_{i->j}, treating them as if lying along rays at time j. Having computed motion-adjusted points, the MLP queries predict coefficients of "new" trajectories, using these to compute corresponding 3D points for images k in temporal window N(j). These new 3D correspondences then render pixels exactly as described in Section 3.1, except now along a curved, motion-adjusted ray.

Comparing the cross-time rendered color C_hat_{j->i}(r) with target pixel C_i(r) proceeds via a motion-disocclusion-aware RGB reconstruction loss: L_pho = sum_r sum_{j in N(i)} W_hat_{j->i}(r) * rho(C_i(r), C_hat_{j->i}(r)), where a generalized Charbonnier loss is applied for RGB loss rho. W_hat_{j->i}(r) represents a motion disocclusion weight computed by accumulated alpha weight differences between times i and j, addressing motion disocclusion ambiguity. When j=i, no scene motion-induced displacement occurs, meaning C_hat_{j->i} = C_hat_i and no disocclusion weights apply.

As observed in NSFF, synthesizing novel views using small temporal windows proves insufficient for recovering complete, high-quality static scene region content, since contents may only appear in spatially distant frames due to uncontrolled camera paths. Following NSFF, we model entire scenes using two separate representations. Dynamic content (c_i, sigma_i) is represented through time-varying models used for cross-time rendering during optimization. Static content (c, sigma) is represented through time-invariant models, rendering identically to time-varying models but aggregating multi-view features without scene motion adjustment (i.e., along epipolar lines). Dynamic and static predictions are combined and rendered to a single output color using NeRF-W methods for combining static and transient models.

Without initialization, scene factorization tends to be dominated by either the time-invariant or time-varying representation. To facilitate factorization, prior work initializes using semantic segmentation masks, assuming all moving objects can be captured via semantic segmentation labels with temporally accurate masks. These assumptions fail in many real-world scenarios. We propose a new motion segmentation module producing segmentation masks that supervise the two-component scene representation. Our idea is inspired by Bayesian learning techniques, integrated into volumetric IBR representations for dynamic videos. Before training the main two-component representation, two lightweight models are jointly trained to obtain motion segmentation mask M_i for each input frame. Static content is modeled with IBRNet rendering via epipolar line feature aggregation; dynamic content is modeled with a 2D convolutional encoder-decoder network D predicting opacity, confidence, and RGB.

The full reconstructed images composite pixelwise from two model outputs: B_hat_i^full(r) = alpha_i^dy(r) * B_hat_i^dy(r) + (1 - alpha_i^dy(r)) * B_hat^st(r). Segmenting moving objects assumes observed pixel color undergoes heteroscedastic aleatoric uncertainty, modeled with Cauchy distributions with time-dependent confidence beta_i^dy. Taking the negative log-likelihood yields a segmentation loss as weighted reconstruction loss. The main time-varying and time-invariant models are then initialized with masks M_i, applying reconstruction loss to time-varying model renderings in dynamic regions and time-invariant model renderings in static regions. Morphological erosion and dilation apply to M_i near mask boundaries, and mask supervision weights decay by factor 5 every 50K steps for dynamic regions.

Monocular reconstruction of complex dynamic scenes is highly ill-posed, and photometric consistency alone is insufficient to avoid bad local optimization minima. Therefore, we adopt regularization schemes consisting of three main parts: L_reg = L_data + L_MT + L_cpt. L_data constitutes data-driven terms with L1 monocular depth and optical flow consistency priors using estimates from Zhang et al. and RAFT. L_MT represents motion trajectory regularization encouraging estimated trajectory fields toward cycle-consistency and spatial-temporal smoothness. L_cpt denotes compactness priors encouraging scene decomposition toward binary outcomes via entropy loss and mitigating floaters through distortion losses. The final combined loss optimizing the main representation is: L = L_pho + L_mask + L_reg.

Camera pose estimation uses COLMAP. For each ray, coarse-to-fine sampling strategies employ 128 per-ray samples. Separate models train from scratch for each scene using Adam optimizer. The main representation network architectures are variant IBRNet architectures. Scenes are reconstructed in Euclidean space without special scene parameterization. Full system optimization on 10-second videos requires approximately two days using 8 NVIDIA A100s, with rendering taking roughly 20 seconds for 768x432 frames. For in-the-wild videos, we synthesize images at eight randomly sampled nearby viewpoints for every input time via estimated depths, providing additional source images to avoid degenerate solutions when camera and object motions are mostly colinear.

Numerical evaluations are conducted on Nvidia Dynamic Scene Dataset and UCSD Dynamic Scenes Dataset. Each dataset comprises eight forward-facing dynamic scenes recorded by synchronized multi-view cameras, with each video containing 100 to 250 frames. We compare our approach against state-of-the-art monocular view synthesis methods: three canonical space-based methods (Nerfies, HyperNeRF) and two scene flow-based methods (NSFF, Dynamic View Synthesis from Gao et al.). For fair comparisons, the same depth, optical flow, and motion segmentation masks used for our approach serve as inputs to other methods. We report rendering quality using PSNR, SSIM, and LPIPS, calculating errors both over entire scenes and restricted to moving regions.

Quantitative results on two benchmark datasets show our approach significantly improves over prior state-of-the-art methods across all error metrics. On the Nvidia Dynamic Scene Dataset, our method achieves 0.957 SSIM, 30.86 PSNR, and 0.027 LPIPS, compared to NSFF's 0.927 SSIM, 28.90 PSNR, and 0.062 LPIPS. On the UCSD Dynamic Scenes Dataset, results show 0.983 SSIM, 36.47 PSNR, and 0.014 LPIPS versus NSFF's 0.952 SSIM, 31.75 PSNR, and 0.034 LPIPS. Notably, PSNR improvements over second-best methods reach 2dB and 4dB on the two datasets respectively, and LPIPS error reductions exceed 50%, suggesting our framework is much more effective at recovering highly detailed scene contents.

Qualitative comparisons reveal that prior dynamic-NeRF methods have difficulty rendering details of moving objects, showing excessively blurred dynamic content including the texture of balloons, human faces, and clothing. In contrast, our approach synthesizes photo-realistic novel views of both static and dynamic scene content closest to ground truth images. On in-the-wild footage of complex dynamic scenes, our approach synthesizes photo-realistic novel views, whereas prior dynamic-NeRF methods fail to recover high-quality details of both static and moving scene contents, such as shirt wrinkles and dog fur. Explicit depth warping produces holes at regions near disocclusions and out-of-view areas.

Ablation studies on the Nvidia Dynamic Scene Dataset validate the effectiveness of our proposed components. Configuration A (baseline IBRNet with time embedding) achieves only 0.905 SSIM, 25.33 PSNR. Configuration B (without temporal consistency via cross-time rendering) reaches 0.911 SSIM, 27.57 PSNR. Configuration C (scene flow fields instead of trajectories) gets 0.935 SSIM, 29.42 PSNR. Configuration E (without time-invariant static model) drops to 0.919 SSIM, 28.19 PSNR. The complete system achieves 0.957 SSIM, 30.77 PSNR, 0.028 LPIPS, confirming that motion trajectory representation, cross-time rendering, static-dynamic decomposition, motion segmentation, and regularization are all essential components.

Our method has several limitations. It is limited to relatively small viewpoint changes compared with methods designed for static or quasi-static scenes. It cannot handle small fast-moving objects due to incorrect initial depth and optical flow estimates. Compared to prior dynamic NeRF methods, the synthesized views are not strictly multi-view consistent, and rendering quality of static content depends on which source views are selected. The approach exhibits sensitivity to degenerate motion patterns from in-the-wild videos where object and camera motion are mostly colinear, though heuristics handle such cases. Additionally, the method can only synthesize dynamic contents appearing at distant times within limited temporal windows.

We presented a new approach for space-time view synthesis from a monocular video depicting a complex dynamic scene. By representing a dynamic scene within a volumetric IBR framework, our approach overcomes limitations of recent methods that cannot model long videos with complex camera and object motion. We have shown that our method can synthesize photo-realistic novel views from in-the-wild dynamic videos, and can achieve significant improvements over prior state-of-the-art methods on the dynamic scene benchmarks. We hope that our work advances the applicability of dynamic view synthesis methods to real-world videos.

Our key contributions include: (1) a motion trajectory field representation based on learned basis functions that efficiently models scene motion across multiple frames with just a single MLP query; (2) a cross-time rendering scheme in motion-adjusted ray space that enforces temporal coherence; (3) an IBR-based Bayesian motion segmentation module that avoids unreliable semantic segmentation assumptions; and (4) a complete system that advances the state-of-the-art in dynamic view synthesis by significant margins on standard benchmarks while extending applicability to challenging in-the-wild scenarios.