Representing and rendering dynamic scenes has been an important but challenging task. Especially, many real-world scenes involve complex motions that need to be modeled with both spatially and temporally sparse input. Recent methods extend 3D Gaussian Splatting (3D-GS) to handle static scenes with remarkable quality and speed, but directly applying 3D-GS per frame is impractical for dynamic content. In this paper, the authors propose 4D Gaussian Splatting (4D-GS), a unified representation for dynamic scenes that combines 3D Gaussians and 4D neural voxels. A decomposed neural voxel encoding algorithm inspired by HexPlane is introduced, and a lightweight MLP predicts Gaussian deformations at novel timestamps. The method achieves real-time rendering under high resolutions, 82 FPS at an 800x800 resolution on an RTX 3090 GPU, while maintaining quality comparable to or better than state-of-the-art methods.

Novel view synthesis (NVS) represents a critical task in the domain of 3D vision and plays a vital role in many applications, e.g. VR, AR, and movie production. NVS aims at rendering images from desired viewpoints or timestamps of scenes, typically requiring accurate scene modeling from sparse 2D input images. Dynamic scenes are common in real scenarios, making rendering challenging since complex motions need to be modeled with both spatially and temporally sparse input. NeRF achieved significant success in novel view synthesis through implicit function representation of scenes, where volume rendering techniques connect 2D images and 3D scenes. However, the original NeRF method incurs substantial training and rendering costs. Various NeRF variants reduced training time from days to minutes, yet the rendering process still bears a non-negligible latency.

Recent 3D Gaussian Splatting (3D-GS) significantly accelerates rendering to real-time levels by representing the scene as 3D Gaussians. Efficient differentiable splatting replaces volume rendering by directly projecting 3D Gaussian points onto the 2D plane. This approach offers real-time rendering speed and explicit scene representation amenable to manipulation. However, 3D-GS focuses on static scenes. Extending it to dynamic scenes as a 4D representation remains challenging. The primary challenge involves modeling complicated point motions from sparse input. While 3D-GS maintains natural geometry priors through point-like Gaussians, constructing 3D Gaussians at each timestamp multiplies storage costs, especially for extended sequences.

The research goal aims to construct a compact representation while maintaining both training and rendering efficiency. The authors propose maintaining only one canonical 3D Gaussian set; for each timestamp, these Gaussians transform via a deformation field network into new positions with modified shapes. This transformation represents motion and deformation. Unlike approaches modeling individual Gaussian motions separately, the spatial-temporal structure encoder can connect different adjacent 3D Gaussians to predict more accurate motions and shape deformation. The contributions include: (1) an efficient 4D Gaussian splatting framework modeling both Gaussian motion and shape changes across time; (2) a multi-resolution encoding method connecting nearby 3D Gaussians via a spatial-temporal structure encoder; (3) real-time rendering up to 82 FPS at 800x800 for synthetic datasets and 30 FPS at 1352x1014 in real datasets.

Implicit radiance fields effectively learn scene representations for novel view synthesis. Researchers extended static assumptions, addressing dynamic scene novel view synthesis. Canonical mapping neural rendering methods employ explicit voxel grids for temporal information modeling, accelerating learning to approximately thirty minutes. K-Planes and related approaches represent further advances in efficient dynamic scene learning by adopting decomposed neural voxels and treating sampled points at each timestamp individually. Though these approaches achieve fast training speeds, real-time rendering for dynamic scenes is still challenging, especially for monocular input. The proposed method targets constructing a highly efficient training and rendering pipeline while maintaining the quality, even for sparse inputs.

3D scene representation remains challenging. The community explored various neural representations including meshes, point clouds, and hybrid approaches. 3D-GS notably achieves pure explicit representation and differentiable point-based splatting methods, enabling real-time rendering of novel views. Dynamic3DGS models dynamic scenes by tracking the position and variance of each 3D Gaussian at each timestamp, utilizing explicit tables storing information at every timestamp. This approach incurs linear memory consumption increase, denoted as O(t*N), in which N is the number of 3D Gaussians. The proposed approach's memory complexity depends only on the number of 3D Gaussians and parameters of the Gaussian deformation field network F, which is denoted as O(N+F). Another method adds temporal Gaussian distribution to original 3D Gaussians, elevating them to 4D, though each 3D Gaussian only focuses on its local temporal space. The proposed approach also models Gaussian motions but with a compact network, resulting in highly efficient training and real-time rendering.

3D Gaussians represent an explicit 3D scene representation as point clouds. Each Gaussian is characterized by a covariance matrix Σ and a center point X, which is referred to as the mean value of the Gaussian: G(X) = exp(-1/2 X^T Σ^-1 X). For differentiable optimization, the covariance matrix decomposes into scaling and rotation matrices: Σ = RSS^TR^T. Each 3D Gaussian is characterized by: position X ∈ R³, color defined by spherical harmonic (SH) coefficients C, opacity α, rotation factor r ∈ R⁴, and scaling factor s ∈ R³. When rendering novel views, differential splatting projects 3D Gaussians onto camera planes. For each pixel, the blending of N ordered overlapping points follows: C = Σ_i∈N c_i α_i Π_j=1^i-1(1-α_j).

All dynamic NeRF algorithms follow the formulation: c, σ = M(x, t), where M maps 6D space to 4D space. Dynamic NeRFs primarily follow canonical-mapping or time-aware volume rendering approaches. Canonical-mapping volume rendering transforms each sampled point into a canonical space by a deformation network: c, σ = NeRF(x + Δx). The proposed 4D Gaussian splatting presents a novel rendering technique. The method transforms 3D Gaussians using a Gaussian deformation field network F at time t directly, and differentiable splatting is followed. This fundamentally differs from NeRF-based methods in that no volume rendering integration is required.

Given a view matrix M=[R,T] and a timestamp t, the 4D Gaussian splatting framework includes 3D Gaussians G and a deformation field network F. Novel-view image rendering follows: I = S(M, G'), where G' = ΔG + G. The Gaussian deformation ΔG is introduced by the deformation field network: ΔG = F(G, t). The spatial-temporal structure encoder H encodes temporal and spatial features: f_d = H(G, t). The multi-head decoder D predicts Gaussian deformation: ΔG = D(f). Then deformed 3D Gaussians G' undergo rendering via differentiable splatting. 4D Gaussian splatting converts the original 3D Gaussians G into another group of 3D Gaussians G' given a timestamp t.

Nearby 3D Gaussians always share similar spatial and temporal information. The spatial-temporal structure encoder includes multi-resolution HexPlane R_l(i,j) and a tiny MLP φ_d. While vanilla 4D neural voxels are memory-consuming, the method adopts a 4D K-Planes module to decompose the 4D neural voxel into 6 planes. The encoder contains 6 multi-resolution plane modules and a tiny MLP: H(G,t) = {R_l(i,j), φ_d | (i,j) ∈ {(x,y),(x,z),(y,z),(x,t),(y,t),(z,t)}, l ∈ {1,2}}. Each voxel module R(i,j) ∈ R^{h×lN_i×lN_j}, where h is the feature hidden dimension and N denotes basic voxel grid resolution. Computing separate voxel features uses bilinear interpolation: f_h = ∪_l Π interp(R_l(i,j)), and a tiny MLP merges all features: f_d = φ_d(f_h).

When all Gaussian features are encoded, the multi-head Gaussian deformation decoder computes desired variables: D = {φ_x, φ_r, φ_s}. Separate MLPs compute deformations for each attribute: position deformation ΔX = φ_x(f_d), rotation deformation Δr = φ_r(f_d), and scaling deformation Δs = φ_s(f_d). The deformed features become: (X', r', s') = (X + ΔX, r + Δr, s + Δs). Final deformed 3D Gaussians are: G' = {X', s', r', σ, C}. Note that opacity σ and color C remain unchanged, with deformation only applied to geometric attributes — position, rotation, and scaling.

3D Gaussian initialization is critical. 3D Gaussians can be well-trained with structure from motion (SfM) points initialization. Similarly, 4D Gaussians require proper 3D Gaussian initialization. The method optimizes 3D Gaussians at the initial 3000 iterations for warm-up and then renders images with 3D Gaussians I = S(M, G) instead of 4D Gaussians. This warm-up strategy provides three benefits: (a) making some 3D Gaussians stay in the dynamic part, which releases the pressure of large deformation learning; (b) learning proper 3D Gaussians and suggesting deformation fields to pay more attention to the dynamic part; (c) avoiding numeric errors. For the loss function, the method uses L1 color loss combined with a grid-based total-variational loss L_tv: L = |I - I_gt| + L_tv.

Implementation uses the PyTorch framework tested on a single RTX 3090 GPU. The model is primarily assessed using synthetic datasets introduced by D-NeRF, designed for monocular settings with camera poses for each timestamp close to randomly generated. Each scene contains dynamic frames ranging from 50 to 200 frames. For real-world evaluation, the study utilizes datasets provided by HyperNeRF and Neu3D as benchmarks. Assessment uses various metrics: peak-signal-to-noise ratio (PSNR), perceptual quality measure LPIPS, structural similarity index (SSIM), as well as DSSIM, MS-SSIM, FPS, training times, and storage. Benchmarking against state-of-the-art methods shows the proposed approach achieves highest rendering quality within the synthetic dataset and exceptionally fast rendering speeds while keeping extremely low storage consumption and convergence time.

Real-world dataset results show that some NeRFs suffer from slow convergence speed, while other grid-based NeRF methods encounter difficulties when attempting to capture intricate object details. The proposed method achieves comparable rendering quality, fast convergence, and excels in free-view rendering speed in indoor cases. The study further demonstrates that if the rendered points are lower than 30,000, the rendering speed can be up to 90 FPS. Achieving real-time rendering requires balancing among all the rendering resolutions and 4D Gaussian representation. The explicit Gaussian representation also enables composition with different 4D Gaussians — different models can be composed following their individually trained deformation fields, similar to Dynamic3DGS.

The explicit HexPlane encoder R_l(i,j) possesses the capacity to retain 3D Gaussians' spatial and temporal information. Removing this module reveals that using only a shallow MLP φ_d falls short in modeling complex deformations across various settings. For the Gaussian deformation decoder, all changes in 3D Gaussians can be explained by separate MLPs {φ_x, φ_r, φ_s}. Results show 4D Gaussians cannot fit dynamic scenes well without modeling 3D Gaussian motion. Additionally, the movement of human body joints is typically manifested as stretching and twisting of surface details, making size and shape adjustments necessary alongside position changes. Training without warmup initialization causes convergence difficulty, confirming the warmup strategy's three benefits: distributing Gaussians to dynamic regions, guiding the deformation field's focus, and avoiding numeric errors.

Though 4D-GS can indeed attain rapid convergence and yield real-time rendering outcomes in many scenarios, there are a few key challenges. Large motions, absent background points, and imprecise camera poses cause optimization difficulty. Additionally, it is still challenging for 4D-GS to split the joint motion of static and dynamic Gaussian parts under monocular settings. Finally, a more compact algorithm needs to be designed to handle urban-scale reconstruction. The method's tracking capability in 3D is demonstrated with pretty low storage — approximately 10 MB in 3D Gaussians G and 8 MB in the Gaussian deformation field network F.

This paper proposes 4D Gaussian Splatting to achieve real-time dynamic scene rendering. An efficient deformation field network is constructed to accurately model Gaussian motions and shape deformations, where adjacent Gaussians are connected via a spatial-temporal structure encoder. The method combines multi-resolution HexPlane decomposition with lightweight multi-head decoders to predict position, rotation, and scaling deformations from a single canonical Gaussian set. Experiments across both synthetic and real-world benchmarks demonstrate the method's advantages in rendering speed, training efficiency, and storage compactness. The approach also shows potential for 4D object tracking and editing, suggesting broader applications beyond novel view synthesis.