We present an approach to modeling an image-space prior on scene motion. Our method learns this prior from motion trajectories extracted from real video sequences depicting natural, oscillatory dynamics such as trees, flowers, candles, and clothes swaying in the wind. We model scene motion in the Fourier domain: given a single image, our trained model uses a frequency-coordinated diffusion sampling process to predict a spectral volume, which can be converted into a motion texture that spans an entire video. This approach enables turning still images into seamlessly looping videos, and allows users to realistically interact with objects in real pictures by interpreting the spectral volumes as image-space modal bases, which approximate object dynamics.

The natural world is always in motion, with even seemingly static scenes containing subtle oscillations as a result of factors such as wind, water currents, respiration, or other natural rhythms. Motion is one of the most salient visual signals, and humans are particularly sensitive to it. While it is easy for humans to interpret or imagine motion in scenes, training a model to learn realistic scene motion is far from trivial. The underlying physical dynamics are hard to measure and capture at scale, but fortunately, in many cases measuring them is unnecessary — the necessary signals for producing plausible motion can often be extracted from observed 2D motion.

Real-world observed motion is multi-modal and grounded in complex physical effects, yet often predictable: candles will flicker in certain ways, trees will sway. This predictability is ingrained in our human perception of real scenes: by viewing a still image, we can imagine a distribution of natural motions conditioned on that image. Recent advances in generative models, in particular conditional diffusion models, have enabled us to model rich distributions, including distributions of real images conditioned on text. In this paper, we explore modeling a generative prior for image-space scene motion, i.e., the motion of all pixels in a single image, trained on motion trajectories automatically extracted from a large collection of real video sequences.

We compute motion in the form of a spectral volume, a frequency-domain representation of dense, long-range pixel trajectories suited to scenes that exhibit oscillatory dynamics. We train a generative model that, conditioned on a single image, can sample spectral volumes from its learned distribution. The predicted spectral volume transforms into a motion texture — a set of per-pixel, long-range pixel motion trajectories — that can be used to animate the image. Compared with priors over raw RGB pixels, priors over motion capture more fundamental, lower-dimensional underlying structure that efficiently explains long-range variations in pixel values, leading to more coherent long-term generation and more fine-grained control over animations.

Generative synthesis: Recent advances in generative models have enabled photorealistic synthesis of images conditioned on text prompts. These models can be augmented to synthesize video sequences by extending the generated image tensors along a time dimension. While these methods produce plausible video sequences, the resulting videos often suffer from artifacts such as incoherent motion, unrealistic temporal variation in textures, and violations of physical constraints like preservation of mass. Animating images: Other techniques take a still picture and animate it. Many recent deep learning methods adopt a 3D-UNet architecture to produce video volumes directly, but because these models are effectively the same video generation models conditioned on image information instead of text, they exhibit similar artifacts. One way to overcome these limitations is to animate an input source image through explicit or implicit image-based rendering, moving the image content around according to motion derived from external sources. Animating images according to motion fields yields greater temporal coherence and realism, but prior methods require additional guidance signals or user input, or utilize limited motion representations.

Motion models and motion priors: In computer graphics, natural oscillatory 3D motion has long been modeled with noise shaped in the Fourier domain and then converted via an inverse Fourier transform to time-domain motion fields. Some of these methods rely on a modal analysis of the underlying dynamics. These spectral techniques were adapted to animate plants, water, and clouds from single 2D pictures by Chuang et al. with additional user annotations. Our work is particularly inspired by Davis et al., who showed how to connect modal analysis of a scene with the motions observed in video, and how to use this analysis to simulate interactive dynamics from a video. We adopt the spectral volume representation from Davis et al., extract this representation from a large set of training videos, and show that it is suitable for predicting motion from single images with diffusion models. Videos as textures: Certain moving scenes can be thought of as dynamic textures that model videos as space-time samples of a stochastic process. In contrast to much of previous work, our method learns priors in advance that can then be applied to single images.

Given a single picture I₀, our goal is to generate a video of length T featuring oscillation dynamics such as those of trees, flowers, or candle flames moving in the breeze. Our system has two modules: a motion prediction module and an image-based rendering module. We first use a latent diffusion model (LDM) to predict a spectral volume for the input image. The predicted spectral volume is then transformed to a sequence of motion displacement fields (a motion texture) using an inverse discrete Fourier transform. This motion determines the position of each input pixel at each future time step. Given a predicted motion texture, our rendering module animates the input RGB image using an image-based rendering technique that splats encoded features from the input image and decodes these splatted features into an output frame with an image synthesis network.

Formally, a motion texture is a sequence of time-varying 2D displacement maps, where the 2D displacement vector at each pixel coordinate from input image I₀ defines the position of that pixel at a future time t. If our goal is to produce a video via a motion texture, then one choice would be to predict a time-domain motion texture directly, but the size of the motion texture would need to scale with the length of the video: generating T output frames implies predicting T displacement fields. To avoid predicting such a large output representation, many prior animation methods either generate video frames autoregressively, or predict each future output frame independently via an extra time embedding. However, neither strategy ensures long-term temporal consistency.

Fortunately, many natural motions can be described as a superposition of a small number of harmonic oscillators represented with different frequencies, amplitudes, and phases. Because the underlying motions are quasi-periodic, it is natural to model them in the frequency domain. We adopt an efficient frequency space representation called a spectral volume from Davis et al. A spectral volume is the temporal Fourier transform of pixel trajectories extracted from a video, organized into images called modal images. Davis et al. further show that, under certain assumptions, the spectral volume evaluated at certain frequencies forms an image-space modal basis that is a projection of the vibration modes of the underlying scene. Prior work in real-time animation has observed that most natural oscillation motions are composed primarily of low-frequency components. We validated this observation: the power spectrum of the motion decreases exponentially with increasing frequency. In practice, we found that the first K=16 Fourier coefficients are sufficient to realistically reproduce the original natural motion.

We select a latent diffusion model (LDM) as the backbone for our motion prediction module, as LDMs are more computationally efficient than pixel-space diffusion models while preserving generation quality. A standard LDM consists of two main modules: (1) a Variational Autoencoder (VAE) that compresses the input to a latent space through an encoder z=E(I), then reconstructs via a decoder I=D(z); and (2) a U-Net based diffusion model that learns to iteratively denoise latent features starting from Gaussian random noise. Our training applies this not to input images but to motion spectra from real video sequences, which are encoded and then diffused for n steps with a pre-defined variance schedule to produce noisy latents zⁿ. The 2D U-Nets are trained to denoise the noisy latents by iteratively estimating the noise used to update the latent feature at each step.

Frequency Adaptive Normalization: One issue we observed is that motion textures have particular distribution characteristics across frequencies — the amplitude spans a range of 0 to 100 and decays approximately exponentially with increasing frequency. As diffusion models require output values between 0 and 1 for stable training, we must normalize the coefficients. If we simply scale the magnitudes based on image width and height, almost all coefficients at higher frequencies will end up close to zero. Models trained on such data can produce inaccurate motions, since even small prediction errors lead to large relative errors after denormalization. To address this, we employ a frequency adaptive normalization technique: we independently normalize Fourier coefficients at each frequency based on statistics computed from the training set. At each individual frequency f_j, we compute the 97th percentile of the Fourier coefficient magnitudes over all input samples as a per-frequency scaling factor. Furthermore, we apply a square root transform to each scaled coefficient to pull it away from extremely small or large values.

Frequency-Coordinated Denoising: The straightforward way to predict a spectral volume with K frequency bands is to output a tensor of 4K channels from a standard diffusion U-Net, but training a model to produce such a large number of channels tends to produce over-smoothed and inaccurate output. An alternative would be to independently predict a motion spectrum map at each individual frequency, but this results in uncorrelated predictions leading to unrealistic motion. Therefore, we propose a frequency-coordinated denoising strategy. We first train an LDM to predict a spectral volume texture map at each individual frequency f_j with extra frequency embedding along with time-step embedding. We then freeze the parameters and introduce attention layers interleaved with 2D spatial layers across K frequency bands. The frequency attention layers coordinate the pre-trained motion latent features across all frequency channels to produce coherent spectral volumes. The average VAE reconstruction error improves from 0.024 to 0.018 when switching from a standard 2D U-Net to our frequency-coordinated denoising module.

We now describe how we take a spectral volume predicted for a given input image I₀ and render a future frame at time t. We first derive motion trajectory fields in the time domain using the inverse temporal FFT applied at each pixel. The motion trajectory fields determine the position of every input pixel at every future time step. To produce a future frame, we adopt a deep image-based rendering technique and perform splatting with the predicted motion field to forward warp the encoded I₀. Since forward warping can lead to holes, and multiple source pixels can map to the same output 2D location, we adopt the feature pyramid softmax splatting strategy proposed in prior work on frame interpolation.

We encode I₀ through a feature extractor network to produce a multi-scale feature map. For each individual feature map at scale j, we resize and scale the predicted 2D motion field according to the resolution. As in Davis et al., we use predicted flow magnitude as a proxy for depth to determine the contributing weight of each source pixel: W(p) = 1/T sum_t ||F_t(p)||_2, computed as the average magnitude of the predicted motion trajectory fields. In other words, we assume large motions correspond to moving foreground objects, and small or zero motions correspond to background objects. We use motion-derived weights instead of learnable ones because in the single-view case, learnable weights are not effective for addressing disocclusion ambiguities. The warped features are then injected into intermediate blocks of an image synthesis decoder network to produce the final rendered image. We jointly train the feature extractor and synthesis networks with start and target frames randomly sampled from real videos, supervising predictions with a VGG perceptual loss.

Image-to-video: Our system enables the animation of a single still picture by first predicting a spectral volume and generating an animation by applying the rendering module. Since the method explicitly models scene motion, this allows production of slow-motion videos by linearly interpolating the motion displacement fields and magnifying or minifying animated motions by adjusting the amplitude of predicted spectral volume coefficients. Seamless looping: It is sometimes useful to generate videos with motion that loops seamlessly. Unfortunately, it is hard to find a large collection of seamlessly looping videos for training. Instead, we devise a motion self-guidance technique that guides the motion denoising sampling process using explicit looping constraints: at each iterative denoising step during inference, an additional motion guidance signal is incorporated alongside standard classifier-free guidance, where each pixel's position and velocity at the start and end frames are enforced to be as similar as possible.

Interactive dynamics from a single image: As shown by Davis et al., the image-space motion spectrum from an observed video, under certain assumptions, is proportional to the projections of vibration mode shapes, and thus a spectral volume can be interpreted as an image-space modal basis. The modal shapes capture underlying oscillation dynamics at different frequencies, and hence can be used to simulate the object's response to a user-defined force such as poking or pulling. We adopt the modal analysis technique, assuming the motion can be explained by the superposition of a set of harmonic oscillators. This allows the image-space 2D motion displacement field for the object's physical response to be written as a weighted sum of Fourier spectrum coefficients modulated by the state of complex modal coordinates at each simulated time step. The state is simulated via a forward Euler method applied to the equations of motion for a decoupled mass-spring-damper system. Our method produces an interactive scene from a single picture, whereas prior methods required a video as input.

Implementation: We use an LDM as the backbone with a VAE of continuous latent dimension 4, trained with an L1 reconstruction loss, a multi-scale gradient consistency loss, and a KL-divergence loss. A 2D U-Net is trained with MSE loss, and attention layers adopted for frequency-coordinated denoising. The VAE and LDM are trained on images of size 256x160, taking approximately 6 days to converge using 16 Nvidia A100 GPUs. For inference, the motion diffusion model runs with DDIM for 250 steps. Generated videos up to 512x288 resolution are created by fine-tuning on higher resolution data. ResNet-34 is adopted as the feature extractor, and the image synthesis network is based on a co-modulation StyleGAN architecture. The rendering module runs in real-time at 25 FPS on a Nvidia V100 GPU. Data: We collected a set of 3,015 videos of natural oscillatory phenomena, yielding more than 150K samples of image-motion pairs after preprocessing.

Quantitative results: Our approach significantly outperforms prior single-image animation baselines in terms of both image and video synthesis quality. Specifically, we achieve FID of 4.03, KID of 0.08 (x100), FVD of 47.1, and DTFVD of 2.53, compared to the best prior methods achieving FID of 10.4 (Endo et al.) and FVD of 166.0. Our much lower FVD and DT-FVD distances suggest that our generated videos are more realistic and more temporally coherent. Sliding window analysis shows that thanks to the global spectral volume representation, our generated videos do not suffer from drift or degradation over time, maintaining consistent quality across the entire video length. Ablation studies confirm that K=16 provides optimal performance; frequency adaptive normalization improves over simple scaling; and frequency-coordinated denoising outperforms both independent prediction and volume-based prediction. The full model also benefits from feature pyramid softmax splatting over average splatting and baseline splatting in the rendering module.

Qualitative results: Spatio-temporal X-t slice visualizations demonstrate that our generated video dynamics more strongly resemble the motion patterns observed in corresponding real reference videos, compared to other methods. Baselines such as Stochastic I2V and MCVD fail to model both appearance and motion realistically over time. Endo et al. produces frames with fewer artifacts but exhibits over-smooth or non-oscillation motions. Comparisons of individual generated frames at t=128 show that our frames exhibit fewer artifacts and distortions, and our 2D motion fields most resemble the reference displacement fields from real videos. In contrast, background content generated by other methods tends to drift, and video frames from other methods exhibit significant color distortion or ghosting artifacts, suggesting that baselines are less stable when generating videos with long time duration.

Limitations: Since our approach only predicts spectral volumes at lower frequencies, it might fail to model general non-oscillating motions or high-frequency vibrations such as those of musical instruments. Furthermore, the quality of generated videos relies on the quality of the motion trajectories estimated from real video sequences. Thus, animation quality can degrade if the motion in training videos consists of large displacements. Moreover, since the approach is based on image-based rendering from input pixels, animation quality can also degrade if the generated videos require the creation of large amounts of content unseen in the input frame.

We present a new approach for modeling natural oscillation dynamics from a single still picture. Our image-space motion prior is represented with spectral volumes, a frequency representation of per-pixel motion trajectories, which we find to be highly suitable for prediction with diffusion models, and which we learn from collections of real world videos. The spectral volumes are predicted using our frequency-coordinated latent diffusion model and are used to animate future video frames using a neural image-based rendering module. We show that our approach produces photo-realistic animations from a single picture and significantly outperforms prior baseline methods, and that it can enable other downstream applications such as creating seamless loops and interactive animations.