This work targets high-quality text-to-video (T2V) generative model by leveraging a pre-trained text-to-image (T2I) model as a basis. The framework operates through cascaded video latent diffusion models, comprising a base T2V model, a temporal interpolation model, and a video super-resolution model. Key contributions include: simple temporal self-attentions with rotary positional encoding adequately capture temporal correlations, and joint image-video fine-tuning produces superior outcomes. The authors introduce Vimeo25M, "a novel text-video dataset consisting of 25 million text-video pairs." Final output reaches 1280x2048 resolution with 61 frames.

"Building upon the successes of T2I models, there has been a growing interest in extending these techniques to the synthesis of videos controlled by text inputs." The authors note that "training an entire T2V system from scratch poses significant challenges as it requires extensive computational resources." LaVie's approach leverages pre-trained models while addressing a critical issue: "fine-tuning solely on video datasets, even with the initialization from a pre-trained LDM, fails to achieve this goal due to the phenomenon of catastrophic forgetting." The framework comprises 3 billion parameters and enables both high-quality synthesis and creative generation capabilities.

"Previous works have leveraged various types of deep generative models, including GANs, VAEs, and VQ-based models" for unconditional video generation. Recent diffusion-based approaches show promise but "learning the entire distribution of video datasets in an unconditional manner remains highly challenging." For text-to-video generation, existing approaches primarily extend T2I models by "incorporating temporal modules, such as temporal convolutions and temporal attention." The paper identifies a key distinction: "In contrast to prior works, our approach distinguishes itself by augmenting a pre-trained Stable Diffusion model with an efficient temporal module and jointly fine-tuning the entire model on both image and video datasets."

The base model introduces two key modifications: First, "for each 2D convolutional layer, we inflate the pre-trained kernel to incorporate an additional temporal dimension." Second, the authors extend the transformer block to include "a temporal attention layer after each spatial layer." Critical to the approach is joint training: "We concatenate M images along the temporal axis to form a T-frame video and train the entire base model to optimize the objectives of both the T2I and T2V tasks." This addresses catastrophic forgetting by maintaining image generation knowledge while learning temporal patterns. The training objective combines video and image losses with balancing coefficient alpha.

This network "takes a 16-frame base video as input and produces an upsampled output consisting of 61 frames." The training approach involves duplicating base video frames and concatenating them with noisy high-frame-rate frames. Notably, "every frame in the output is newly synthesized" — each frame generated through interpolation replaces the corresponding input frame. The model is conditioned on text prompts for guided interpolation, maintaining semantic coherence across the temporal expansion.

The VSR component increases resolution to 1280x2048. The authors "leverage a pre-trained diffusion-based image 4x upscaler as a prior." Unlike the base model, "the spatial layers in the pre-trained upscaler remain fixed, our focus lies in fine-tuning the inserted temporal layers." Training uses patch-wise processing on 320x320 patches while maintaining capability for arbitrary sizes at inference. This design keeps the strong spatial prior intact while only learning temporal consistency.

On UCF-101, LaVie achieves FVD of 526.30 compared to Video LDM's 550.61, demonstrating 24.31 point improvement. For MSR-VTT CLIPSIM metric, the method achieves 0.2949, competitive with Make-A-Video's 0.3049. Human evaluation with 30 raters shows LaVie achieves 75.00% preference over ModelScope and 75.58% over VideoCrafter. However, challenges persist: "all three approaches struggle to achieve a satisfactory score in terms of 'motion smoothness,' indicating the ongoing challenge of generating coherent and realistic motion." The Vimeo25M dataset comprises 25 million text-video pairs in high-definition, widescreen, and watermark-free formats, with approximately 16.89% receiving aesthetics scores greater than 6, surpassing WebVid10M's 7.22%.

The paper presents LaVie as "a text-to-video foundation model that produces high-quality and temporally coherent results." Key achievements include leveraging cascaded diffusion models, introducing the Vimeo25M dataset, and demonstrating joint training effectiveness. The work "serves as an initial step towards achieving high-quality T2V generation," with future directions toward longer videos and movie-level quality synthesis from scripts. Acknowledged limitations include difficulty with multi-subject generation and accurate hand depiction.