By decomposing the image formation process into a sequential application of denoising autoencoders, diffusion models (DMs) can achieve impressive results in image synthesis and beyond. Additionally, their formulation allows for a guiding mechanism to control the image generation process without retraining. However, since these models typically operate directly in pixel space, optimization of powerful DMs often consumes hundreds of GPU days and inference is expensive due to sequential evaluations. To enable DM training on limited computational resources while retaining their quality and flexibility, we apply them in the latent space of powerful pretrained autoencoders. In contrast to previous work, training diffusion models on such a representation allows for the first time to reach a near-optimal point between complexity reduction and detail preservation, greatly boosting visual fidelity.

By introducing cross-attention layers into the model architecture, we turn diffusion models into powerful and flexible generators for general conditioning inputs such as text or bounding boxes and enable high-resolution synthesis in a convolutional manner. Our latent diffusion models (LDMs) achieve a new state of the art for image inpainting and highly competitive performance on various tasks, including unconditional image generation, semantic scene synthesis, and super-resolution, while significantly reducing computational requirements compared to pixel-based diffusion models.

Image synthesis is one of the central goals of computer vision and graphics, holding enormous commercial potential in content creation, image editing, and data augmentation. Recent advances in diffusion models have produced impressive results across many tasks, from class-conditional image generation to super-resolution, inpainting, and text-guided synthesis. However, the quality of these models comes at a price: training state-of-the-art diffusion models can require hundreds of GPU days, and their sequential sampling process makes inference costly. This raises the question of whether there is a way to achieve the quality of DMs while being more efficient.

Our key insight is that likelihood-based models tend to spend a disproportionate amount of capacity on modeling imperceptible, high-frequency details of the data. We propose to explicitly separate the compressive learning phase from the generative learning phase by first training an autoencoder that provides a lower-dimensional and thereby efficient representational space, and then training diffusion models in this latent space. We call the resulting models Latent Diffusion Models (LDMs).

Our approach offers several advantages: (i) LDMs scale more gracefully than pixel-based or transformer-based approaches, enabling faithful and detailed reconstructions from compressed latent representations; (ii) they achieve competitive performance on multiple generation tasks while significantly lowering computational costs; (iii) by avoiding the need to delicately weigh reconstruction against generative learning, we decouple the autoencoder training from the diffusion model training; (iv) the convolutional nature of our latent space enables application to high-resolution images of approximately 1024 x 1024 pixels; (v) we design a general cross-attention-based conditioning mechanism for multi-modal training; and (vi) we release pretrained latent diffusion and autoencoding models.

Generative Adversarial Networks (GANs) allow for efficient sampling of high-resolution images with good perceptual quality but can be difficult to optimize and struggle to capture the full data distribution, leading to mode collapse. Variational Autoencoders (VAEs) and flow-based models enable efficient synthesis but typically produce blurrier samples compared to GANs. Autoregressive models (ARMs) achieve strong performance in density estimation but are computationally expensive at inference due to their sequential nature, and scaling to higher-resolution images remains challenging.

Diffusion models have recently emerged as a powerful class of generative models, achieving state-of-the-art results in image synthesis. However, they operate in pixel space, requiring extensive computational resources. Recent two-stage approaches attempt to address efficiency concerns: VQ-VAEs use autoregressive models over discretized latent codes, while VQGANs employ adversarial training objectives for the first stage. However, the high compression rates required for feasible autoregressive training limit the overall performance of these models. Our LDMs scale more gently to higher-dimensional latent spaces due to their convolutional backbone, avoiding the restrictive compression bottleneck.

Existing efforts to improve the efficiency of diffusion models include faster sampling strategies such as DDIM and advanced noise scheduling techniques. Other works explore cascaded generation pipelines that first synthesize low-resolution images and then apply diffusion-based super-resolution models. While these approaches reduce inference cost, they do not fundamentally address the computational burden of training in high-dimensional pixel space. Our approach is orthogonal and complementary: by moving the diffusion process into a learned latent space, we reduce both training and sampling costs at the architectural level.

Our perceptual compression model is based on an autoencoder trained with a combination of a perceptual loss and a patch-based adversarial objective. Given an image x in R^(H x W x 3), the encoder E encodes x into a latent representation z = E(x), and the decoder D reconstructs the image from the latent, giving x_tilde = D(z) = D(E(x)). The encoder downsamples the image by a factor f = H/h = W/w, where we investigate different downsampling factors f in {1, 2, 4, 8, 16, 32}.

To avoid arbitrarily high-variance latent spaces, we experiment with two kinds of regularizations. The first, KL-reg, imposes a slight KL-penalty towards a standard normal distribution on the learned latent, similar to a VAE. The second, VQ-reg, uses a vector quantization layer within the decoder. Because our subsequent diffusion model is designed to work with the two-dimensional structure of the learned latent space, we can use relatively mild compression rates and achieve very good reconstructions. This is in contrast to previous works that relied on aggressive, arbitrarily high compression of the learned space for their autoregressive models.

Diffusion models are probabilistic models designed to learn a data distribution p(x) by gradually denoising a normally distributed variable. The learning corresponds to a fixed-length Markov chain of T steps. The training objective can be simplified to a denoising score-matching objective: L_DM = E[ || epsilon - epsilon_theta(x_t, t) ||^2 ] with t uniformly sampled from {1, ..., T}, where epsilon_theta is the denoising network (a time-conditional UNet) predicting the noise added to the input.

For our Latent Diffusion Models, we leverage the perceptual compression model to obtain an efficient, low-dimensional latent space in which high-frequency, imperceptible details are abstracted away. Compared to the high-dimensional pixel space, this space is more suitable for likelihood-based generative models, as they can now focus on the important, semantic bits of the data. The diffusion loss becomes: L_LDM := E[ || epsilon - epsilon_theta(z_t, t) ||^2 ], where z_t is obtained from the encoder E during training. The neural backbone epsilon_theta is realized as a time-conditional UNet. Since the forward process is fixed, z_t can be efficiently obtained from E during training, and samples can be decoded to image space with a single pass through D.

Beyond unconditional generation, diffusion models can model conditional distributions p(z|y) to guide the synthesis process. We augment the underlying UNet backbone with the cross-attention mechanism, which is effective for learning attention-based models of various input modalities. To pre-process conditioning inputs y from various domains (e.g., text prompts, semantic maps, or other image-to-image translation inputs), we introduce a domain-specific encoder tau_theta that projects y to an intermediate representation tau_theta(y) in R^(M x d_tau), which is then mapped to the intermediate layers of the UNet via the cross-attention layer.

The cross-attention is implemented as: Attention(Q, K, V) = softmax(QK^T / sqrt(d)) * V, where the queries Q = W_Q * phi_i(z_t) come from a flattened intermediate representation of the UNet, and the keys and values K = W_K * tau_theta(y), V = W_V * tau_theta(y) come from the conditioning encoder output. This mechanism is general enough to subsume class-conditional generation (y is a class label), text-to-image generation (y is text encoded by a language model such as a BERT tokenizer), and layout-to-image generation (y is a semantic layout). The complete objective then combines the LDM loss with the conditioning: L_LDM := E[ || epsilon - epsilon_theta(z_t, t, tau_theta(y)) ||^2 ].

We first analyze the impact of the downsampling factor f on LDM performance. Our experiments reveal that small downsampling factors (LDM-1, LDM-2) result in slow training progress, as the diffusion model must still handle relatively high-dimensional latent spaces. Conversely, overly large values of f cause stagnating fidelity after an initial fast convergence, since too much perceptual detail is lost during compression. We find that LDM-4 through LDM-8 strike a good balance between efficiency and perceptually faithful results, with LDM-4 and LDM-8 offering the best trade-off across most evaluated metrics and tasks.

On unconditional image generation, we evaluate on CelebA-HQ 256x256, FFHQ 256x256, LSUN-Churches 256x256, and LSUN-Bedrooms 256x256. Our LDM-4 achieves a FID of 5.11 on CelebA-HQ, establishing a new state-of-the-art result. Across datasets, LDMs outperform prior diffusion-based approaches on most benchmarks while using roughly half the parameters of the leading competitor ADM. Notably, LDMs deliver these results with substantially fewer computational resources, requiring only a fraction of the training compute of pixel-based diffusion models.

For text-to-image synthesis, we train a 1.45 billion parameter KL-regularized LDM conditioned on language representations from a BERT tokenizer, on the LAION-400M dataset. Evaluated on MS-COCO, our model achieves an FID of 12.63 with classifier-free guidance, which is on par with recent state-of-the-art autoregressive and diffusion methods while requiring significantly less computational overhead. We note that this result is achieved without CLIP-based reranking or filtering strategies that are commonly applied by competing methods, suggesting further room for improvement.

In image inpainting, our model achieves state-of-the-art performance with a FID of 9.39 on the Places dataset, outperforming both previous diffusion-based and GAN-based inpainting methods. User preference studies confirm that evaluators prefer our inpainting results over those of competing approaches. For super-resolution, LDM-SR shows competitive performance, outperforming SR3 in terms of FID while SR3 achieves a better Inception Score. These results demonstrate the versatility of the LDM framework across diverse image synthesis tasks, consistently delivering strong performance with reduced computational requirements.

We have presented Latent Diffusion Models, an approach to significantly improve both the training and sampling efficiency of denoising diffusion models without degrading their quality. Based on an analysis of the interplay between the perceptual compression stage and the generative diffusion stage, we find that our approach achieves a near-optimal balance between computational cost and synthesis quality. Our cross-attention-based conditioning mechanism enables favorable results across a wide range of conditional image synthesis tasks without requiring task-specific architectures, establishing LDMs as versatile and efficient generative models.

Despite significant efficiency gains, the sequential sampling process of LDMs is still slower than that of GANs. Furthermore, the use of LDMs can be limited when high precision is required at the pixel level, as the reconstruction capability of the autoencoder can become a bottleneck for tasks that require fine-grained accuracy. The accessibility of powerful generative models enables various creative applications in art, design, and content creation, but also facilitates potential misuse such as deepfakes and misinformation. We emphasize the importance of responsible deployment and the need for continued research into training data privacy and bias mitigation.