We present high quality image synthesis results using diffusion probabilistic models, a class of latent variable models inspired by considerations from nonequilibrium thermodynamics. Our best results are obtained by training on a weighted variational bound designed according to a novel connection between diffusion probabilistic models and denoising score matching with Langevin dynamics, and our models naturally admit a progressive lossy decompression scheme that can be interpreted as a generalization of autoregressive decoding. On the unconditional CIFAR10 dataset, we obtain an Inception score of 9.46 and a state-of-the-art FID score of 3.17. On 256x256 LSUN, we obtain sample quality similar to ProgressiveGAN.

Deep generative models of all kinds have recently demonstrated high quality samples in a wide variety of data modalities. GANs, autoregressive models, flows, and VAEs have synthesized striking image and audio samples, and energy-based modeling and score matching have produced images comparable to those of GANs. This paper presents progress in diffusion probabilistic models. A diffusion probabilistic model is a parameterized Markov chain trained using variational inference to produce samples matching the data after finite time. Transitions of the chain learn to reverse a diffusion process, which is a Markov chain that gradually adds noise to the data in the opposite direction of sampling until signal is destroyed.

Diffusion models are straightforward to define and efficient to train, but to the best of our knowledge, there has been no demonstration that they are capable of generating high quality samples. We show that diffusion models actually are capable of generating high quality samples, sometimes better than the published results on other types of generative models. In addition, we show that a certain parameterization of diffusion models reveals an equivalence with denoising score matching over multiple noise levels during training and with annealed Langevin dynamics during sampling. We obtained our best sample quality results using this parameterization, so we consider this connection to be one of our primary contributions.

Despite their sample quality, our models do not have competitive log likelihoods compared to other likelihood-based models (although our log likelihoods are better than those obtained by large estimates reported for energy based models and score matching). We find that the majority of our models' lossless codelengths are consumed to describe imperceptible image details. We present a more refined analysis of this phenomenon in terms of lossy compression, and we show that the sampling procedure of diffusion models is a type of progressive decoding that resembles autoregressive decoding along a generalized bit ordering.

Diffusion probabilistic models are latent variable models of the form p_θ(x₀) := ∫ p_θ(x_0:T) dx_1:T, where x₁, ..., x_T are latents of the same dimensionality as the data x₀ ~ q(x₀). The joint distribution p_θ(x_0:T) is called the reverse process, defined as a Markov chain with learned Gaussian transitions starting at p(x_T) = N(x_T; 0, I). What distinguishes diffusion models from other latent variable models is that the approximate posterior q(x_1:T|x₀), called the forward process or diffusion process, is fixed to a Markov chain that gradually adds Gaussian noise according to a variance schedule β₁, ..., β_T.

A notable property of the forward process is that it admits sampling x_t at an arbitrary timestep t in closed form: using the notation α_t := 1 - β_t and α_t := ∏_s=1^t α_s, we have q(x_t|x₀) = N(x_t; √α_t x₀, (1 - α_t)I). Training is efficient by optimizing random terms of the variational bound L with stochastic gradient descent. The forward process posteriors, conditioned on x₀, are tractable: q(x_t-1|x_t, x₀) = N(x_t-1; μ_t(x_t, x₀), β_tI), enabling closed-form KL divergence computation rather than high-variance Monte Carlo estimates.

The variational bound can be further rewritten to be evaluated with closed-form expressions rather than Monte Carlo estimates, decomposing into: L_T, which compares the final latent distribution to the prior; L_t-1 terms for 1 < t ≤ T, each a KL divergence between two Gaussians that can be computed exactly; and L₀, a reconstruction term. All KL divergences are between Gaussians, so they can be calculated in closed form instead of with high variance Monte Carlo estimates, a significant advantage over other variational methods that require such approximations.

We ignore the fact that the forward process variances β_t are learnable by reparameterization and instead fix them to constants. Thus, the approximate posterior q has no learnable parameters, so L_T is a constant during training and can be ignored. This design choice simplifies the optimization and removes the need to jointly learn the forward process, focusing all capacity on the reverse process parameterization.

For the reverse process p_θ(x_t-1|x_t) = N(x_t-1; μ_θ(x_t, t), Σ_θ(x_t, t)), we first set Σ_θ(x_t, t) = σ_t²I to untrained time dependent constants. Experimentally, both σ_t² = β_t and σ_t² = β_t had similar results. With this choice, L_t-1 reduces to comparing the true denoising mean to the predicted mean μ_θ. The training objective becomes: L_t-1 = E_q[ (1/2σ_t²) || μ̃_t(x_t, x₀) - μ_θ(x_t, t) ||² ] + C, where C is a constant independent of θ.

We further reparameterize using x_t(x₀, ε) = √α_t x₀ + √(1 - α_t) ε for ε ~ N(0, I), which reveals that μ_θ must predict (1/√α_t)(x_t - (β_t/√(1 - α_t)) ε). We propose the parameterization: μ_θ(x_t, t) = (1/√α_t)(x_t - (β_t/√(1 - α_t)) ε_θ(x_t, t)), where ε_θ is a neural network that predicts the noise ε from x_t. The sampling procedure then computes x_t-1 = (1/√α_t)(x_t - (β_t/√(1 - α_t)) ε_θ(x_t, t)) + σ_tz, resembling Langevin dynamics with ε_θ as a learned gradient of the data density.

With the reverse process and decoder defined above, the variational bound is clearly differentiable with respect to θ and ready for training. However, we found it beneficial to sample quality to train on a simplified variant: L_simple(θ) := E_t,x0,ε [ || ε - ε_θ(√α_t x₀ + √(1 - α_t) ε, t) ||² ], where t is uniform between 1 and T. Since our simplified objective discards the weighting in the original variational bound, it is a weighted variational bound that emphasizes different aspects of reconstruction. Our diffusion process setup causes the simplified objective to down-weight loss terms corresponding to small t, training the network to focus on more difficult, larger noise-level denoising tasks.

To summarize, training amounts to: (1) sample x₀ from the data, (2) sample t uniformly from {1,...,T}, (3) sample ε ~ N(0, I), (4) take a gradient descent step on || ε - ε_θ(√α_t x₀ + √(1 - α_t) ε, t) ||². Sampling amounts to: start from x_T ~ N(0, I), then iteratively compute x_t-1 from x_t using ε_θ for t = T, T-1, ..., 1. The algorithm is remarkably simple: it resembles denoising score matching training over multiple noise levels with annealed Langevin dynamics sampling. The complete procedure uses a U-Net backbone with group normalization, self-attention at 16x16 resolution, and Transformer sinusoidal position embeddings for time conditioning.

We set T = 1000 for all experiments. The forward process variances are set to constants increasing linearly from β₁ = 10^-4 to β_T = 0.02. These were chosen to be small relative to data scaled to [-1, 1], ensuring that the reverse and forward processes have approximately the same functional form while minimizing the signal-to-noise ratio at x_T. On CIFAR10, our model achieves an Inception Score of 9.46 and FID of 3.17. Our unconditional model achieves better sample quality than most models in the literature, including class conditional models. We find that training on the true variational bound yields better codelengths, but the simplified objective yields the best sample quality.

Ablation studies reveal several key findings. Predicting ε, as proposed, performs approximately as well as predicting the posterior mean when trained on the variational bound with fixed variances, but much better with the simplified objective. Learning reverse process variances Σ_θ via a parameterized diagonal led to unstable training and poorer sample quality compared to fixed variances. On LSUN 256x256, our samples achieve quality similar to ProgressiveGAN, a strong GAN baseline. These results confirm that each design choice — fixed variances, ε-prediction, and simplified objective — contributes to the final performance.

Our analysis of progressive coding reveals that the majority of the lossless codelength describes imperceptible image details. While lossless codelengths are not competitive with other likelihood-based generative models, we conclude that diffusion models have an inductive bias that makes them excellent lossy compressors. Treating L₁+...+L_T as rate and L₀ as distortion, our best CIFAR10 model has 1.78 bits/dim rate and 1.97 bits/dim distortion (corresponding to 0.95 RMSE on a 0-255 scale). More than half of the lossless codelength describes imperceptible distortions. The progressive generation reveals that large scale image features appear first and fine details appear last, hinting at what may be understood as conceptual compression.

We have presented high quality image samples using diffusion models, and we have found connections among diffusion models and variational inference for training Markov chains, denoising score matching and annealed Langevin dynamics (and energy-based models by extension), autoregressive models, and progressive lossy compression. Since diffusion models seem to have excellent inductive biases for image data, investigating their utility for other data modalities and as components in other types of generative models and machine learning systems is a fruitful area for future work.