DDPM: the forward process

The energy-based models note ended on a cost: an energy-based model is trained with short, biased MCMC chains run inside every parameter update, so the model it learns depends on its sampler as much as on its data. Diffusion models take sampling out of training. They fix a simple noising process that turns data into Gaussian noise and learn to undo it one small step at a time. Each step back is close to a Gaussian, so a network learns it by regression on noisy samples that the fixed process produces for free. That network turns out to learn the score of the noised data, the negative gradient of an energy and the direction that Langevin sampling follows (score matching note). The noisy latents keep the data’s dimension, the network is unconstrained, and the price is a bound on the likelihood instead of the likelihood itself. DDPM, Denoising Diffusion Probabilistic Models (Ho, Jain and Abbeel, 2020), is the model that brought diffusion to image quality comparable with GANs. This note covers its first half, the forward process: a fixed procedure that slowly turns an image into Gaussian noise. The second half, a neural network that runs the procedure backwards, is in the next note.

Notation. $x_0$ is a data point (an image) and $x_t$ its noised version after $t$ steps, $t = 1, \dots, T$. $q$ denotes the fixed forward (noising) process: $q(x_0)$ is the distribution of the data and $q(x_t \mid x_{t-1})$ one noising step. $p_\theta(x_{t-1} \mid x_t)$ is the learned reverse step, with network parameters $\theta$. $\varepsilon \sim \mathcal N(0, 1)$ is standard Gaussian noise drawn fresh at every step, and the second argument of $\mathcal N(\mu, \sigma^2)$ is the variance. We work in one dimension; for images every pixel is noised independently and $1$ becomes the identity matrix $I$, so the target is $\mathcal N(0, I)$.

A little history#

• In 2015, when GANs were the state of the art in image generation, Sohl-Dickstein et al. (2015) published Deep Unsupervised Learning using Nonequilibrium Thermodynamics. It introduced the idea of diffusion models: destroy the structure of the data with a slow noising process, then learn to reverse it.
• Five years later DDPM simplified the parameterization and the loss and reached image quality competitive with GANs. Its training objective is still derived from a variational lower bound on the likelihood, the same tool as in the VAE.
• Shortly afterwards diffusion models overtook GANs on standard image benchmarks (Dhariwal and Nichol, 2021).

The problem#

We do not know the distribution of real images, but we want to sample new images from it. In diffusion notation this distribution is written $q(x_0)$, because it is where the forward process $q$ starts.

The diffusion idea is to connect $q(x_0)$ to a distribution we can sample from. Take a real image $x_0$, the original photo, and add a little noise to get $x_1$, then a little more to get $x_2$, and so on until $x_T$ is pure noise. Each forward step is fixed and needs no learning. The model, the learned reverse step $p_\theta(x_{t-1} \mid x_t)$, goes the other way: from $x_t$ it produces $x_{t-1}$, one step back towards the image. To generate, we sample pure noise $x_T$ and apply the model $T$ times.

$$\begin{gathered} \underset{\text{image}}{x_0} \;\longrightarrow\; x_1 \;\longrightarrow\; \cdots \;\longrightarrow\; \underset{\text{noise}}{x_T}, \\ x_{t-1} \;\xleftarrow{\;p_\theta(x_{t-1} \mid x_t)\;}\; x_t . \end{gathered}$$

Figure

The forward chain adds noise; the model walks back

A first attempt: just add noise#

The simplest step adds noise with variance $\beta$:

$$\begin{gathered} x_1 = x_0 + \sqrt{\beta}\,\varepsilon, \quad \varepsilon \sim \mathcal N(0, 1) \\ \Longrightarrow\quad q(x_1 \mid x_0) = \mathcal N(x_0, \beta), \end{gathered}$$

where $\beta$ is the noise variance. Note the square root: $c\,\varepsilon$ has variance $c^2$, so noise with variance $\beta$ is $\sqrt\beta\,\varepsilon$. The next step does the same to $x_1$: $q(x_2 \mid x_1) = \mathcal N(x_1, \beta)$.

Now we can shorten the chain. Two steps add two independent noises, and the variances of independent Gaussians add up:

$$\begin{aligned} x_2 &= x_1 + \sqrt\beta\,\varepsilon_2 \\ &= x_0 + \sqrt\beta\,\varepsilon_1 + \sqrt\beta\,\varepsilon_2 \\ &= x_0 + \sqrt{2\beta}\,\varepsilon, \\ q(x_2 \mid x_0) &= \mathcal N(x_0, 2\beta), \end{aligned}$$

where $\varepsilon = (\varepsilon_1 + \varepsilon_2)/\sqrt2$ is again a standard normal. Whenever several noises are merged below, $\varepsilon$ denotes the merged standard normal in this sense, so such equalities hold in distribution.

Continuing in the same way, after $t$ steps

$$\begin{gathered} x_t = x_0 + \sqrt{t\beta}\,\varepsilon, \\ q(x_t \mid x_0) = \mathcal N(x_0, t\beta). \end{gathered}$$

This is a useful shortcut: we can jump from $x_0$ to any $x_t$ in one draw. But look at the two parameters:

• the mean does not change: it stays $x_0$ forever;
• the variance increases indefinitely: $t\beta \to \infty$.

So the chain never reaches $\mathcal N(0, 1)$: the variance grows without bound, and there is no fixed limiting distribution from which generation could start. The image’s influence fades only relative to the growing spread. This process is called variance-exploding diffusion (Song et al., 2021, §3.4). Models built on it stop at a large finite noise level $\sigma_{\max}$ and start sampling from $\mathcal N(0, \sigma_{\max}^2)$ (Song et al., 2021, §2.1).

The objective, and the rescaled step#

What we want is

$$q(x_t \mid x_0) \xrightarrow[t\to\infty]{} \mathcal N(0, 1),$$

whatever the starting image. This needs a new way to go from step $t-1$ to step $t$, and in particular the step has to change the mean each time, pulling it towards $0$. In general, Sohl-Dickstein et al. allow any Markov kernel $T_\pi(x_t \mid x_{t-1}; \beta_t)$ that leaves a simple target distribution $\pi$ unchanged, $\pi(y) = \int T_\pi(y \mid y'; \beta)\,\pi(y')\,dy'$ (2015, §2.1, Eq. 1–2): a binomial kernel for binary data, a Gaussian kernel for real values. For real values we look for a step of the form $x_t = a\,x_{t-1} + \sqrt\beta\,\varepsilon$. The Gaussian kernel of diffusion models, introduced by Sohl-Dickstein et al. (2015, Table App.1) and kept by DDPM (Ho, Jain and Abbeel, 2020, Eq. 2), scales the previous point by $a = \sqrt{1-\beta}$:

$$\begin{gathered} x_t = \sqrt{1-\beta}\;x_{t-1} + \sqrt{\beta}\;\varepsilon, \\ q(x_t \mid x_{t-1}) = \mathcal N\!\big(\sqrt{1-\beta}\;x_{t-1},\; \beta\big). \end{gathered}$$

Figure

One step at a time, or straight from x₀ to xₜ

Why exactly $\sqrt{1-\beta}$? The fresh noise is independent of $x_{t-1}$, so the variance obeys $\operatorname{Var}(x_t) = a^2 \operatorname{Var}(x_{t-1}) + \beta$. This recursion settles at $1$, the variance of the target, only if $a^2 + \beta = 1$. So $a = \sqrt{1-\beta}$ is the only positive scale that keeps a unit variance at $1$, and it also pulls any other variance towards $1$ (callout below). With this $a$ the jump from $x_0$ to any $x_t$ takes an especially simple closed form, whose two squared coefficients sum to $1$ (Ho, Jain and Abbeel, 2020, Eq. 4):

$$\begin{gathered} x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\;\varepsilon, \\ \bar\alpha_t = (1-\beta)^t, \end{gathered}$$ $$q(x_t \mid x_0) = \mathcal N\!\big(\sqrt{\bar\alpha_t}\,x_0,\; 1-\bar\alpha_t\big).$$

With $\bar\alpha_0 = 1$ the formula also covers $t = 0$. So we can produce $x_t$ for any timestep directly from $x_0$, in one draw, without running the $t$ steps before it.

Why this reaches N(0, 1)#

If $\beta \in (0, 1)$, then $\bar\alpha_t = (1-\beta)^t \to 0$ as $t \to \infty$. Then the image term $\sqrt{\bar\alpha_t}\,x_0 \to 0$, while the noise coefficient $\sqrt{1-\bar\alpha_t} \to 1$, so the noise term tends to plain $\varepsilon$ with variance $1$:

$$q(x_t \mid x_0) \xrightarrow[t\to\infty]{} \mathcal N(0, 1).$$

For example, with a constant $\beta = 0.02$:

$t$	$\bar\alpha_t = 0.98^t$	mean coefficient $\sqrt{\bar\alpha_t}$	noise std $\sqrt{1-\bar\alpha_t}$	naive noise std $\sqrt{t\beta}$
1	0.980	0.990	0.141	0.141
10	0.817	0.904	0.428	0.447
50	0.364	0.603	0.797	1.000
100	0.133	0.364	0.931	1.414
200	0.0176	0.133	0.991	2.000

For small $t$ the two chains almost agree. Then the naive one passes the target spread and keeps growing, while DDPM’s approaches $1$ from below. At a finite $T$ the image term $\sqrt{\bar\alpha_T}\,x_0$ is small only relative to the scale of $x_0$, which is one reason images are scaled to $[-1, 1]$ before diffusion (next note).

Figure

Variance exploding versus variance preserving

The same happens to the whole data distribution. Its marginal at step $t$ is $q(x_t) = \int q(x_t \mid x_0)\,q(x_0)\,dx_0$. To see that it also tends to $\mathcal N(0, 1)$, write $x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\varepsilon$ with $x_0 \sim q(x_0)$ independent of $\varepsilon$. The first term is a fixed random variable times a factor that goes to $0$, so it tends to $0$ for any data distribution, and the second term tends to $\varepsilon$; hence $x_t \to \mathcal N(0, 1)$ in distribution. In one dimension, a two-humped density spreads out, its humps move towards $0$ and merge (between $t = 200$ and $t = 300$ for the data in the figure below), and it ends as a single bell curve centred at $0$. A finite batch of samples looks jagged along the way, but the density itself stays smooth: it is the data density convolved with a Gaussian. For a mixture of Gaussians the marginal stays a mixture and can be written down exactly:

$$\begin{aligned} q(x_0) &= \textstyle\sum_k \pi_k\,\mathcal N(\mu_k, \sigma_k^2), \\ q(x_t) &= \textstyle\sum_k \pi_k\,\mathcal N\!\big(\sqrt{\bar\alpha_t}\,\mu_k,\; s_{k,t}^2\big), \\ s_{k,t}^2 &= \bar\alpha_t\sigma_k^2 + 1 - \bar\alpha_t. \end{aligned}$$

Both components are pulled towards mean $0$ and variance $1$, so they merge into one standard normal.

Figure

The data distribution dissolves into N(0, 1)

In practice: a β schedule#

In practice $\beta$ is not constant but varies with the step, $\beta_1, \dots, \beta_T$; this is the schedule. The derivation above goes through unchanged with $(1-\beta)$ replaced by $(1-\beta_t)$ at step $t$:

$$\begin{gathered} q(x_t \mid x_{t-1}) = \mathcal N\!\big(\sqrt{1-\beta_t}\,x_{t-1},\; \beta_t\big), \\ \bar\alpha_t = \textstyle\prod_{i=1}^{t} (1-\beta_i), \\ q(x_t \mid x_0) = \mathcal N\!\big(\sqrt{\bar\alpha_t}\,x_0,\; 1-\bar\alpha_t\big). \end{gathered}$$

The usual shorthand is $\alpha_t = 1-\beta_t$, so $\bar\alpha_t = \prod_{i \le t}\alpha_i$. The DDPM paper uses $T = 1000$ steps and a linear schedule from $\beta_1 = 10^{-4}$ to $\beta_T = 0.02$ (Ho, Jain and Abbeel, 2020, §4). Later work found that a cosine-shaped schedule, which destroys information more evenly over time, works better at low resolution (Nichol and Dhariwal, 2021, §3.2). Small steps at the start keep the early $x_t$ close to the image; larger steps later finish the job. With this schedule $\bar\alpha_t$ is about $0.5$ at $t \approx 259$ (it drops below $0.5$ at $t = 260$), falls below $0.01$ at $t = 674$ and ends at $\bar\alpha_{1000} \approx 4.0\times10^{-5}$, so $x_{1000}$ is noise for all practical purposes.

Figure

A linear β schedule and the resulting α̅ₜ

A schedule can also be read as a signal-to-noise ratio, $\mathrm{SNR}_t = \bar\alpha_t / (1 - \bar\alpha_t)$: the squared coefficient of the image term divided by the variance of the noise, which is the ratio of the two variances for data of unit variance. For the linear schedule it falls from about $10^4$ at $t = 1$ to $4\times10^{-5}$ at $t = T$. The fast samplers of the latent diffusion note work with its logarithm.

Linear, quadratic, sigmoid and cosine schedules#

The linear schedule is one choice among several (Weng, 2021, reviews them). Write $u_t = (t - 1)/(T - 1) \in [0, 1]$ for the position in the chain. With the same endpoints $\beta_1 = 10^{-4}$ and $\beta_T = 0.02$, three common shapes are defined through $\beta_t$:

$$\begin{aligned} \text{linear:}\quad & \beta_t = \beta_1 + (\beta_T - \beta_1)\,u_t, \\ \text{quadratic:}\quad & \beta_t = \big(\sqrt{\beta_1} + (\sqrt{\beta_T} - \sqrt{\beta_1})\,u_t\big)^2, \\ \text{sigmoid:}\quad & \beta_t = \beta_1 + (\beta_T - \beta_1)\,\operatorname{sigmoid}(12\,u_t - 6). \end{aligned}$$

The quadratic schedule interpolates the noise amplitude $\sqrt{\beta_t}$ linearly, so its steps stay small for longer. The sigmoid schedule is flat at both ends and makes its change in the middle; since $\operatorname{sigmoid}(\pm 6)$ is not exactly $0$ or $1$, its $\beta_t$ actually run from $1.5\times10^{-4}$ to $0.01995$. The cosine schedule (Nichol and Dhariwal, 2021, §3.2, Eq. 17) prescribes $\bar\alpha_t$ instead and derives the steps from it:

$$\begin{gathered} \bar\alpha_t = \frac{f(t)}{f(0)}, \qquad f(t) = \cos^2\!\Big(\frac{t/T + s}{1 + s}\cdot\frac{\pi}{2}\Big), \\ \beta_t = 1 - \frac{\bar\alpha_t}{\bar\alpha_{t-1}} , \end{gathered}$$

with the offset $s = 0.008$, chosen so that the noise amplitude of the first step stays just below the width $1/127.5$ of one pixel level, and with $\beta_t$ clipped at $0.999$. Without the clip, $f(T) = 0$ would give $\beta_T = 1$; with it, $\bar\alpha_{1000} \approx 2.4 \times 10^{-9}$.

Figure

Four noise schedules: how α̅ₜ falls

With $T = 1000$:

schedule	$\bar\alpha_t < 0.5$ from	$\bar\alpha_t < 0.01$ from	$\bar\alpha_{1000}$
linear	$t = 260$	$t = 674$	$4.0 \times 10^{-5}$
quadratic	$t = 419$	$t = 851$	$7.3 \times 10^{-4}$
sigmoid	$t = 440$	$t = 721$	$4.0 \times 10^{-5}$
cosine	$t = 497$	$t = 936$	$2.4 \times 10^{-9}$

The linear schedule destroys the image fastest: half of the signal variance is gone by $t = 260$, and from $t = 674$ on less than $1\%$ is left, so the last third of the chain works on almost pure noise. Nichol and Dhariwal found this wasteful at $32 \times 32$ and $64 \times 64$: a model trained with the linear schedule barely loses FID when the first $20\%$ of the reverse process, its noisiest end, is skipped. Their cosine schedule was designed to make $\bar\alpha_t$ fall almost linearly in the middle and change little near both ends; it crosses $0.5$ close to $T/2$. On ImageNet $64 \times 64$ ($T = 4000$, $L_{\text{simple}}$, 200K iterations) it improved FID from $31.3$ to $27.0$ and the likelihood from $3.77$ to $3.68$ bits per dimension (their Table 1). The quadratic schedule shows the opposite risk: it ends at $\bar\alpha_{1000} \approx 7.3 \times 10^{-4}$, so a fraction $\sqrt{\bar\alpha_T} \approx 0.027$ of the image survives in $x_T$, and $q(x_T \mid x_0)$ sits further from the $\mathcal N(0, I)$ where sampling starts than for the other three.

Notebook#

The diffusion notebook of the HSE-LAMBDA Deep Generative Models repository (pinned commit; cell numbers count every cell from 1) is based on the ACIDS-IRCAM diffusion tutorial. It has no saved outputs, so it documents procedures rather than results.

The notebook uses toy settings: a 2-D Swiss roll and $T = 100$ steps. Its first demo runs the step-by-step chain with a constant $\beta = 0.0005$, for which $\bar\alpha_{100} \approx 0.951$. The rest of the notebook uses a sigmoid-shaped schedule with start $10^{-5}$ and end $10^{-2}$, for which $\bar\alpha_{100} \approx 0.605$; since the sigmoid of $\pm 6$ is not exactly $0$ or $1$, the actual $\beta_t$ run from about $3.5\times10^{-5}$ to $1.0\times10^{-2}$. It samples $x_t$ directly with the closed form (q_sample, alphas_bar_sqrt, one_minus_alphas_bar_sqrt). Neither toy schedule gets close to $\bar\alpha = 0$: the mean coefficient $\sqrt{\bar\alpha_{100}}$ is still about $0.98$ and $0.78$, so $x_{100}$ carries much of the data signal. Real models need many more steps or larger $\beta$.

• Cells 4–6 sample the 2-D Swiss roll used throughout.
• Cells 8–10: the markdown of cell 8 introduces the general Markov kernel $T_\pi(x_t \mid x_{t-1}; \beta_t)$ of Sohl-Dickstein et al. and then picks the Gaussian one. forward_process implements $x_t = \sqrt{1-\beta_t}\,x_{t-1} + \sqrt{\beta_t}\,\varepsilon$, with $\beta$ as a variance and $\sqrt\beta$ as the noise amplitude, and runs it for 100 steps with constant $\beta = 0.0005$. The returned list starts with the clean data at index 0.
• Cell 12: make_beta_schedule with the linear, quadratic and sigmoid schedules written above (defaults $T = 1000$, start $10^{-5}$, end $10^{-2}$; the notebook calls it with $T = 100$). With these endpoints and $T = 100$, $\bar\alpha_{100}$ is $0.605$ for the linear and the sigmoid schedule and $0.707$ for the quadratic one. The cosine schedule is not implemented.
• Cells 13–16: the closed-form sampler q_sample with the sigmoid schedule, using alphas_bar_sqrt and one_minus_alphas_bar_sqrt. The helper extract(a, t, x) picks the coefficient a[t] for each example’s timestep and reshapes it so that it broadcasts over the data dimensions. Coefficient index $i$ in the code corresponds to $t = i + 1$ in the formulas, so the plot label $x_0$ already refers to the first noised marginal. These are direct draws from $q(x_t \mid x_0)$ for each $t$, not one correlated trajectory.

The reverse samplers (cells 24 and 47) start from $\mathcal N(0, I)$, but with the sigmoid schedule $q(x_{100} \mid x_0) = \mathcal N(0.78\,x_0,\; 0.40\,I)$, so the starting distribution does not match the end of the forward chain. The reverse process, the training objective and the sampler are in cells 17–48; they are described in the next note.

Next#

We now have a diffusion process that turns any image into noise that is indistinguishable from $\mathcal N(0, I)$ in practice; for DDPM’s schedule, $\bar\alpha_T \approx 4\times10^{-5}$. To generate, we have to invert it, and that is done with a deep neural network. How to set up and train that network is the topic of DDPM: reverse process and training objective.

References#

• J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan, S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. ICML, 2015.
• J. Ho, A. Jain, P. Abbeel. Denoising diffusion probabilistic models. NeurIPS, 2020.
• A. Q. Nichol, P. Dhariwal. Improved denoising diffusion probabilistic models. ICML, 2021.
• P. Dhariwal, A. Nichol. Diffusion models beat GANs on image synthesis. NeurIPS, 2021.
• Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, B. Poole. Score-based generative modeling through stochastic differential equations. ICLR, 2021.
• C. Luo. Understanding diffusion models: A unified perspective. arXiv:2208.11970, 2022.
• L. Weng. What are diffusion models? Blog post, 2021 (updated 2024). Section “Parameterization of β_t”.
• HSE-LAMBDA, Deep Generative Models course repository: 02_DPM_Models.ipynb.