DDPM: reverse process and training objective

The forward process gave us a diffusion process: a fixed chain that turns an image into Gaussian noise. To generate images we have to invert it, and that is the job of a deep neural network. This note derives what the network is trained to do. The route is the same as for the VAE: the likelihood is intractable, so we minimise an upper bound on the negative log-likelihood instead. The bound then simplifies, step by step, into a plain mean squared error on noise. A first reading can skip the callouts (the algebra of the KL split, the 2015 form of the bound, the discretized decoder); the recap at the end collects the chain in six lines.

Notation. The forward step adds noise with variance $\beta_t \in (0, 1)$ at step $t = 1, \dots, T$. We write

$$\alpha_t = 1 - \beta_t, \qquad \bar\alpha_t = \prod_{s=1}^{t} \alpha_s, \qquad \bar\alpha_0 = 1,$$

so the forward step and its closed-form shortcut from the previous note are

$$\begin{gathered} q(x_t \mid x_{t-1}) = \mathcal N\!\big(\sqrt{\alpha_t}\,x_{t-1},\; \beta_t I\big), \\ q(x_t \mid x_0) = \mathcal N\!\big(\sqrt{\bar\alpha_t}\,x_0,\; (1 - \bar\alpha_t) I\big). \end{gathered}$$

The second argument of $\mathcal N(\mu, \sigma^2)$ is always the variance. $q$ denotes the fixed forward process and $q(x_0)$ the data distribution where it starts, $p_\theta$ the learned reverse process, $\theta$ the parameters of the neural network, and $p(x_T) = \mathcal N(0, I)$ the noise distribution where generation starts.

The reverse process#

The image $x_0$ sits at one end of the chain and pure noise $x_T$ at the other. Going forward, the step $q(x_t \mid x_{t-1})$ is known. Going backward we need $p_\theta(x_{t-1} \mid x_t)$: the probability density that we have to learn in order to invert the process.

Figure

The forward step is known; the backward step must be learned

What should $\theta$ optimise? The usual answer for a generative model: make real images likely. We minimise the negative log-likelihood, averaged over the data,

$$\mathbb{E}_{x_0 \sim q(x_0)}\big[-\log p_\theta(x_0)\big] \;\to\; \min_\theta ,$$

which is the same as maximising the likelihood, the density the model assigns to real images. The derivation below works with one image $x_0$; the average over images is added back at the end.

Complete forward and reverse processes#

The complete forward process is the joint distribution of all noisy versions of an image. By the chain rule of probability,

$$\begin{aligned} & q(x_1, \dots, x_T \mid x_0) \\ &\quad = q(x_1 \mid x_0)\, q(x_2 \mid x_1, x_0) \\ &\qquad \cdots q(x_T \mid x_{T-1}, \dots, x_0). \end{aligned}$$

The forward process is a Markov chain: each next state depends only on the previous one, so every factor keeps only its last condition:

$$\begin{aligned} & q(x_1, \dots, x_T \mid x_0) \\ &\quad = q(x_1 \mid x_0)\, q(x_2 \mid x_1) \cdots q(x_T \mid x_{T-1}). \end{aligned}$$

With the shorthand $x_{s:T} = (x_s, \dots, x_T)$, so that $x_{1:T}$ are the noisy states and $x_{0:T}$ the whole chain, the complete forward and reverse processes are

$$\begin{gathered} \underbrace{q(x_{1:T} \mid x_0) = \prod_{t=1}^{T} q(x_t \mid x_{t-1})}_{\text{complete forward process}} , \\[4pt] \underbrace{p_\theta(x_{0:T}) = p(x_T) \prod_{t=1}^{T} p_\theta(x_{t-1} \mid x_t)}_{\text{complete reverse process}} . \end{gathered}$$

The reverse process starts from $p(x_T) = \mathcal N(0, I)$. This factor has no parameters and needs no knowledge about the data: it is the fixed standard normal, the same for every dataset. All learning sits in the reverse steps $p_\theta(x_{t-1} \mid x_t)$.

The exact log-likelihood is intractable#

To get the exact log-likelihood of an image, take the joint reverse distribution and integrate out all intermediate states. This is marginalisation, extracting a marginal from the joint distribution:

$$-\log p_\theta(x_0) = -\log \int p_\theta(x_{0:T})\, dx_{1:T} .$$

Can we use it? In DDPM each reverse step is a Gaussian whose mean is computed from $x_t$ by a neural network (made precise below). The network is nonlinear in $x_t$, so, unlike the forward chain of linear Gaussian steps, the composition of reverse steps is not Gaussian and the integral has no closed form. This is the same obstacle as for the VAE decoder. What remains is numerical integration, so look at what the integral means. Every value of $x_{1:T}$ is one path from a noise sample $x_T$ to the image $x_0$, and the integral adds up the probabilities of all such paths. For images, each $x_t$ has as many dimensions as there are pixels, and there are $T$ of them (hundreds or thousands of steps), so the integral runs over $T \times (\text{number of pixels})$ dimensions. We cannot really compute all of those paths.

Figure

The likelihood integral sums over every path from noise to the image

An upper bound via Jensen’s inequality#

The trick is the one from the VAE. Multiply and divide by the complete forward process $q(x_{1:T} \mid x_0)$, which we know and can sample from (it is a product of Gaussians, positive everywhere, so the division is safe):

$$\begin{aligned} & -\log p_\theta(x_0) \\ &\quad = -\log \int q(x_{1:T} \mid x_0)\, \frac{p_\theta(x_{0:T})}{q(x_{1:T} \mid x_0)}\, dx_{1:T} \\ &\quad = -\log \mathbb{E}_{q(x_{1:T} \mid x_0)}\!\left[ \frac{p_\theta(x_{0:T})}{q(x_{1:T} \mid x_0)} \right]. \end{aligned}$$

The logarithm is concave, so Jensen’s inequality, $\log \mathbb{E}[Y] \ge \mathbb{E}[\log Y]$, moves it inside the expectation and turns the equality into a bound:

$$\begin{aligned} -\log p_\theta(x_0) &\le -\mathbb{E}_{q(x_{1:T} \mid x_0)}\!\left[ \log \frac{p_\theta(x_{0:T})}{q(x_{1:T} \mid x_0)} \right]. \end{aligned}$$

The right-hand side is an upper bound on the negative log-likelihood. It is the negative ELBO: equivalently, the ELBO is a lower bound on the log-likelihood. The bound only involves quantities we can sample and evaluate. Minimising it caps the negative log-likelihood from above, but the two are not the same. The gap is $D_{\mathrm{KL}}\big(q(x_{1:T} \mid x_0)\,\|\,p_\theta(x_{1:T} \mid x_0)\big)$, exactly as in the VAE (how loose is the bound), and it also depends on $\theta$. Write $B$ for the bound and $G$ for the gap. Then $-\log p_\theta(x_0) = B - G$, so a training step changes the negative log-likelihood by $\Delta B - \Delta G$. A falling bound alone therefore guarantees nothing: if $B$ drops from 10 to 9 while $G$ shrinks from 3 to 1, the negative log-likelihood rises from 7 to 8. Minimising the bound improves the likelihood only when the bound falls by more than the gap shrinks; if the gap stays the same or grows, the likelihood improves at least as much as the bound.

Splitting the bound into KL terms#

The algebra is in Appendix A of the DDPM paper (Ho, Jain and Abbeel, 2020, Eq. 5) and is sketched in the callout below. The bound becomes a sum of three kinds of terms:

$$\begin{aligned} & \mathbb{E}_q\Bigg[\, \underbrace{D_{\mathrm{KL}}\big(q(x_T \mid x_0) \,\|\, p(x_T)\big)}_{\text{does not depend on }\theta} \\ & + \sum_{t > 1} D_{\mathrm{KL}}\big(q(x_{t-1} \mid x_t, x_0) \,\|\, p_\theta(x_{t-1} \mid x_t)\big) \\ & \underbrace{-\, \log p_\theta(x_0 \mid x_1)}_{\text{reconstruction}} \,\Bigg]. \end{aligned}$$

Here $\mathbb{E}_q$ averages over the noisy states $x_{1:T} \sim q(x_{1:T} \mid x_0)$ of one image. Each term needs only some of them: the $t$-th KL depends only on $x_t$, so its average is over $x_t \sim q(x_t \mid x_0)$, and the reconstruction term depends only on $x_1$. In training we also average over images $x_0 \sim q(x_0)$.

Read the three terms one by one.

• First term. $q(x_T \mid x_0)$ is the fixed forward process and $p(x_T)$ is the fixed standard normal. Nothing here depends on $\theta$, so for training it is a constant. (It is also tiny for a well-chosen schedule: DDPM’s linear schedule reaches $\bar\alpha_{1000} \approx 4 \times 10^{-5}$, so $q(x_T \mid x_0) \approx \mathcal N(0, I)$. A toy schedule with $T = 100$ steps and $\beta_t$ rising from $10^{-5}$ to $10^{-2}$ ends near $\bar\alpha_{100} \approx 0.6$ and does not get there.)
• Last term. $-\log p_\theta(x_0 \mid x_1)$ measures how likely the model is to recreate the image from the slightly noisier $x_1$. At $x_1$ barely any noise is left, so for the derivation we set this term aside. This is a simplification: DDPM keeps the term as a discretized decoder for 8-bit pixel values (Ho, Jain and Abbeel, 2020, §3.3, Eq. 13; callout below), and in its simplified loss the $t = 1$ noise-regression term approximates it (§3.4).
• Middle terms. These do the work. Each compares the “true posterior” $q(x_{t-1} \mid x_t, x_0)$ with the model’s reverse diffusion step $p_\theta(x_{t-1} \mid x_t)$.

Simplified, the objective is

$$\begin{aligned} \mathbb{E}_q\Bigg[ \sum_{t > 1} D_{\mathrm{KL}}\Big( & \underbrace{q(x_{t-1} \mid x_t, x_0)}_{\text{true posterior}} \\ & \Big\|\, \underbrace{p_\theta(x_{t-1} \mid x_t)}_{\text{reverse step}} \Big) \Bigg]. \end{aligned}$$

The true posterior#

The three states and the three conditional distributions fit into one small graph. $x_{t-1}$ is noisy, $x_t$ is more noisy, and $x_0$ is the original image.

Figure

Forward step, model step and the true posterior

The forward step $q(x_t \mid x_{t-1})$ goes right. The model’s step $p_\theta(x_{t-1} \mid x_t)$ goes left, using only $x_t$. The true posterior $q(x_{t-1} \mid x_t, x_0)$ also goes left, but it looks at both $x_t$ and the original image $x_0$.

The true posterior has a closed-form expression (derived below). So why do we need a neural network at all? Because the true posterior requires knowing $x_0$. At generation time we have no image; we have to produce one starting from pure noise alone. The network $p_\theta(x_{t-1} \mid x_t)$ must therefore imitate the true posterior without seeing $x_0$.

So we train the model’s reverse step to match the true posterior:

$$\begin{gathered} p_\theta(x_{t-1} \mid x_t) = \mathcal N\!\big(\mu_\theta(x_t, t),\; \sigma_\theta^2(x_t, t)\, I\big) \\ \big\updownarrow \\ q(x_{t-1} \mid x_t, x_0) = \mathcal N\!\big(\tilde\mu_t(x_t, x_0),\; \tilde\beta_t I\big). \end{gathered}$$

Why Gaussians?

1. Small steps keep the reverse step Gaussian. For a diffusion with small steps, the reverse of a Gaussian Markov chain is also (approximately) Gaussian: as $\beta_t \to 0$, the reverse step $q(x_{t-1} \mid x_t)$ has the same Gaussian functional form as the forward step (Feller, 1949; used by Sohl-Dickstein et al., 2015, §2.2). For finite steps $q(x_{t-1} \mid x_t)$ is not exactly Gaussian, because it mixes over all images $x_0$ in the data. The posterior conditioned on $x_0$, $q(x_{t-1} \mid x_t, x_0)$, is exactly Gaussian for any $\beta_t$ when $t \ge 2$. At $t = 1$ it collapses to the point mass at $x_0$ ($\tilde\beta_1 = 0$), which is why that step is handled by the reconstruction term.
2. Everything is just simpler. KL divergences between Gaussians have closed forms, and sampling from a Gaussian is trivial.

The closed form. Let $t \ge 2$. By Bayes’ theorem, $q(x_{t-1} \mid x_t, x_0) \propto q(x_t \mid x_{t-1})\, q(x_{t-1} \mid x_0)$. Both factors are Gaussian in $x_{t-1}$. The first has the exponent $-(x_t - \sqrt{\alpha_t}\,x_{t-1})^2/(2\beta_t) = -\alpha_t\,(x_{t-1} - x_t/\sqrt{\alpha_t})^2/(2\beta_t)$ (per coordinate), so as a function of $x_{t-1}$ it has mean $x_t/\sqrt{\alpha_t}$ and precision $\alpha_t/\beta_t$. The second is $\mathcal N\big(\sqrt{\bar\alpha_{t-1}}\,x_0,\; 1 - \bar\alpha_{t-1}\big)$. Completing the square in the product, the precisions add and the mean is the precision-weighted average of the two means:

$$\begin{aligned} \frac{1}{\tilde\beta_t} &= \frac{\alpha_t}{\beta_t} + \frac{1}{1 - \bar\alpha_{t-1}} = \frac{\alpha_t (1 - \bar\alpha_{t-1}) + \beta_t}{\beta_t (1 - \bar\alpha_{t-1})} \\ &= \frac{1 - \bar\alpha_t}{\beta_t (1 - \bar\alpha_{t-1})}, \\ \tilde\mu_t &= \tilde\beta_t \left( \frac{\sqrt{\alpha_t}}{\beta_t}\, x_t + \frac{\sqrt{\bar\alpha_{t-1}}}{1 - \bar\alpha_{t-1}}\, x_0 \right). \end{aligned}$$

The numerator simplifies by $\alpha_t(1 - \bar\alpha_{t-1}) + \beta_t = \alpha_t - \bar\alpha_t + \beta_t = 1 - \bar\alpha_t$, using $\bar\alpha_t = \alpha_t \bar\alpha_{t-1}$ and $\alpha_t + \beta_t = 1$. Inverting the precision and multiplying out the mean gives

$$\begin{aligned} \tilde\mu_t(x_t, x_0) &= \frac{\sqrt{\bar\alpha_{t-1}}\,\beta_t}{1 - \bar\alpha_t}\, x_0 \\ &\quad + \frac{\sqrt{\alpha_t}\,(1 - \bar\alpha_{t-1})}{1 - \bar\alpha_t}\, x_t, \\ \tilde\beta_t &= \frac{1 - \bar\alpha_{t-1}}{1 - \bar\alpha_t}\, \beta_t . \end{aligned}$$

The true posterior mean is a weighted mix of the image and the current noisy state.

Matching two Gaussians: KL becomes L2#

We would still have to predict both $\mu_\theta$ and $\sigma_\theta$. DDPM simplifies further: fix $\sigma_\theta$ to a schedule $\sigma_t$, so it is not learned, and only learn the mean. Ho et al. use $\sigma_t^2 = \beta_t$ or $\sigma_t^2 = \tilde\beta_t$ and report similar results (Ho, Jain and Abbeel, 2020, §3.2); Nichol and Dhariwal (2021), §3.1, later learn a variance that interpolates between the two.

With fixed variances, the KL divergence between the two Gaussians depends on $\theta$ only through the distance between the means. For $q = \mathcal N(\tilde\mu_t, \tilde\beta_t I)$ and $p_\theta = \mathcal N(\mu_\theta, \sigma_t^2 I)$ in $d$ dimensions, the KL divergence between two 1-D Gaussians, summed over the $d$ coordinates, gives

$$\begin{aligned} D_{\mathrm{KL}}(q \,\|\, p_\theta) &= \frac{1}{2\sigma_t^2}\,\big\|\tilde\mu_t - \mu_\theta\big\|^2 \\ &\quad + \underbrace{\frac d2\left(\frac{\tilde\beta_t}{\sigma_t^2} - 1 - \log\frac{\tilde\beta_t}{\sigma_t^2}\right)}_{\text{const; }0\text{ if }\sigma_t^2 = \tilde\beta_t} . \end{aligned}$$

The same result follows from the general formula for two Gaussians in $d$ dimensions with positive definite covariances,

$$\begin{aligned} & D_{\mathrm{KL}}\big(\mathcal N(\mu_q, \Sigma_q) \,\|\, \mathcal N(\mu_p, \Sigma_p)\big) \\ &\quad = \tfrac12\Big[\operatorname{tr}(\Sigma_p^{-1}\Sigma_q) - d + \ln\frac{\det\Sigma_p}{\det\Sigma_q} \\ &\qquad\quad + (\mu_p - \mu_q)^\top \Sigma_p^{-1} (\mu_p - \mu_q)\Big], \end{aligned}$$

with $\Sigma_q = \tilde\beta_t I$ and $\Sigma_p = \sigma_t^2 I$: the trace gives $d\,\tilde\beta_t/\sigma_t^2$, the log-determinant ratio $d \ln(\sigma_t^2/\tilde\beta_t)$, and the quadratic form $\lVert\tilde\mu_t - \mu_\theta\rVert^2/\sigma_t^2$.

So in fact the KL term is an L2 loss, $\frac{1}{2\sigma_t^2}\|\tilde\mu_t - \mu_\theta\|^2$, plus a constant. For one pair $(x_t, x_0)$ the L2 part would be zero with $\mu_\theta$ on $\tilde\mu_t$, and the whole KL term would vanish only if, in addition, $\sigma_t^2 = \tilde\beta_t$. But $\tilde\mu_t$ depends on $x_0$, which the network does not see, and many images are compatible with the same $x_t$. Since $\mu_\theta$ sees only $x_t$, the best it can do is the average of $\tilde\mu_t$ over all images $x_0$ compatible with $x_t$, because the conditional mean minimises the expected squared error:

$$\mu_\theta^*(x_t, t) = \mathbb{E}_q\big[\tilde\mu_t(x_t, x_0) \mid x_t\big],$$

the average over $x_0 \sim q(x_0 \mid x_t)$. By the tower property this is also $\mathbb{E}_q[x_{t-1} \mid x_t]$, the mean of the true reverse step $q(x_{t-1} \mid x_t)$. At the optimum the KL term is therefore not zero: what remains reflects how much $x_t$ leaves the image undetermined.

Figure

With equal variances, the KL divergence is an L2 loss on the means

So:

$$\mathbb{E}_q\!\left[ \sum_{t > 1} \frac{1}{2\sigma_t^2}\, \big\| \tilde\mu_t - \mu_\theta(x_t, t) \big\|^2 \right] \;\to\; \min_\theta .$$

From means to noise#

The true posterior mean still contains $x_0$. Remove it with the reparameterisation of the forward shortcut, as in the VAE:

$$\begin{gathered} x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\, \varepsilon, \quad \varepsilon \sim \mathcal N(0, I) \\ \Longrightarrow\quad x_0 = \frac{1}{\sqrt{\bar\alpha_t}}\big(x_t - \sqrt{1 - \bar\alpha_t}\, \varepsilon\big). \end{gathered}$$

Substitute into $\tilde\mu_t$. The key identity is $\bar\alpha_t = \alpha_t \bar\alpha_{t-1}$, so $\sqrt{\bar\alpha_{t-1}}/\sqrt{\bar\alpha_t} = 1/\sqrt{\alpha_t}$. The coefficient of $x_t$ collects to

$$\begin{aligned} &\frac{\beta_t}{\sqrt{\alpha_t}\,(1 - \bar\alpha_t)} + \frac{\sqrt{\alpha_t}\,(1 - \bar\alpha_{t-1})}{1 - \bar\alpha_t} \\ &\quad= \frac{\beta_t + \alpha_t - \bar\alpha_t}{\sqrt{\alpha_t}\,(1 - \bar\alpha_t)} = \frac{1 - \bar\alpha_t}{\sqrt{\alpha_t}\,(1 - \bar\alpha_t)} = \frac{1}{\sqrt{\alpha_t}}, \end{aligned}$$

using $\beta_t + \alpha_t = 1$. The coefficient of $\varepsilon$ is $-\frac{\beta_t}{\sqrt{\alpha_t}\,(1-\bar\alpha_t)}\sqrt{1 - \bar\alpha_t} = -\frac{\beta_t}{\sqrt{\alpha_t}\sqrt{1 - \bar\alpha_t}}$. Hence

$$\tilde\mu_t = \frac{1}{\sqrt{\alpha_t}}\left( x_t - \frac{\beta_t}{\sqrt{1 - \bar\alpha_t}}\, \varepsilon \right).$$

The denoiser has access to $x_t$ and $t$, but not to $\varepsilon$. So give the model’s mean exactly the same form and let the network predict only the missing piece, the noise (Ho, Jain and Abbeel, 2020, Eq. 11):

$$\mu_\theta(x_t, t) = \frac{1}{\sqrt{\alpha_t}}\left( x_t - \frac{\beta_t}{\sqrt{1 - \bar\alpha_t}}\, \varepsilon_\theta(x_t, t) \right).$$

Now both $\tilde\mu_t$ and $\mu_\theta(x_t, t)$ are expressed using only $x_t$ and a noise. In the difference $x_t$ cancels, leaving $\tilde\mu_t - \mu_\theta = \frac{\beta_t}{\sqrt{\alpha_t}\sqrt{1 - \bar\alpha_t}}\big(\varepsilon_\theta - \varepsilon\big)$, and the loss becomes

$$\begin{aligned} L = \mathbb{E}_q\Bigg[ & \sum_{t > 1} \frac{\beta_t^2}{2\sigma_t^2\, \alpha_t\, (1 - \bar\alpha_t)} \\ & \cdot \big\| \varepsilon - \varepsilon_\theta(x_t, t) \big\|^2 \Bigg]. \end{aligned}$$

This is Eq. 12 of Ho, Jain and Abbeel (2020), summed over $t$. Here $\varepsilon_\theta(x_t, t)$ is the network’s prediction of the noise. As for the means, it cannot recover the particular $\varepsilon$ that produced $x_t$, because it does not see $x_0$. The optimum is $\varepsilon_\theta^*(x_t, t) = \mathbb{E}[\varepsilon \mid x_t]$, the average noise consistent with $x_t$; this is the same optimum as $\mu_\theta^*$, since $\tilde\mu_t$ is an affine function of $\varepsilon$ for fixed $x_t$. The denoiser is a noise predictor.

Final simplification: a random timestep per sample#

Summing over all $T - 1$ terms for every training image is too expensive: it costs $T - 1$ network evaluations per image. Instead, pick a random timestep for each sample. Since $\sum_{t=2}^{T} f(t) = (T - 1)\, \mathbb{E}_{t \sim \mathrm{Uniform}\{2, \dots, T\}}\big[f(t)\big]$, one uniformly drawn $t$ gives an unbiased estimate of the sum divided by $T - 1$, and a constant factor does not change the minimiser. Averaging also over images $x_0 \sim q(x_0)$ and noises $\varepsilon \sim \mathcal N(0, I)$, with $t \sim \mathrm{Uniform}\{2, \dots, T\}$, the final loss is

$$\begin{gathered} \mathbb{E}_{x_0,\, t,\, \varepsilon}\!\left[ \frac{\beta_t^2}{2\sigma_t^2\, \alpha_t\, (1 - \bar\alpha_t)}\, \big\| \varepsilon - \varepsilon_\theta(x_t, t) \big\|^2 \right], \\ x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\, \varepsilon . \end{gathered}$$

Pairing timesteps. The estimate stays unbiased however the timesteps of a batch depend on each other, as long as each one on its own is uniform. A variance-reduction trick used in the public DDIM code (Song, Meng and Ermon, 2021) draws $t$ for half of the batch and gives the other half the mirrored steps, $t' = T + 1 - t$ when $t$ runs over $1, \dots, T$, which are uniform too. Let $f(t)$ be the expected loss at step $t$. A mirrored pair estimates the mean of $f$ with variance $\tfrac12\big[\operatorname{Var} f(t) + \operatorname{Cov}\big(f(t), f(t')\big)\big]$, two independent draws with $\tfrac12 \operatorname{Var} f(t)$. If $f$ is monotone in $t$, then $f(t)$ and $f(T + 1 - t)$ move in opposite directions, the covariance is negative by Chebyshev’s sum inequality, and the pair wins. Toy example: $T = 4$ and $f(t) = 1/t$, a loss that is largest at the least noisy step. Two independent draws give variance $0.0423$, a mirrored pair $0.0109$, about $3.9$ times less. The per-step loss of a real network is only roughly monotone in $t$, so the gain in practice is an empirical matter.

The training step. One training step needs no chain at all: draw an image, a timestep and a noise, jump straight to $x_t$ with the shortcut, and regress the noise.

Figure

One training step: draw, jump, predict the noise

Dropping the weight is a deliberate reweighting, not a cosmetic change. With the linear schedule of the paper ($T = 1000$, $\beta_t$ from $10^{-4}$ to $0.02$) and $\sigma_t^2 = \beta_t$, the weight simplifies to $\beta_t / (2\alpha_t(1 - \bar\alpha_t))$. It is $0.273$ at $t = 2$, falls to a minimum of $0.0050$ at $t = 349$, and climbs back to $0.0102$ at $t = T$. The true bound puts about $55$ times more weight at $t = 2$ than at the minimum. $L_{\text{simple}}$ weights every step equally, which shifts effort towards the harder, noisier steps. Ho et al. found that this gives better samples at the price of a slightly worse likelihood (§3.4 and Table 1: FID 3.17 against 13.51 for the true bound, but $3.75$ against $3.70$ bits per dimension).

Figure

The bound's weight favours the nearly clean steps

Sampling: running the chain backwards#

Training never runs the chain, but generation does. Start from pure noise $x_T \sim \mathcal N(0, I)$ and apply the learned reverse step $T$ times. Each step samples from $p_\theta(x_{t-1} \mid x_t) = \mathcal N(\mu_\theta(x_t, t), \sigma_t^2 I)$, using the noise form of the mean derived above, with fresh noise $z \sim \mathcal N(0, I)$ for $t > 1$ and $z = 0$ for $t = 1$ (Ho, Jain and Abbeel, 2020, Algorithm 2):

$$\begin{aligned} x_{t-1} = {} & \frac{1}{\sqrt{\alpha_t}}\left( x_t - \frac{\beta_t}{\sqrt{1 - \bar\alpha_t}}\, \varepsilon_\theta(x_t, t) \right) \\ & + \sigma_t z . \end{aligned}$$

Each step removes the network’s estimate of the noise, rescales, and adds a little fresh noise; the last step returns the mean without extra noise. Generation therefore costs $T$ network evaluations, one per step, which is the main practical drawback of DDPM. Later samplers such as DDIM (Song, Meng and Ermon, 2021) reuse the same trained $\varepsilon_\theta$ with far fewer steps.

The denoising network#

The derivation treats $\varepsilon_\theta(x_t, t)$ as a function of two arguments. In practice it is one network for all steps, a U-Net, that receives the timestep as an embedding added inside every block.

Figure

The DDPM denoiser: a U-Net that receives the step t

1. The noisy image $x_t$ enters the encoder. Each level halves the resolution; DDPM adds self-attention at $16 \times 16$.
2. The timestep $t$ becomes a vector of sines and cosines $e(t)$.
3. A small MLP turns $e(t)$ into $m(t)$, and every residual block adds its own linear projection $A_k\,m(t)$ to its channels.
4. The decoder doubles the resolution level by level; skip connections bring in the encoder’s features of the same size.
5. The output $\varepsilon_\theta(x_t, t)$ has the shape of $x_t$ and is compared with the true noise $\varepsilon$.

One network for all timesteps#

A separate network per step would mean $T = 1000$ networks, each trained on a thousandth of the updates. Neighbouring steps solve almost the same problem: $x_{t-1}$ and $x_t$ differ by one small step: a rescaling by $\sqrt{1 - \beta_t} \ge 0.99$ and noise of variance $\beta_t \le 0.02$. DDPM therefore shares the parameters across time and tells the network which step it is at (Ho, Jain and Abbeel, 2020, §4). The step is needed because the same input calls for different outputs at different noise levels.

Example. For data $\mathcal N(0, 0.5^2)$ in one dimension, $\varepsilon$ and $x_t$ are jointly Gaussian with covariance $\sqrt{1 - \bar\alpha_t}$, so the optimal prediction is

$$\mathbb E[\varepsilon \mid x_t] = \frac{\sqrt{1 - \bar\alpha_t}}{0.25\,\bar\alpha_t + 1 - \bar\alpha_t}\, x_t .$$

With the linear schedule and $x_t = 1$ it is $0.17$ at $t = 10$ ($\bar\alpha_t = 0.998$), where $x_t$ is mostly image, and $1.02$ at $t = 500$ ($\bar\alpha_t = 0.079$), where $x_t$ is mostly noise.

A U-Net#

The output $\varepsilon_\theta(x_t, t)$ has the shape of the input, one noise value per pixel and channel, so the network solves a dense prediction problem, like segmentation. To tell noise from image detail at one pixel it needs context from far away and precision at the pixel itself. A U-Net (Ronneberger, Fischer and Brox, 2015) provides both. An encoder halves the resolution level by level, so that the coarse levels see the whole image; a decoder doubles it back; and skip connections pass the encoder’s features at each resolution to the decoder level of the same size, where they are concatenated with the upsampled features, so that fine detail reaches the output. Noise-conditional score networks use a U-Net variant for the same reason (Song and Ermon, 2019, §4.1 and App. A.3).

DDPM’s network (Ho, Jain and Abbeel, 2020, Appendix B):

• the backbone of an unmasked PixelCNN++, a U-Net built from Wide ResNet residual blocks;
• group normalization (Wu and He, 2018) instead of weight normalization: statistics over groups of channels within one image, so that, unlike batch normalization, they do not depend on the other images of the batch;
• four feature-map resolutions for $32 \times 32$ images ($32 \times 32$ down to $4 \times 4$) and six for $256 \times 256$, with two residual blocks per resolution;
• self-attention at the $16 \times 16$ resolution, between the convolutional blocks;
• the timestep added to every residual block through the Transformer sinusoidal embedding;
• 35.7 million parameters for CIFAR-10, 114 million for LSUN and CelebA-HQ at $256 \times 256$, and about 256 million for a larger LSUN Bedroom variant.

Telling the network the timestep#

DDPM encodes $t$ the way a Transformer encodes a position (positional encodings):

$$\begin{gathered} e(t)_{2i} = \sin(\omega_i t), \qquad e(t)_{2i+1} = \cos(\omega_i t), \\ \omega_i = 10000^{-2i/d}, \qquad i = 0, \dots, d/2 - 1 . \end{gathered}$$

A small MLP turns the code into a feature vector, $m(t) = W_2\,\mathrm{SiLU}(W_1\, e(t))$ with $\mathrm{SiLU}(z) = z \cdot \mathrm{sigmoid}(z)$, and every residual block $k$ has its own linear map $A_k$ to its number of channels $C_k$. The result is added to the block’s feature maps as one bias per channel, the same at every pixel:

$$h_k[c, i, j] \;\leftarrow\; h_k[c, i, j] + \big(A_k\, m(t)\big)_c .$$

Why not feed $t$ as a plain number? A scalar enters the first layer as one more input, and the network has to build every dependence on $t$ from it. The sinusoidal code is a vector whose components turn at rates from $1$ down to about $10^{-4}$ radians per step, so linear combinations of it can follow both fast and slow changes in $t$, and nearby steps get nearby codes. The order of the components, interleaved or sines first, does not matter: the next layer is linear and can permute its inputs.

Other ways to give a network the step or another condition:

• Scale and shift. The embedding sets a scale and a shift per channel after normalization, $h \leftarrow y_s(t) \odot \mathrm{GroupNorm}(h) + y_b(t)$: adaptive group normalization (Dhariwal and Nichol, 2021, §3.1). It is the diffusion counterpart of the adaptive instance normalization (AdaIN) with which StyleGAN injects its style vector (Karras, Laine and Aila, 2019, Eq. 1). On ImageNet $128 \times 128$ it improved FID from $15.08$, for addition followed by group normalization as in DDPM, to $13.06$ (their Table 3). Noise-conditional score networks use the same idea with one learned scale and shift per noise level (score matching note).
• An extra input channel. Append $t/T$ to the image as a constant channel. This is the simplest option, but the step then enters only the first convolution and has to be carried through every later layer.
• A learned vector per step. A table with one trainable vector per step, added to or multiplied with a hidden layer, works for small networks and short chains. The $T$ vectors share nothing, though: each is trained only by the examples drawn at its own step.
• Other conditions. A class label or a pooled text embedding can be projected to the size of the time embedding and added to it, as Imagen does (Saharia et al., 2022, §2.5); a whole sequence of text vectors enters through cross-attention (latent diffusion).

Why the pixels are scaled to [−1, 1]#

DDPM maps the pixel values $0, \dots, 255$ linearly to $[-1, 1]$ so that the reverse process works on inputs of a consistent scale, starting from the standard normal prior (Ho, Jain and Abbeel, 2020, §3.3). Three reasons make this matter:

1. The chain must end at the prior. The forward process removes the image only by shrinking it, $x_T = \sqrt{\bar\alpha_T}\,x_0 + \sqrt{1 - \bar\alpha_T}\,\varepsilon$, and with DDPM’s schedule $\sqrt{\bar\alpha_T} = 0.0064$. For values in $[-1, 1]$ the leftover image is at most $0.0064$, negligible next to noise of standard deviation $1$. For raw values in $[0, 255]$ it would reach $1.6$: $x_T$ would still show the image, and sampling from $\mathcal N(0, I)$ would start from the wrong distribution.
2. Every step sees inputs of the same size. By the variance recursion $\operatorname{Var}(x_t) = \bar\alpha_t \operatorname{Var}(x_0) + 1 - \bar\alpha_t$, which lies between $\operatorname{Var}(x_0)$ and $1$ when $\operatorname{Var}(x_0) \le 1$, as values in $[-1, 1]$ guarantee. One network can then serve all steps; for raw pixel values the variance would fall from thousands to $1$ along the chain.
3. The data are centred like the prior, and the discretized decoder has bins of width $2/255$ with its open edge bins at $\pm 1$.

To display an image, $x$ is mapped back by $255\,(x + 1)/2$ after clipping to $[-1, 1]$. Noisy states leave the interval routinely: a pixel with $x_0 = 0.9$ at a step with $\bar\alpha_t = 0.5$ has $x_t \sim \mathcal N(0.64,\ 0.5)$, which falls outside $[-1, 1]$ with probability $0.31$. Without the clipping, the conversion to 8-bit integers turns such values into wrong colours.

Training in practice: averaged weights and clipped gradients#

Two stabilisers are common in diffusion training loops.

An exponential moving average of the weights. Keep a second copy $\bar\theta$ of the parameters and update it after every optimizer step:

$$\bar\theta \;\leftarrow\; \mu\, \bar\theta + (1 - \mu)\, \theta .$$

Unrolled over $n$ steps, $\bar\theta$ is a weighted average of the past iterates with weights $(1 - \mu)\,\mu^{n-k}$, plus $\mu^n$ times its starting value: roughly an average over the last $1/(1 - \mu)$ steps. Training continues with $\theta$; samples are drawn with $\bar\theta$. The reason is the noise of the gradient: every example brings its own image, timestep and noise, so the iterates $\theta$ jitter around a good region, and their average lies closer to its centre. DDPM uses $\mu = 0.9999$ (Appendix B), an average over about $10{,}000$ steps. Nichol and Dhariwal (2021) swept the rate on class-conditional ImageNet $64 \times 64$ and found $0.9999$ and $0.99995$ best (their Figure 17). Worked example: the starting value keeps the weight $\mu^n$, which is $0.9999^{10\,000} = 0.37$ after ten thousand steps and $0.9999^{50\,000} = 0.0067$ after fifty thousand. The averaged weights are therefore worth using only after training has run several times longer than the window. With $\mu = 0.9$ the window is about $10$ steps, too short to average out much.

Gradient clipping. Before each step, rescale the gradient $g$ when its norm exceeds a threshold $c$, $g \leftarrow g \cdot \min(1, c/\lVert g\rVert)$ (training background). The direction is kept and only rare large gradients are shortened: with $c = 1$, a gradient of norm $4$ is divided by $4$, and one of norm $0.5$ is left as it is. The loss of a batch varies with the timesteps it happens to contain, and clipping keeps an unusual batch from moving the weights far.

DDPM’s settings (Appendix B): Adam with learning rate $2 \times 10^{-4}$ ($2 \times 10^{-5}$ at $256 \times 256$), batch size 128 for CIFAR-10 and 64 for larger images, dropout $0.1$ on CIFAR-10 and none elsewhere, random horizontal flips, the EMA rate $0.9999$ and $T = 1000$. The CIFAR-10 model trained for 800K steps at 21 steps per second on a TPU v3-8, about 10.6 hours; sampling a batch of 256 images took 17 seconds.

Recap: from the likelihood to one training step#

1. Bound. $-\log p_\theta(x_0)$ is at most the negative ELBO; the gap is a KL divergence, so lowering the bound does not by itself lower the negative log-likelihood.
2. Posterior targets. The bound splits into KL terms that compare each reverse step $p_\theta(x_{t-1} \mid x_t)$ with the Gaussian true posterior $q(x_{t-1} \mid x_t, x_0)$, whose mean $\tilde\mu_t$ is known in closed form.
3. Weighted noise loss. With fixed variances each KL term is an L2 distance between means plus a constant, and with $\mu_\theta$ written through a noise predictor it becomes $\frac{\beta_t^2}{2\sigma_t^2\, \alpha_t\, (1 - \bar\alpha_t)}\, \lVert \varepsilon - \varepsilon_\theta(x_t, t) \rVert^2$.
4. Reweighting. $L_{\text{simple}}$ drops that weight. It is a different objective, not the bound: it shifts effort towards the noisier steps and gives better samples at a slightly worse likelihood.
5. One training step. Draw $x_0$, $t$ and $\varepsilon$, form $x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\, \varepsilon$, and take a gradient step on $\lVert \varepsilon - \varepsilon_\theta(x_t, t) \rVert^2$.
6. Sampling. From $x_T \sim \mathcal N(0, I)$ apply $x_{t-1} = \mu_\theta(x_t, t) + \sigma_t z$ for $t = T, \dots, 1$, with no noise at the last step.

Notebook#

The diffusion notebook of the HSE-LAMBDA Deep Generative Models repository (pinned commit; cell numbers count every cell, markdown included, from 1; no saved outputs) trains two versions of the reverse process on a 2-D Swiss roll of 10,000 points with $T = 100$ and a sigmoid schedule between $10^{-5}$ and $10^{-2}$ ($\beta_1 \approx 3.5 \times 10^{-5}$, $\beta_{100} \approx 1.0 \times 10^{-2}$). Code indices are 0-based, so code step t = 0 is $t = 1$ in the formulas.

• Cells 17–18 (restated in cell 37): the true posterior. q_posterior_mean_variance uses the coefficients $\beta_t\sqrt{\bar\alpha_{t-1}}/(1-\bar\alpha_t)$ and $(1-\bar\alpha_{t-1})\sqrt{\alpha_t}/(1-\bar\alpha_t)$ and the variance $\tilde\beta_t$, as derived above. At code step t = 0 the variance is $\tilde\beta_1 = 0$, the degenerate case of the closed form, and the “clipped” log-variance still takes $\log 0 = -\infty$. In loss_likelihood_bound (cell 31) this makes the KL infinite whenever a batch contains code step t = 0, so the loss printed in cell 33 can be inf; the gradients stay finite, because the infinite part does not depend on $\theta$. Using $\tilde\beta_2$ for the first entry avoids this.
• Cells 19–24: the original diffusion model of Sohl-Dickstein et al. One network is shared by all steps and predicts a mean and a log-variance. Its ConditionalLinear layers multiply the output of a linear layer by a learned vector per step, $\gamma_t \odot (W h + b)$, with $\gamma$ an nn.Embedding(n_steps, 128) initialised uniformly on $[0, 1]$: the learned vector per step, not a sinusoidal code. p_sample and p_sample_loop run the reverse chain.
• Cells 25–29: the bound. Cell 26 writes the bound as the per-step log-ratios of the first callout. Cell 27 then gives two more forms, neither quite right: the first (a prior term, all $T$ KL terms and one forward entropy) is not the standard bound, and the second puts the entropy terms of the 2015 form with the signs they have in $K$, the lower bound on the log-likelihood, into a loss that should be $-K$, next to an extra prior term. normal_kl (cell 28) is the KL between diagonal Gaussians and entropy the Gaussian entropy $\tfrac12(1 + \ln 2\pi) + \tfrac12\ln\sigma^2$ per dimension. compute_loss (cell 29) gets the signs right: it returns $\big(T \cdot \mathrm{KL} + H_q(X_1 \mid X_0) - H_q(X_T \mid X_0) + H_p(X_T) - H_{\mathcal N(0,1)}\big)/\ln 2$, averaged over the batch and the two coordinates: a one-step estimate of $-K$ minus the entropy of a unit Gaussian, in bits per dimension, i.e. Sohl-Dickstein’s $-(K - L_{\text{null}})$. The factor $T$ scales a single random step up to the sum, which in the bound has $T - 1$ terms ($t = 2, \dots, T$) and does not include code step t = 0.
• Cells 30–33: training at random timesteps. Cell 30 explains the mirrored timesteps with $t = 5 \leftrightarrow 94$ in code indices ($t = 6 \leftrightarrow 95$ in the formulas): loss_likelihood_bound draws 65 steps and appends their partners n_steps - t - 1, keeping 128. The loop of cell 33 uses Adam with learning rate $10^{-3}$, batches of 128 (79 per pass over the 10,000 points), 5001 passes, and clips the gradient norm at $1.0$.
• Cells 34–36: DDPM. ConditionalModelDDPM (three ConditionalLinear layers) predicts the noise, and p_sample_ddpm implements the sampling step above with $\sigma_t = \sqrt{\beta_t}$. It adds fresh noise at every step, including the last one, where Algorithm 2 adds none. The text of cell 34 says that $\tilde\beta_t$ is usually preferred for its better results; Ho et al. found the two choices similar (§3.2).
• Cells 38–41: the variational loss with the discretized decoder at the first step. Cell 38 gives the KL between Gaussians with full covariances. discretized_gaussian_log_likelihood (cell 39) implements Eq. 13, with $\Phi$ approximated by $\tfrac12\big(1 + \tanh\big(\sqrt{2/\pi}\,(x + 0.044715\,x^3)\big)\big)$ (largest error $1.8 \times 10^{-4}$) and the open edge bins applied for $x < -0.999$ and $x > 0.999$. loss_variational (cell 41) averages the KL and the decoder term over dimensions and divides by $\ln 2$, bits per dimension, and picks the decoder term for code step t = 0 with torch.where. The loss is shown for illustration and never trained. As written it calls p_mean_variance with the global model, the mean-and-variance network of cell 33, since the DDPM model returns only a noise estimate. The decoder also assumes 8-bit pixels in $[-1, 1]$, while the Swiss roll is continuous and about 31% of its points have a coordinate outside $[-1, 1]$, where the edge rule assigns them a whole tail.
• Cells 42–43: noise_estimation_loss, the unweighted $L_{\text{simple}}$, (noise - predicted_noise).square().mean(), with the same mirrored timesteps. The text of cell 42 calls $L_{\text{simple}}$ a direct minimisation of the KL terms; as shown above, it reweights them.
• Cells 44–48: the moving average of the weights and the DDPM training loop (Adam $10^{-3}$, batch 128, 1000 passes, gradient norm clipped at $1.0$). The EMA class defaults to $\mu = 0.999$, but the loop uses $\mu = 0.9$, a window of about 10 updates, shorter than one pass over the data. apply_shadow would copy $\bar\theta$ into the model but is never called, so sampling uses the raw weights. The plotting cells (33 and 48) show x_seq[10*j], the reverse chain after $10j$ steps (from $x_{90}$ down to the final sample), but title the panels $q(x_{100j})$: wrong scale, wrong direction, and $q$ instead of $p_\theta$.

The homework notebook 04_hw_diffusion of the same repository (same cell convention) trains a DDPM on CelebA face photos at $32 \times 32$. Most of its code is a template with blanks to fill in, but it keeps saved outputs of a reference run on a Tesla T4 GPU; the numbers below are those outputs.

• Cells 9–11: data. Batches of 128 images normalised to $[-1, 1]$, as above. Cell 10 prints the image shape as $3 \times 64 \times 64$ although image_size = 32, while the grids of cells 11 and 26 show $32 \times 32$ images: the saved outputs come from more than one run.
• Cells 13–14 restate the forward process, the noise-prediction mean, the two choices $\sigma_t^2 = \beta_t$ or $\tilde\beta_t$ and Algorithms 1 and 2, and name a U-Net or Imagen’s Efficient U-Net as the network. The Efficient U-Net was designed for Imagen’s super-resolution stages: it moves parameters to the low-resolution blocks, scales skip connections by $1/\sqrt2$ and reverses the order of the downsampling and upsampling operations to speed up the forward pass (Saharia et al., 2022, App. B.1).
• Cell 16: the DDPM class holds the schedule (n_steps, min_beta, max_beta, left blank), forward draws $x_t$ from $x_0$ in one step, and backward calls the network for $\varepsilon_\theta(x_t, t)$.
• Cells 18–19. show_forward noises a batch to 25, 50, 75 and 100% of $T$. generate_new_images runs Algorithm 2 from $x_T \sim \mathcal N(0, I)$, adds noise only while code step t > 0, so not at the last step, as Algorithm 2 prescribes, and writes the chain to a GIF.
• Cells 20–21: the U-Net. The markdown lists the options for feeding $t$ (AdaIN as in StyleGAN, an extra channel, a sinusoidal encoding) and takes the sinusoidal one, adding that a projected text embedding could be added the same way (not used here). sinusoidal_embedding(n, d) fills a frozen nn.Embedding table with $d = 100$ sines and cosines for each of the $n = 1000$ steps. _make_te is the Linear–SiLU–Linear MLP that maps the code to the channel count of a block; forward adds the output of te1 to the input image, and the blocks left blank can do the same with their own _make_te maps.
• Cells 22–23: the reference model has 650,430 parameters; the count includes the frozen embedding table.
• Cells 26–27: forward grids at 25–100% of $T$, and 16 samples of the untrained network, which are noise.
• Cells 29–31: training. training_loop implements Algorithm 1 with nn.MSELoss, i.e. $L_{\text{simple}}$, and saves the checkpoint with the lowest training loss of an epoch. The saved log has 1272 batches of 128 per epoch, consistent with the 162,770 images of the CelebA training split. The loss fell from $0.037$ after epoch 1 to $0.033$ after epoch 16; 200 epochs were planned and the log stops at the start of epoch 17.
• Cells 33–36 reload the best checkpoint and generate 100 faces into a GIF; there are no saved samples.

Issues:

• show_images (cell 7) maps $x$ to $255\,(x + 1)/2$ and casts to uint8 without clipping. Noised images leave $[-1, 1]$, so the forward grids of cell 26 show saturated, noise-like colours already at 25% of $T$; clip to $[-1, 1]$ before the cast. How much of the face would remain depends on the schedule, which the template leaves blank; with DDPM’s linear schedule $\bar\alpha_{250} = 0.52$, so more than half of the signal variance is left at 25%.
• The dataset class of cell 10 returns bare images, so a batch from the loader is a tensor of images, and batch[0] in show_first_batch, show_forward and training_loop picks the first image instead of the batch. The template comes from a dataset that returns (image, label) pairs; with this dataset, use the batch itself.
• Cell 33 rebuilds the model as DDPM(UNet(), n_steps=n_steps, device=device): min_beta and max_beta are not passed, so the class defaults are used, and UNet() takes its default 1000 steps. A schedule other than the defaults has to be passed again by hand.
• sinusoidal_embedding computes $d$ frequencies $w_k = 10000^{-2k/d}$, $k = 0, \dots, d - 1$, but fills only $d$ values. The Transformer encoding uses the first $d/2$ frequencies, each once in a sine and once in a cosine. Taking the cosines from the second half instead gives frequencies of at most $10^{-4}$, for which $\cos(w_k t) \ge 0.995$ over all $t \le 1000$: those components barely change.

Next#

DDPM reaches its loss through a likelihood bound. The next note reaches the same loss without one: it learns the gradient of the log-density of the noised data and samples by following it, and shows that the noise predictor of DDPM is that gradient up to a known scale.

References#

• J. Ho, A. Jain, P. Abbeel. Denoising diffusion probabilistic models. NeurIPS, 2020.
• J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan, S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. ICML, 2015.
• W. Feller. On the theory of stochastic processes, with particular reference to applications. Proceedings of the First Berkeley Symposium on Mathematical Statistics and Probability, 1949.
• J. Song, C. Meng, S. Ermon. Denoising diffusion implicit models. ICLR, 2021. Code: ermongroup/ddim (mirrored timesteps in runners/diffusion.py).
• A. Q. Nichol, P. Dhariwal. Improved denoising diffusion probabilistic models. ICML, 2021.
• C. Luo. Understanding diffusion models: A unified perspective. arXiv:2208.11970, 2022.
• P. Dhariwal, A. Nichol. Diffusion models beat GANs on image synthesis. NeurIPS, 2021. §3 (the U-Net of DDPM) and §3.1, Table 3 (adaptive group normalization).
• Y. Song, S. Ermon. Generative modeling by estimating gradients of the data distribution. NeurIPS, 2019. §4.1 and Appendix A (architecture of the score network).
• O. Ronneberger, P. Fischer, T. Brox. U-Net: Convolutional networks for biomedical image segmentation. MICCAI, 2015.
• Y. Wu, K. He. Group normalization. ECCV, 2018.
• T. Karras, S. Laine, T. Aila. A style-based generator architecture for generative adversarial networks. CVPR, 2019. §2, Eq. 1 (AdaIN).
• C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. Denton, S. K. S. Ghasemipour, B. K. Ayan, S. S. Mahdavi, R. G. Lopes, T. Salimans, J. Ho, D. J. Fleet, M. Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS, 2022. Imagen: §2.5 (conditioning) and Appendix B.1 (Efficient U-Net).
• HSE-LAMBDA, Deep Generative Models course repository: 02_DPM_Models.ipynb and 04_hw_diffusion.ipynb.