Generative adversarial networks

The VAE trains its generator through a bound on the likelihood and pays for it with blurry samples. A generative adversarial network (GAN) avoids likelihoods altogether (Goodfellow et al., 2014). The generator is a network that turns noise into samples, and it is trained only through a second network, the discriminator, which tries to tell generated samples from real ones. The model is implicit: we can sample from it, but we cannot evaluate its density.

Two facts organize this note; the rest shows where they come from.

• If the discriminator is optimal, the generator’s objective becomes the Jensen–Shannon divergence in disguise: $2\,D_{\mathrm{JS}}(p_{\text{data}}\,\|\,p_g) - 2\ln 2$. Training the generator then means minimizing $D_{\mathrm{JS}}(p_{\text{data}}\,\|\,p_g)$.
• If the supports of the two distributions do not overlap and the discriminator is (near-)optimal, the minimax generator loss gives no useful gradient: JS is stuck at its maximum $\ln 2$ wherever $p_g$ sits, and the optimal discriminator is constant around the generated samples.

Notation. $p_{\text{data}}(x)$ is the data density, $p_z(z)$ the noise prior, $G$ the generator with parameters $\theta_g$, $p_g(x)$ the density of the generated samples $G(z)$, and $D$ the discriminator with parameters $\theta_d$; $D(x) \in [0, 1]$ is the probability that $x$ is real, with $\ln 0 = -\infty$. A network with a sigmoid output takes values in $(0, 1)$ and can only approach the endpoints. Logarithms are natural.

Reading route. A first pass follows the game, the alternating updates, the optimal discriminator and its JS identity, the non-saturating loss computed from logits, mode collapse, and then the Wasserstein GAN. The f-GAN and least-squares extensions are optional depth.

The setup#

Sample noise $z \sim p_z(z)$, for example a standard Gaussian, and pass it through the generator. The output $G(z)$ is a generated sample, and its distribution is $p_g(x)$. The discriminator receives either a generated sample, labelled $y = 0$ (fake), or a data point $x \sim p_{\text{data}}(x)$, labelled $y = 1$ (real), and outputs the probability that its input is real.

Figure

The GAN game as a picture

The generator never sees real data directly. It learns only through the discriminator’s output, and the discriminator learns to separate the two streams.

A minimax game#

The two networks play a minimax game: the discriminator maximizes its success, and the generator minimizes the discriminator’s success. Success is measured by the value function

$$\begin{aligned} V(G, D) ={}& \mathbb{E}_{x\sim p_{\text{data}}}\big[\ln D(x)\big] \\ &+ \mathbb{E}_{z\sim p_z}\big[\ln\big(1 - D(G(z))\big)\big], \end{aligned}$$

and the game is $\min_G \max_D V(G, D)$.

The discriminator wants $D(x)$ close to $1$ on real data and $D(G(z))$ close to $0$ on fakes; both make $V$ larger. The generator wants $D(G(z))$ close to $1$, which makes the second term very negative.

Where the value function comes from: binary cross-entropy#

The discriminator is a binary classifier, so start from the binary cross-entropy between a label $y$ and a prediction $\hat y$:

$$H(y, \hat y) = -\big[\,y\ln\hat y + (1 - y)\ln(1 - \hat y)\,\big].$$

• For a real sample, $y = 1$, $\hat y = D(x)$ and $H = -\ln D(x)$.
• For a fake sample, $y = 0$, $\hat y = D(G(z))$ and $H = -\ln\big(1 - D(G(z))\big)$.

For one real and one fake sample the total is $H = -\big[\ln D(x) + \ln\big(1 - D(G(z))\big)\big]$. Averaging over many such pairs turns the sums into expectations:

$$\begin{aligned} \mathbb{E}[-H] ={}& \mathbb{E}_{x\sim p_{\text{data}}}\big[\ln D(x)\big] \\ &+ \mathbb{E}_{z\sim p_z}\big[\ln\big(1 - D(G(z))\big)\big], \end{aligned}$$

which for continuous distributions reads

$$\begin{aligned} &\int p_{\text{data}}(x)\ln D(x)\,dx \\ &\quad + \int p_z(z)\ln\big(1 - D(G(z))\big)\,dz . \end{aligned}$$

This is exactly $V(G, D)$. So $V$ is the negative binary cross-entropy of the discriminator: maximizing $V$ over $D$ is the same as training $D$ as an ordinary classifier with a BCE loss. The two terms have equal weight because the classifier sees real and fake samples in equal proportion, a class prior of $\tfrac12$. With other proportions both $V$ and the optimal discriminator below change.

How to optimize: alternate the players#

The two players take turns, each with a minibatch of size $m$ (Goodfellow et al., 2014, Algorithm 1).

1. Fix $G$. Take $m$ real samples $x^{(1)},\dots,x^{(m)}$ and $m$ noise samples $z^{(1)},\dots,z^{(m)}$, and update $\theta_d$ by gradient ascent along

$$\begin{aligned} \nabla_{\theta_d}\,\frac1m\sum_{i=1}^{m}\Big[&\ln D\big(x^{(i)}\big) \\ &+ \ln\Big(1 - D\big(G(z^{(i)})\big)\Big)\Big]. \end{aligned}$$

2. Fix $D$. Take $m$ fresh noise samples and update $\theta_g$ by gradient descent along

$$\nabla_{\theta_g}\,\frac1m\sum_{i=1}^{m}\ln\Big(1 - D\big(G(z^{(i)})\big)\Big).$$

The generator step uses only the second term because the first term does not depend on $G$: $\ln D(x)$ involves real data and $\theta_d$ only, so its gradient with respect to $\theta_g$ is zero. Step 1 may be repeated $k$ times per generator step; the original paper used $k = 1$. How to choose $k$ is discussed under balancing the two players.

Global optimality: the generator matches the data#

The game is won when the generator’s distribution equals the data distribution. The proof has two steps (Goodfellow et al., 2014, Proposition 1 and Theorem 1).

Step 1: the optimal discriminator for a fixed generator#

As in Goodfellow et al. (2014), Section 4, we assume that both distributions have densities on the same space and that $D$ can be any function (infinite capacity). When $\dim z < \dim x$ the first assumption fails, and the paragraph on non-overlapping supports below covers that case.

Rewrite the second expectation over generated samples: $x = G(z)$ has density $p_g$, so $\mathbb{E}_{z\sim p_z}\big[\ln(1 - D(G(z)))\big] = \mathbb{E}_{x\sim p_g}\big[\ln(1 - D(x))\big]$. Then, for fixed $G$,

$$\begin{aligned} V(G, D) = \int_x \Big[\,&p_{\text{data}}(x)\ln D(x) \\ &+ p_g(x)\ln\big(1 - D(x)\big)\Big]\,dx . \end{aligned}$$

$D$ can be chosen freely at every $x$, so the integrand can be maximized pointwise. Write $a = p_{\text{data}}(x)$, $b = p_g(x)$ and $s = D(x)$. If $a, b > 0$, the function $f(s) = a\ln s + b\ln(1 - s)$ on $(0, 1)$ has

$$\begin{aligned} f'(s) &= \frac{a}{s} - \frac{b}{1 - s} = 0 \;\Longrightarrow\; s^* = \frac{a}{a + b}, \\ f''(s) &= -\frac{a}{s^2} - \frac{b}{(1 - s)^2} < 0, \end{aligned}$$

so $s^*$ is the unique maximum, and it lies inside $(0, 1)$. If exactly one density vanishes, $f$ is monotone on $[0, 1]$ and its maximum sits at an endpoint: $s^* = 1$ where $b = 0$ (only data) and $s^* = 0$ where $a = 0$ (only fakes). The formula $a/(a + b)$ gives these values too; a sigmoid output can only approach them. Where both densities vanish, $D$ does not matter (Goodfellow et al., 2014, Proposition 1).

Figure

Each point x is a one-variable maximization

Hence, for a fixed generator, the best discriminator is

$$D^*_G(x) = \frac{p_{\text{data}}(x)}{p_{\text{data}}(x) + p_g(x)} .$$

Figure

The optimal discriminator compares the two densities

Step 2: substitute the optimal discriminator and recognize JS#

Plug $D^*_G$ back into $V$. Since $1 - D^*_G = \dfrac{p_g}{p_{\text{data}} + p_g}$,

$$\begin{aligned} C(G) :={}& \max_D V(G, D) = V(G, D^*_G) \\ ={}& \mathbb{E}_{x\sim p_{\text{data}}}\ln\frac{p_{\text{data}}(x)}{p_{\text{data}}(x) + p_g(x)} \\ &+ \mathbb{E}_{x\sim p_g}\ln\frac{p_g(x)}{p_{\text{data}}(x) + p_g(x)} . \end{aligned}$$

The Jensen–Shannon divergence (defined here) compares each distribution with the mixture $m = \tfrac12(p_1 + p_2)$. Written with expectations,

$$\begin{aligned} D_{\mathrm{JS}}(p_1\,\|\,p_2) &= \tfrac12 D_{\mathrm{KL}}(p_1\,\|\,m) \\ &\quad + \tfrac12 D_{\mathrm{KL}}(p_2\,\|\,m) \\ &= \tfrac12\,\mathbb{E}_{x\sim p_1}\ln\frac{p_1(x)}{m(x)} \\ &\quad + \tfrac12\,\mathbb{E}_{x\sim p_2}\ln\frac{p_2(x)}{m(x)} . \end{aligned}$$

To make the denominators match, divide each by $2$ inside the logarithm and pay $\ln 2$ for each expectation, because $\ln\frac{p}{p+q} = \ln\frac{p}{(p+q)/2} - \ln 2$. With $m = \tfrac12(p_{\text{data}} + p_g)$,

$$\begin{aligned} C(G) ={}& \mathbb{E}_{x\sim p_{\text{data}}}\ln\frac{p_{\text{data}}(x)}{m(x)} \\ &+ \mathbb{E}_{x\sim p_g}\ln\frac{p_g(x)}{m(x)} - 2\ln 2 , \end{aligned}$$

that is,

$$C(G) = 2\,D_{\mathrm{JS}}(p_{\text{data}}\,\|\,p_g) - 2\ln 2 .$$

Since $D_{\mathrm{JS}} \ge 0$, with equality only when $p_{\text{data}} = p_g$, the minimum over $G$ is reached only at $p_g = p_{\text{data}}$:

$$\min_G C(G) = -2\ln 2 \approx -1.386 ,$$

where $D^* = \tfrac12$ everywhere. For the two densities in the optimal-discriminator figure, $D_{\mathrm{JS}} \approx 0.430$ nats and $C(G) \approx -0.526$, computed by numerical integration both from the definition of $V$ and from the JS formula.

This also explains the warning about non-overlapping supports. If $p_{\text{data}}$ and $p_g$ live on disjoint sets, then $D^* = 1$ on the data and $D^* = 0$ on the fakes, $D_{\mathrm{JS}} = \ln 2$ and $C(G) = 0$, no matter where exactly $p_g$ sits. So $C$ is flat in the generator’s parameters. Backpropagation, however, does not differentiate $C$: it sends the gradient of $\ln\big(1 - D(G(z))\big)$ through a fixed $D$, and that gradient contains $\nabla_x D$ at the generated samples. When the supports are disjoint compact sets, or manifolds that do not perfectly align, there is an optimal discriminator that equals $0$ on a whole neighbourhood of the generated samples, so $\nabla_x D^* = 0$ there and the generator’s gradient vanishes (Arjovsky and Bottou, 2017, Theorems 2.1–2.2). For a discriminator within $\varepsilon$ of $D^*$, in value and gradient, the gradient norm is at most $M\varepsilon/(1 - \varepsilon)$, where $M$ bounds the generator’s Jacobian (Theorem 2.4).

This is not an exotic case. A generator built from affine maps and pointwise nonlinearities maps $z$ into a countable union of manifolds of dimension at most $\dim z$ (Lemma 1), and natural images are believed to concentrate near a low-dimensional manifold. Two such manifolds almost never overlap on a set of positive measure: they may cross, but only on a lower-dimensional set (Lemmas 2–3). So a perfect discriminator exists and $D_{\mathrm{JS}} = \ln 2$ however close the two manifolds are (Theorems 2.2–2.3). It is the same saturation of JS that the fitting note observed in one dimension. The WGAN note replaces JS with the Wasserstein-1 distance to fix it.

Training as a picture#

Figure

How the two players move during training

1. At the beginning, $p_g$ is far from $p_{\text{data}}$ and the discriminator is arbitrary.
2. After updating $\theta_d$, $D$ approaches $D^*$: high over the data, low over the generated samples.
3. After updating $\theta_g$, the gradient of $D$ guides $G(z)$ towards regions that $D$ classifies as data, so $p_g$ moves towards $p_{\text{data}}$.
4. Finally, $p_g = p_{\text{data}}$ and the discriminator cannot tell them apart: $D(x) = \tfrac12$.

The picture is idealized. The theory assumes that $D$ reaches its optimum after every generator step and that both players can represent any function. In practice both are networks trained a few steps at a time, and the alternating updates can oscillate or collapse onto a few modes of the data (mode collapse) instead of converging (Goodfellow, 2016, Section 5.1.1). The remedies are collected under mode collapse and how to fight it.

Which generator loss is used in practice?#

The minimax loss for the generator, $\mathbb{E}_z\ln\big(1 - D(G(z))\big)$, is rarely minimized directly. Already the original paper recommends (Goodfellow et al., 2014, Section 3) training the generator to maximize $\ln D(G(z))$ instead, that is, to minimize

$$\mathcal L_G = -\mathbb{E}_z\ln D\big(G(z)\big),$$

the non-saturating loss. In code this is the BCE of the discriminator’s output on generated samples with the “real” label.

The problem with the minimax loss. When $D$ is good, it rejects fakes confidently: $d = D(G(z)) \approx 0$. Then $\ln(1 - d) \approx \ln 1 = 0$. A small value does not by itself mean a small gradient; what matters is the slope of the loss with respect to $d$, and further back through the network:

$$\begin{aligned} \frac{\partial}{\partial d}\ln(1 - d) &= -\frac{1}{1 - d} \approx -1, \\ \frac{\partial}{\partial d}\big(-\ln d\big) &= -\frac1d \to -\infty \quad (d \to 0). \end{aligned}$$

At $d = 0.05$ the slopes are $-1.05$ and $-20$. The discriminator ends in a sigmoid, $d = \sigma(\ell)$, and $\partial d/\partial\ell = d(1 - d)$. Per logit the gradients are therefore $-d \to 0$ for the minimax loss and $-(1 - d) \to -1$ for the non-saturating one. The minimax loss saturates at the logit, which is where the gradient enters the network: its gradient vanishes exactly when the generator is worst, while the non-saturating gradient is largest there.

Figure

Saturating and non-saturating generator losses

Both losses decrease as $D(G(z))$ grows, so they push the generator in the same direction, and for the optimal discriminator they share the fixed point $p_g = p_{\text{data}}$ (Goodfellow et al., 2014, Section 3). The non-saturating version is no longer the minimax game exactly. With $D = D^*$, its expected gradient equals the gradient of $D_{\mathrm{KL}}(p_g\,\|\,p_{\text{data}}) - 2\,D_{\mathrm{JS}}(p_{\text{data}}\,\|\,p_g)$ (Arjovsky and Bottou, 2017, Theorem 2.5). The first term is a reverse KL: it punishes fake-looking samples heavily and missing modes only lightly, one explanation for mode dropping. The non-saturating loss fixes the weak gradients early in training, but not the disjoint-support problem of the previous section: if the discriminator’s error around $D^*$ is modelled as independent noise, its updates become unstable rather than vanishing, with infinite variance (Theorem 2.6).

Computing the losses from logits#

The discriminator network ends in a logit $\ell$, and $D = \sigma(\ell)$. Both logarithms in the losses are functions of $\ell$ that never need $\sigma(\ell)$ itself:

$$\begin{aligned} \ln D &= \ln\sigma(\ell) = -\operatorname{softplus}(-\ell),\\ \ln(1 - D) &= \ln\sigma(-\ell) = -\operatorname{softplus}(\ell), \end{aligned}$$

with $\operatorname{softplus}(t) = \ln(1 + e^{t})$. The first line is $\sigma(\ell) = 1/(1 + e^{-\ell})$, and the second uses $1 - \sigma(\ell) = \sigma(-\ell)$. Softplus is evaluated as $\max(t, 0) + \ln(1 + e^{-\lvert t\rvert})$, so the exponential never overflows. The discriminator’s loss $-V$ becomes $\mathbb E_x\operatorname{softplus}(-\ell(x)) + \mathbb E_z\operatorname{softplus}(\ell(G(z)))$ and the non-saturating generator loss $\mathbb E_z\operatorname{softplus}(-\ell(G(z)))$.

Computing $\sigma$ first and the logarithm afterwards fails exactly where the discriminator is confident. In 32-bit floating point, $\sigma(\ell)$ rounds to exactly $1$ once $\ell$ exceeds about 17, so $\ln(1 - \sigma(\ell))$ returns $-\infty$ where the softplus form returns about $-17$, and no gradient passes through the rounded sigmoid. Deep-learning libraries provide the stable form as binary cross-entropy with logits (binary_cross_entropy_with_logits in PyTorch).

Training GANs in practice#

The theory above assumes an optimal discriminator and networks of unlimited capacity. Real GANs are two finite networks trained a few steps at a time, and a set of empirical rules decides whether training works at all. None of them is a theorem; each is given here with the reason it helps.

Architecture: the DCGAN guidelines#

The first convolutional GANs that trained reliably on images followed five rules (Radford, Metz and Chintala, 2016, Section 3):

1. No pooling. The discriminator downsamples with strided convolutions and the generator upsamples with fractionally strided (transposed) convolutions, so both networks learn their own resampling.
2. Batch normalization in both networks, except at the generator’s output layer and the discriminator’s input layer, where it made the samples oscillate.
3. No fully connected hidden layers. The generator’s first layer is a matrix multiplication whose output is reshaped into a small spatial map, $4 \times 4$ with 1024 channels in the paper’s $64 \times 64$ generator. The discriminator’s last convolutional features are flattened into a single sigmoid output.
4. ReLU in the generator and a Tanh at its output, so that images lie in $[-1, 1]$, like the training images scaled to that range.
5. LeakyReLU in the discriminator, with slope $0.2$.

The training details became defaults as well (Section 4): Adam with learning rate $2\cdot10^{-4}$ and $\beta_1 = 0.5$, because the usual $\beta_1 = 0.9$ made training oscillate; minibatches of 128; weights initialized from $\mathcal N(0, 0.02^2)$; and a 100-dimensional uniform noise vector.

The rules have reasons that earlier notes explain. Strided convolutions and LeakyReLU keep gradients dense: max-pooling sends a gradient to only one input of each window, and a ReLU sends none for negative inputs, while LeakyReLU keeps the slope $0.2$ there (see common remedies for vanishing gradients). Batch normalization was what made deep generators start learning, and it kept them from collapsing all samples to one point (Section 3). Inside the discriminator it also makes each output depend on the statistics of the whole minibatch, a weak form of the minibatch discrimination below (Salimans et al., 2016, Sections 3.2 and 3.5).

Later architectures change some rules but keep their purpose.

• Upsampling. A transposed convolution whose kernel size is not divisible by its stride overlaps unevenly and can leave checkerboard artifacts. Resizing followed by an ordinary convolution avoids the uneven overlap (transposed convolution or upsample + conv?).
• A fully convolutional discriminator. The last layer can be a convolution whose kernel covers the whole remaining feature map, for example $4 \times 4$ on a $4 \times 4$ map, which returns one logit per image without a fully connected layer. Radford et al. found that global average pooling instead increased stability but slowed convergence (Section 3).
• Dropout in the discriminator. Dropping whole feature maps at random, as in the denoising autoencoder, regularizes the discriminator so that it fits the training images less closely.
• The latent code. No theory fixes its size. A code that is too small cannot represent the variety of the data; DCGAN uses 100 dimensions, and 100 to 256 are common. A convolutional generator can also start from a small spatial grid of noise, such as $4 \times 4$ with a few channels, and double the resolution in every block, $4 \to 8 \to 16 \to 32 \to 64$.
• Sampling from a narrowed prior. Scaling the noise down at sampling time, for example drawing $z \sim \mathcal N(0, 0.7^2 I)$ instead of $\mathcal N(0, I)$, gives more typical but less varied images. The samples then come from a different distribution than the one the generator was trained on. This prior narrowing is a truncation-like heuristic, often loosely called the truncation trick, which trades recall for precision; strictly, scaling a Gaussian is not sampling a truncated one, which BigGAN does by resampling values beyond a threshold. Any metric computed on such samples measures the modified distribution, so the sampling scale has to be reported with it.

Balancing the two players#

The discriminator is the generator’s only source of gradient, so the two must improve at comparable speeds. The original paper already warns that a generator trained too long against a fixed discriminator maps too many values of $z$ to the same $x$ (Goodfellow et al., 2014, Section 6). The knobs are the number $k$ of discriminator steps per generator step, the two learning rates and the capacity of each network.

• The discriminator saturates: it rejects the fakes with near certainty, and the generator’s gradients shrink. The minimax loss saturates, and with disjoint supports even the optimal discriminator gives no direction. Update the discriminator less often, lower its learning rate, or reduce its capacity.
• The generator struggles, or collapses to a few modes, while the discriminator’s feedback is unstable: the discriminator lags behind and points in poor directions. Update it more often, $k > 1$.

Neither extreme works. Too many discriminator steps make the generator’s gradients vanish; too few leave a discriminator that cannot give meaningful feedback. The next note changes the objective so that a strong critic helps rather than hurts.

Three habits complete the picture. Gradient-norm clipping (common remedies) guards against occasional explosive updates; Nowozin, Cseke and Tomioka found Adam and clipping useful in their largest experiment (2016, Section 3.2). A small architecture or dataset first checks that the pipeline works. And samples should be looked at often: the two losses move against each other and do not measure sample quality, so samples, and later metrics such as the Inception Score, are what reveals a collapse early.

Label smoothing and label flipping#

One-sided label smoothing trains the discriminator against the target $\alpha = 0.9$ instead of $1$ on real samples, so that it never becomes fully confident (Salimans et al., 2016, Section 3.4). Why only on real samples? Let the targets be $\alpha$ for real and $\beta$ for fake samples. The integrand of Step 1 becomes $A\ln s + B\ln(1 - s)$ with $A = \alpha\,p_{\text{data}} + \beta\,p_g$ and $B = (1 - \alpha)\,p_{\text{data}} + (1 - \beta)\,p_g$, so its maximum is at $A/(A + B)$:

$$D^*(x) = \frac{\alpha\,p_{\text{data}}(x) + \beta\,p_g(x)}{p_{\text{data}}(x) + p_g(x)} .$$

Look at a region where the generator puts much more mass than the data, so that the ratio $r = p_{\text{data}}/p_g$ is small. There $D^* \approx \beta + (\alpha - \beta)\,r$ to first order in $r$.

• With $\beta = 0$, the non-saturating loss $-\ln D^* \approx -\ln\alpha - \ln r$ follows the log density ratio, and its gradient does not shrink as $r$ does.
• With $\beta > 0$, it is $-\ln D^* \approx -\ln\beta - \frac{\alpha - \beta}{\beta}\,r$, whose gradient $-\frac{\alpha - \beta}{\beta}\,r\,\nabla\ln r$ carries the small factor $r$: fakes far from the data barely move.

So only the real label is smoothed. With $\beta = 0$ the optimal output is at most $\alpha$: where the two densities are equal it is $0.45$ instead of $0.5$.

Random label flipping swaps the labels of a small random fraction $\varepsilon$ of the discriminator’s training examples. The binary cross-entropy $H(y, \hat y) = -[y\ln\hat y + (1 - y)\ln(1 - \hat y)]$ is linear in the target $y$, so in expectation flipping replaces the targets $1$ and $0$ by $1 - \varepsilon$ and $\varepsilon$. It is two-sided label smoothing with $\alpha = 1 - \varepsilon$ and $\beta = \varepsilon$, plus noise. It keeps the discriminator from becoming overconfident, but by the argument above its fake half weakens the pull on fakes far from the data, so it is used sparingly, if at all.

Mode collapse and how to fight it#

In mode collapse the generator maps most of its noise to a few outputs that the current discriminator accepts. Salimans et al. (2016, Section 3.2) describe the mechanism. The discriminator judges each sample on its own, so its gradient points the same way for all similar samples, and nothing tells the generator’s outputs to differ from each other. When the discriminator learns to reject the collapsed mode, the generator moves to another one, and the two can chase each other indefinitely. The reverse-KL term of the non-saturating loss (above) is a second reason: dropping a mode costs it little. The remedies attack different parts of this mechanism.

• Minibatch discrimination lets the discriminator compare samples (Salimans et al., 2016, Section 3.2). An intermediate feature vector $f(x_i) \in \mathbb R^A$ is multiplied by a learned tensor $T \in \mathbb R^{A\times B\times C}$, which gives a matrix $M_i \in \mathbb R^{B\times C}$. For each of its rows $b$, the closeness of two samples is $c_b(x_i, x_j) = \exp\big(-\lVert M_{i,b} - M_{j,b}\rVert_1\big)$, and the layer outputs $o(x_i)_b = \sum_{j} c_b(x_i, x_j)$, summed over the minibatch. The vector $o(x_i) \in \mathbb R^B$ is concatenated with $f(x_i)$ and fed to the next layer, separately for real and generated minibatches. A collapsed batch contains many near-identical samples and therefore large values of $o$, which the discriminator can learn to reject.
• Unrolled GANs let the generator anticipate the discriminator’s answer (Metz et al., 2017, Sections 2.2–2.4). Write the value function as $V(\theta_G, \theta_D)$. Starting from $\theta_D^0 = \theta_D$, take $K$ gradient-ascent steps of the discriminator, $\theta_D^{k+1} = \theta_D^k + \eta\,\partial V(\theta_G, \theta_D^k)/\partial\theta_D^k$, and define the surrogate $V_K(\theta_G, \theta_D) = V\big(\theta_G, \theta_D^K(\theta_G, \theta_D)\big)$. The generator descends $V_K$, backpropagating through the $K$ steps, while the discriminator is still updated with $V$. By the chain rule the gradient of $V_K$ has two terms (Section 2.4): the usual $\partial V/\partial\theta_G$ at the unrolled discriminator, and a term through $d\theta_D^K/d\theta_G$ that describes how the discriminator would react to the generator’s move. The first term alone pushes toward a collapse: against a fixed discriminator, the best generator puts all its mass on the single point that the discriminator likes most. The second term sees that a collapsed generator would quickly be rejected, which discourages the collapse. $K = 0$ is the standard GAN. If the $K$ inner steps converge to an optimal discriminator, $K \to \infty$ approaches the true generator objective $V(\theta_G, \theta_D^*(\theta_G))$; for a neural discriminator the inner ascent need not converge, or may reach only a local optimum, so a large $K$ is an approximation, not a guarantee.
• History. Historical averaging adds $\big\lVert\theta - \tfrac1t\sum_{i=1}^{t}\theta[i]\big\rVert^2$ to each player’s cost, where $\theta[i]$ are the parameters at past step $i$: a pull toward the average of all past parameters that damps oscillations (Salimans et al., 2016, Section 3.3). Training the discriminator partly on a buffer of samples from earlier generators, half of each discriminator minibatch in Shrivastava et al. (2017), Section 2.3, keeps it from forgetting the modes the generator visited before, so the generator cannot return to them unpunished.
• Several discriminators. GMAN trains the generator against $N$ discriminators and combines their values with a softened maximum, from the mean to the maximum as a parameter $\lambda$ goes from $0$ to $\infty$ (Durugkar, Gemp and Mahadevan, 2017, Section 4.1). The mean is a lenient teacher and the maximum a harsh adversary.
• A different distance. The Wasserstein GAN, with a gradient penalty or spectral normalization on its critic, gives useful gradients even when the supports do not overlap. The original WGAN, trained with weight clipping, showed no evidence of mode collapse in any of its authors’ experiments (Arjovsky, Chintala and Bottou, 2017, Section 4.3).

Extensions#

Conditional GAN#

A conditional GAN (Mirza and Osindero, 2014) gives both networks a label $c$, such as a class: the generator produces $G(z, c)$ and the discriminator judges pairs $D(x, c)$. Let $p_{\text{data}}(x, c)$ be the joint distribution of data and labels and $p_c$ its label marginal. The value function becomes

$$\begin{aligned} V_{\text{cond}} ={}& \mathbb{E}_{(x,c)\sim p_{\text{data}}}\big[\ln D(x,c)\big] \\ &+ \mathbb{E}_{c,\,z}\big[\ln\big(1-D(G(z,c),c)\big)\big], \end{aligned}$$

where the second expectation is over $c \sim p_c$ and $z \sim p_z$. This is the original game played on pairs $(x, c)$: the generated pairs have the joint $p_g(x, c) = p_c(c)\,p_g(x \mid c)$, and the argument of the previous sections, applied to pairs, shows that the optimum matches the joints, $p_g(x, c) = p_{\text{data}}(x, c)$. A realistic image paired with the wrong label lies outside the data joint, so the discriminator can reject it, and the generator learns to follow the condition. It is the GAN counterpart of the conditional VAE.

f-GAN#

Optional depth: the game for any f-divergence through convex conjugates. The rest of the note does not depend on it.

The original GAN minimizes JS. An f-GAN (Nowozin, Cseke and Tomioka, 2016) generalizes the game to any f-divergence

$$D_f(P\,\|\,Q) = \int q(x)\, f\!\left(\frac{p(x)}{q(x)}\right) dx ,$$

for a convex, lower semicontinuous $f$ with $f(1) = 0$; $f(u) = u\ln u$ gives $D_{\mathrm{KL}}(P\,\|\,Q)$. The key tool is the convex conjugate $f^*(t) = \sup_u \{ut - f(u)\}$. Rearranged, its definition says $f(u) \ge tu - f^*(t)$ for every $t$: each line with slope $t$ and intercept $-f^*(t)$ lies below the graph of $f$, and for differentiable $f$ the line with $t = f'(u)$ touches it at $u$. For convex, lower semicontinuous $f$ these lines recover $f$ exactly, $f(u) = \sup_t \{tu - f^*(t)\}$ (the Fenchel–Moreau theorem). For example, $f(u) = \tfrac12(u - 1)^2$, half the Pearson $\chi^2$ generator, gives $f^*(t) = t + \tfrac12 t^2$: the supremum of $tu - \tfrac12(u-1)^2$ is at $u = 1 + t$.

Figure

The convex conjugate read off a tangent line

Substitute this representation at every point, with $u(x) = p(x)/q(x)$, and let the slope $t = T(x)$ depend on $x$ (Nguyen, Wainwright and Jordan, 2010; Nowozin, Cseke and Tomioka, 2016, Section 2.2, Eqs. 4–5):

$$\begin{aligned} D_f(P\,\|\,Q) &= \int q(x) \sup_t \big\{ t u(x) - f^*(t) \big\}\,dx \\ &= \sup_T \int \Big[ p(x)\,T(x) \\ &\qquad - q(x)\,f^*\big(T(x)\big) \Big]\, dx \\ &\ge \sup_{T \in \mathcal T} \Big\{\mathbb{E}_{x\sim P}\big[T(x)\big] \\ &\qquad - \mathbb{E}_{x\sim Q}\big[f^*(T(x))\big]\Big\}. \end{aligned}$$

The second line picks the best slope separately at every point. Over all functions $T$ with values in the domain of $f^*$ this loses nothing: for differentiable $f$ the supremum is attained at $T^*(x) = f'\big(p(x)/q(x)\big)$. The inequality appears in the third line, where $T$ is restricted to a class $\mathcal T$, such as a neural network family.

With $P = p_{\text{data}}$, $Q = p_g$ and $T$ a neural network, the right-hand side is a two-player game: $T$ plays the discriminator’s role and maximizes, and the generator minimizes. Both expectations need only samples, never densities, which is what makes the bound usable for an implicit model. The original GAN is one case, with a generator that is not normalized. With $f(u) = u\ln u - (u + 1)\ln(u + 1)$ we have $f(1) = -2\ln 2 = -\ln 4$ rather than $0$: this $f$ is a normalized generator plus the constant $-\ln 4$, and since $\int q\,(-\ln 4)\,dx = -\ln 4$ the divergence is shifted by the same constant, $D_f(P\,\|\,Q) = 2\,D_{\mathrm{JS}}(P\,\|\,Q) - \ln 4$, and the conjugate is $f^*(t) = -\ln(1 - e^t)$ for $t < 0$. Writing $T(x) = \ln D(x)$ with $D(x) \in (0, 1)$ gives $f^*(T(x)) = -\ln(1 - D(x))$, and the bound becomes exactly $V(G, D)$ (Nowozin, Cseke and Tomioka, 2016, Section 2.4, Tables 1 and 6).

Output activations. The bound needs $T(x)$ inside the domain of $f^*$, which for most divergences is not the whole real line. f-GAN therefore lets the network output an unconstrained number $v(x)$ and maps it through a fixed increasing output activation $g_f$, so that $T(x) = g_f(v(x))$ (Section 2.4, Eq. 7). The game becomes

$$\begin{aligned} F ={}& \mathbb{E}_{x\sim P}\big[g_f(v(x))\big] \\ &- \mathbb{E}_{x\sim Q}\big[f^*\big(g_f(v(x))\big)\big] . \end{aligned}$$

The table lists the usual choices (Tables 1 and 2). The convention is the one above: $D_f(P\,\|\,Q) = \int q\, f(p/q)\,dx$ with $P = p_{\text{data}}$ and $Q = p_g$, and the generators $f$ are those of Nowozin et al. A condition after $f^*(t)$ is its domain.

Divergence	$f(u)$	$g_f(v)$	$f^*(t)$
$D_{\mathrm{KL}}(P\Vert Q)$	$u\ln u$	$v$	$e^{t-1}$
$D_{\mathrm{KL}}(Q\Vert P)$	$-\ln u$	$-e^{-v}$	$-1-\ln(-t)$, $t<0$
Pearson $\chi^2$	$(u-1)^2$	$v$	$t + t^2/4$
squared Hellinger	$(\sqrt u - 1)^2$	$1 - e^{-v}$	$t/(1-t)$, $t<1$
$2D_{\mathrm{JS}}$	$u\ln u - (u+1)\ln\frac{u+1}{2}$	$\ln 2 - \ln(1 + e^{-v})$	$-\ln(2 - e^{t})$, $t<\ln 2$
GAN: $2D_{\mathrm{JS}} - \ln 4$, unnormalized	$u\ln u - (u+1)\ln(u+1)$	$-\ln(1 + e^{-v})$	$-\ln(1 - e^{t})$, $t<0$

Three remarks on the table.

• Every $g_f$ is increasing, so a large $v(x)$ always means “looks real”.
• Nowozin et al. label the fifth row Jensen–Shannon, but with this $f$ the divergence is twice $D_{\mathrm{JS}}$ as defined in the entropy note. Its $f$ exceeds the GAN row’s by $(u+1)\ln 2$, so the two divergences differ by the constant $\ln 4$, and the two rows train identically.
• The Pearson row uses $f(u) = (u - 1)^2$. The generator $\tfrac12(u - 1)^2$ of the example above gives half that divergence and the conjugate $t + t^2/2$.

Compose before computing. Evaluating $g_f$ and then $f^*$ loses precision for the same reason as $\sigma$ followed by $\ln$ (above), and Nowozin et al. recommend implementing the composition $f^*(g_f(v))$ directly (Section 2.4, footnote 1). It is simple for every row: from top to bottom, $e^{v-1}$, $v - 1$, $v + v^2/4$, $e^{v} - 1$, $\operatorname{softplus}(v) - \ln 2$ and $\operatorname{softplus}(v)$. For the GAN row $g_f(v) = -\operatorname{softplus}(-v) = \ln\sigma(v)$, so $F$ is exactly minus the discriminator’s binary cross-entropy computed from its logit $v$.

The generator’s step. As for the GAN, the generator can maximize $\mathbb E_{x\sim Q}\big[g_f(v(x))\big]$ instead of minimizing the fake term of $F$. This keeps the stationary point, and Nowozin et al. found it useful for divergences other than the GAN’s as well (Section 3.2).

Least-squares GAN#

Optional depth: a squared-error discriminator loss and what its optimum minimizes. The rest of the note does not depend on it.

The non-saturating loss repairs the gradient for fakes that the discriminator confidently rejects. The least-squares GAN (LSGAN) also repairs the opposite case, fakes that the discriminator already accepts but that are still far from the data (Mao et al., 2017). Its discriminator has no sigmoid. It is a regression network with a real-valued output, trained toward the value $b$ on real samples and $a$ on fakes, and the generator wants its samples to be scored $c$ (Section 3.2, Eq. 2):

$$\begin{aligned} \min_D\;& \tfrac12\,\mathbb E_{x\sim p_{\text{data}}}\big[(D(x) - b)^2\big] \\ &+ \tfrac12\,\mathbb E_{z\sim p_z}\big[(D(G(z)) - a)^2\big],\\ \min_G\;& \tfrac12\,\mathbb E_{z\sim p_z}\big[(D(G(z)) - c)^2\big]. \end{aligned}$$

The common choice is the 0–1 coding $a = 0$, $b = c = 1$ (Eq. 9): the discriminator regresses real samples to 1 and fakes to 0, and the generator asks for 1. Mao et al. found that it performs like the $\chi^2$ coding below and used it in their experiments (Section 3.2.3).

Why least squares helps. The decision boundary is where $D$ lies halfway between $a$ and $b$. A fake can lie on the real side of that boundary, so the discriminator already scores it as real, and still be far from the data. With the sigmoid cross-entropy such a sample has almost no loss and almost no gradient: the non-saturating loss $-\ln\sigma(\ell)$ flattens as the logit $\ell$ grows, with slope $-(1 - \sigma(\ell)) \to 0$. The squared loss is flat only at $D = c$, so a fake scored beyond $c$ is pulled back, but only as far as the level set $D = c$. The discriminator is fixed during the generator’s step, so this moves such fakes toward the decision boundary, and for training to succeed the boundary should run through the data (Section 3.2.1).

Figure

Least squares pull far-away fakes back

The extra gradient comes from the shape of a learned real-valued discriminator, which keeps growing away from the boundary, as the linear one in the figure does. The unconstrained optimal discriminator below lies between $a$ and $b$ everywhere, and far from both distributions its squared loss can flatten too.

The optimal discriminator and Pearson $\chi^2$. For a fixed generator, the discriminator’s loss can be minimized pointwise, as in Step 1. The integrand $\tfrac12 p_{\text{data}}(D - b)^2 + \tfrac12 p_g(D - a)^2$ is a parabola in $D$ with its minimum at (Eq. 5)

$$D^*(x) = \frac{b\,p_{\text{data}}(x) + a\,p_g(x)}{p_{\text{data}}(x) + p_g(x)} .$$

Add to the generator’s loss the term $\tfrac12\,\mathbb E_{x\sim p_{\text{data}}}\big[(D(x) - c)^2\big]$; it does not depend on $G$, so it changes no gradient. At $D = D^*$, using $D^* - c = \big((b - c)\,p_{\text{data}} + (a - c)\,p_g\big)/(p_{\text{data}} + p_g)$, the extended loss is (Eqs. 4 and 6)

$$\begin{aligned} 2C(G) &= \int \big(p_{\text{data}} + p_g\big)\big(D^* - c\big)^2\,dx \\ &= \int \frac{\big((b - c)\,p_{\text{data}} + (a - c)\,p_g\big)^2}{p_{\text{data}} + p_g}\,dx . \end{aligned}$$

Choose $b - c = 1$ and $b - a = 2$, for example $a = -1$, $b = 1$, $c = 0$ (Eq. 8). Then $a - c = -1$ and the numerator is $(p_{\text{data}} - p_g)^2$. With the mixture $m = \tfrac12(p_{\text{data}} + p_g)$ we have $p_{\text{data}} - p_g = 2(m - p_g)$ and $p_{\text{data}} + p_g = 2m$, so

$$C(G) = \int \frac{\big(p_g(x) - m(x)\big)^2}{m(x)}\,dx = \chi^2_{\text{Pearson}}(p_g\,\|\,m),$$

the Pearson row of the f-GAN table with $p_g$ in place of $P$ and $m$ in place of $Q$. This mirrors the JS identity of the original GAN, which compares both distributions with the same mixture through KL divergences. $C(G)$ is zero only when $p_g = m$, that is, $p_g = p_{\text{data}}$. Mao et al. write the same quantity as $\chi^2_{\text{Pearson}}(p_{\text{data}} + p_g\,\|\,2p_g)$ (Eq. 7), with the reference measure as the first argument. For the two Gaussians of the figure, $C(G) \approx 0.550$ (a toy value, by numerical integration).

The 0–1 coding does not meet these conditions: there $b - c = 0$, and $2C(G) = \int p_g^2/(p_{\text{data}} + p_g)\,dx$, which equals $\tfrac12$, not $0$, at $p_g = p_{\text{data}}$. It is still minimized there. The integrand is convex in $p_g(x)$, and at $p_g = p_{\text{data}}$ its derivative with respect to $p_g(x)$, $(p_g^2 + 2p_g\,p_{\text{data}})/(p_{\text{data}} + p_g)^2$, equals $\tfrac34$ at every $x$: the optimality condition for a convex objective under the constraint $\int p_g = 1$.

Strengths and weaknesses#

Strengths.

• Sampling is one forward pass of the generator, and training needs only backpropagation: no Markov chains, no inference over latent variables during learning and no intractable integral (Goodfellow et al., 2014, Section 6).
• The model can represent very sharp, even degenerate distributions (Section 6). Nothing averages plausible images into a blur, as the squared error of a Gaussian decoder does.
• The same game accepts labels, other divergences and other losses, as the extensions above show.

Weaknesses.

• No density. $p_g(x)$ has no explicit form (Section 6), so the likelihood of test data cannot be computed, and models cannot be compared by it. The two losses do not measure sample quality, and there is no clear stopping criterion. Evaluation needs sample-based metrics, the subject of the Inception Score and the notes after it.
• No way back. The generator maps $z$ to $x$, but nothing maps an image back to its code. Finding the code of a given image needs an optimization over $z$ or an extra network trained for the purpose.
• Unstable training. The alternating updates can oscillate, and the two players must be kept in balance.
• Mode collapse. The generator can cover a few modes well and drop the rest.
• Memorization. A generator can reproduce training images, so samples should be compared with their nearest training images. Radford et al. de-duplicated their largest training set to make memorization less likely (2016, Section 4.1.1).

Notebook#

The GANs notebook of the HSE-LAMBDA Deep Generative Models repository (pinned commit; cell numbers count every cell from 1) implements all three models. The notebook has no saved outputs; it documents procedures, not results.

• Cells 1–2: the minimax objective and the list of tricks the notebook uses: one-sided label smoothing, gradient clipping, dropout in the discriminator, a reduced latent variance at sampling time, a balanced update ratio and batch normalization in the generator.
• Cells 5–16: unconditional GAN on MNIST zeros, resized to $32\times32$ and scaled to $[-1, 1]$, batch size 128 with drop_last=True.
  • Cell 9: the networks. The latent is a spatial grid, 4 channels on a $4 \times 4$ grid, not a vector, and it is drawn inside the generator’s forward. Each UpsampleBlock doubles the resolution bilinearly and then applies twice a $3 \times 3$ convolution, batch norm and LeakyReLU(0.2). Three blocks take $4 \times 4$ to $32 \times 32$ with 32, 64 and 128 channels, and a $1 \times 1$ convolution with Tanh gives the image. Each DownsampleBlock applies a stride-2 $3 \times 3$ convolution and a $3 \times 3$ convolution, each followed by batch norm, LeakyReLU(0.2) and Dropout2d(0.3), and three blocks take $32 \times 32$ to $4 \times 4$. The discriminator is fully convolutional: a final $4 \times 4$ convolution turns the $4 \times 4 \times 128$ map into one logit. The docstrings speak of four blocks and of a global average, but the code has three blocks and no pooling. Against the DCGAN guidelines, the generator uses LeakyReLU instead of ReLU and resize-convolutions instead of transposed convolutions, the generator’s channel count grows rather than shrinks with the resolution, and the discriminator applies batch norm right after its first convolution.
  • Cell 10 sets the hyperparameters: learning rate $10^{-4}$ for both networks, 30 epochs, one discriminator step per generator step (update_ratio = 1), gradient-norm clipping at $1$ and one-sided label smoothing $0.9$.
  • Cell 11 gives the guidance on the update ratio summarized under balancing the two players.
  • Cell 12 builds the networks and the optimizers: Adam with the default betas for $G$ and betas $(0.5, 0.999)$ for $D$, each with a step schedule that halves the learning rate every 20 epochs.
  • Cell 13 is the training loop with binary_cross_entropy_with_logits. The discriminator loss is the mean of BCE on real images (target 0.9) and on detached fakes (target 0), which is $-V$ up to the factor $\tfrac12$ and the smoothing. The generator loss is the non-saturating BCE towards the real label. Here the smoothed target $0.9$ is also used for the generator, so the exact loss is $-\mathbb{E}_z\big[0.9\ln D(G(z)) + 0.1\ln(1 - D(G(z)))\big]$, minimized at $D(G(z)) = 0.9$ rather than $1$. The schedulers are stepped once per epoch. Every 5 epochs the loop saves a checkpoint into weights/, as does the conditional loop of cell 23; the notebook never creates that directory, so torch.save fails unless it already exists.
  • Cells 14–16 plot the losses, sample a grid and list common tricks: one-sided label smoothing, random label flipping, batch normalization in both networks, “avoiding sparse gradients” with the log-sigmoid (the logit form of the losses), and mode collapse with minibatch discrimination, a history of old generators and WGAN as remedies. Cell 15 multiplies the latent by $0.7$ when sampling, so the grid shows the generator applied to a reduced-variance prior rather than samples from $p_g$ itself, the truncation-like prior narrowing; cells 25 and 33 do the same for the other two models.
• Cells 17–26: conditional GAN on CIFAR-100. Cell 19 sets batch size 32, learning rate $2\cdot10^{-4}$ and Adam betas $(0.5, 0.999)$ for both networks, no learning-rate schedule, 30 epochs, smoothing $0.9$ (again also in the generator’s target) and gradient clipping at $1$. In cell 22 the generator embeds the label with Embedding(100, 16), reshaped into one $4 \times 4$ channel next to the 4 noise channels, and the discriminator embeds it with Embedding(100, 1024), reshaped into one $32 \times 32$ channel next to the RGB image. The blocks are those of cell 9, with dropout 0.3 in the discriminator. Cell 23 alternates one discriminator and one generator step. Cell 26 lists what the condition buys: control over the class, samples focused on class-specific features, and variety across classes.
• Cells 27–33: f-GAN. Activation_g and Conjugate_f (cell 29) implement $g_f$ and $f^*$ from the table for "KLD", "RKL", "CHI", "SQH", "JSD" and "GAN", in the same convention: the Pearson conjugate is $t^2/4 + t$. VLoss is the mean of $g_f(v)$ and QLoss the mean of $-f^*(g_f(v))$. Cell 31 selects JS and reuses the unconditional networks: learning rate $10^{-4}$, Adam with default betas for $G$ and $(0.5, 0.999)$ for $D$, 100 epochs, no clipping, smoothing or schedule. In cell 32 the generator maximizes $T$ on generated samples rather than minimizing the fake term of the bound, the non-saturating surrogate of the generator’s step; the direct form is left as a commented alternative. Two implementation notes. The activation and the conjugate are evaluated one after the other with plain exp and log. In 32-bit floating point the JS and GAN conjugates return $+\infty$ once a fake’s output $v$ exceeds about 17, because $2 - e^{t}$ or $1 - e^{t}$ rounds to $0$, and the GAN activation returns $-\infty$ for $v$ below about $-89$, where exp(-v) overflows. The composed forms $\operatorname{softplus}(v) - \ln 2$ and $\operatorname{softplus}(v)$ avoid both (compose before computing). And loss_real.backward(retain_graph=True) keeps a graph that is never used again.
• Cells 34–36: the choice of latent size (a 100-dimensional vector or a small $4 \times 4$ grid, doubled by each upsampling layer), transposed convolutions against upsampling followed by a convolution, a comparison of variants (the original GAN, WGAN, LSGAN, f-GAN and the conditional GAN), the key challenges with their remedies (gradient penalties, unrolled GANs and several discriminators against mode collapse; label smoothing, balanced learning rates and spectral normalization for stability; the Inception Score, FID and KID for evaluation), practical tips and a list of pros and cons. The body covers them under training GANs in practice, least-squares GAN and strengths and weaknesses; KID is covered with FID.

Next#

The disjoint-support argument showed that JS gives no gradient whenever the supports are disjoint, even when the generator’s samples are close to the data. The next note replaces it with a distance that shrinks continuously as the two distributions approach each other: the Wasserstein distance.

References#

• I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio. Generative adversarial nets. NeurIPS, 2014. Preprint arXiv:1406.2661.
• M. Arjovsky, L. Bottou. Towards principled methods for training generative adversarial networks. ICLR, 2017.
• T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, X. Chen. Improved techniques for training GANs. NeurIPS, 2016.
• M. Mirza, S. Osindero. Conditional generative adversarial nets. arXiv:1411.1784, 2014.
• S. Nowozin, B. Cseke, R. Tomioka. f-GAN: Training generative neural samplers using variational divergence minimization. NeurIPS, 2016.
• X. Nguyen, M. J. Wainwright, M. I. Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56(11), 2010.
• I. Goodfellow. NIPS 2016 tutorial: Generative adversarial networks. arXiv:1701.00160, 2016.
• A. Radford, L. Metz, S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. ICLR, 2016. DCGAN; Section 3: architecture guidelines; Section 4: training details.
• X. Mao, Q. Li, H. Xie, R. Y. K. Lau, Z. Wang, S. P. Smolley. Least squares generative adversarial networks. ICCV, 2017. Section 3.2: losses, benefits and the Pearson χ² connection.
• L. Metz, B. Poole, D. Pfau, J. Sohl-Dickstein. Unrolled generative adversarial networks. ICLR, 2017.
• I. Durugkar, I. Gemp, S. Mahadevan. Generative multi-adversarial networks. ICLR, 2017.
• A. Shrivastava, T. Pfister, O. Tuzel, J. Susskind, W. Wang, R. Webb. Learning from simulated and unsupervised images through adversarial training. CVPR, 2017. Section 2.3: a history of generated images for the discriminator.
• M. Arjovsky, S. Chintala, L. Bottou. Wasserstein GAN. arXiv:1701.07875, 2017; ICML 2017.
• HSE-LAMBDA, Deep Generative Models course repository: GANs.ipynb.