Beyond CycleGAN: multimodal, contrastive and semantic image synthesis

The StarGAN note ended with one generator that gives many styles across many domains: StarGAN v2 keeps a cycle back to the input and draws the target appearance from a learned style space. Other routes lead away from CycleGAN as well. MUNIT (Huang et al., 2018) splits an image into a content code shared by two domains and a style code specific to each, and translates by swapping the style. CUT (Park et al., 2020) drops the cycle and the second generator altogether and ties each output patch to the input patch it came from with a contrastive loss. pix2pixHD and SPADE (Wang et al., 2018a; Park et al., 2019) start from a segmentation mask instead of a photo, and SPADE makes the mask survive through every layer. And super-resolution with GANs (Ledig et al., 2017; Wang et al., 2018b) is the paired task where the problem of the pixel loss is most visible: the image that minimizes the squared error is the blurry average of all plausible answers.

Notation. As in the CycleGAN note, $x$ is an input image, $G$ a generator, $D$ a discriminator and $\|a - b\|_1$ the sum of absolute differences (implementations average it). For MUNIT, the domains are $X_1$ and $X_2$ with images $x_1, x_2$; each domain $i$ has a content encoder $E^c_i$, a style encoder $E^s_i$ and a decoder $G_i$; $c$ is a content code and $s$ a style code with prior $q(s) = \mathcal N(0, I)$. For CUT, $\hat y = G(x)$ is the output and $G_{\mathrm{enc}}$ the first half of the generator. For semantic synthesis, $m$ is a segmentation mask that gives a class label to every pixel. For super-resolution, $x^{\mathrm{LR}}$ is the low-resolution input, $x^{\mathrm{HR}}$ the true high-resolution image and $\varphi$ a feature map of a pretrained VGG network.

MUNIT: one content space, a style space per domain#

Two autoencoders with a shared content code#

MUNIT assumes that an image is produced from two codes: a content code $c$, shared by both domains, and a style code $s_i$, specific to domain $i$ (Huang et al., 2018, Section 3.1). Content is “the underlying spatial structure” and style “the rendering of the structure”: the pose and layout of a cat against its fur and colouring. A pair of corresponding images would be $x_1 = G^*_1(c, s_1)$ and $x_2 = G^*_2(c, s_2)$ with the same $c$. Because $s_2$ is random, one $x_1$ has a whole distribution of translations, not one.

Each domain has an autoencoder whose code is split in two, $(c_i, s_i) = (E^c_i(x_i), E^s_i(x_i))$. Translation swaps decoders (Section 3.2):

$$x_{1 \to 2} = G_2\big(E^c_1(x_1),\, s_2\big), \qquad s_2 \sim \mathcal N(0, I).$$

A new $s_2$ gives a new output. Taking $s_2 = E^s_2(x_2)$ from a real image of the target domain instead gives example-guided translation, in which the output copies the style of the example (Section 5.5).

The two codes have different shapes, and the shapes do much of the separating (Huang et al., 2018, Appendix B.2):

• Content is a feature map. The content encoder downsamples twice and adds four residual blocks, so a $256 \times 256$ image becomes a $64 \times 64 \times 256$ tensor. All its layers use instance normalization, which removes the per-channel mean and spread of each feature map, the statistics that carry much of an image’s style.
• Style is a vector of 8 numbers. The style encoder ends in global average pooling and a linear layer, so it keeps no spatial layout, and it uses no instance normalization, which would remove exactly what it should keep.
• The decoder injects the style through AdaIN: an MLP turns $s$ into the scale and shift of every AdaIN layer in four residual blocks, and two upsampling blocks produce the image.

Losses in image and latent space#

MUNIT trains the encoders and decoders to be inverses in both directions (Huang et al., 2018, Eqs. 1–5):

$$\begin{aligned} \mathcal L^{x_1}_{\mathrm{recon}} &= \mathbb E_{x_1}\, \big\|G_1\big(E^c_1(x_1), E^s_1(x_1)\big) - x_1\big\|_1, \\ \mathcal L^{c_1}_{\mathrm{recon}} &= \mathbb E_{c_1, s_2}\, \big\|E^c_2\big(G_2(c_1, s_2)\big) - c_1\big\|_1, \\ \mathcal L^{s_2}_{\mathrm{recon}} &= \mathbb E_{c_1, s_2}\, \big\|E^s_2\big(G_2(c_1, s_2)\big) - s_2\big\|_1, \end{aligned}$$

with $c_1 = E^c_1(x_1)$ for a real $x_1$ and $s_2 \sim \mathcal N(0, I)$, plus the same three terms with the domains swapped, and an adversarial loss $\mathcal L^{x_2}_{\mathrm{GAN}}$ in which $D_2$ tells real images of $X_2$ from translations $G_2(c_1, s_2)$. The full objective is

$$\begin{aligned} \min_{E_1, E_2, G_1, G_2}\; \max_{D_1, D_2}\;\; & \mathcal L^{x_1}_{\mathrm{GAN}} + \mathcal L^{x_2}_{\mathrm{GAN}} + \lambda_x \big(\mathcal L^{x_1}_{\mathrm{recon}} + \mathcal L^{x_2}_{\mathrm{recon}}\big) \\ & + \lambda_c \big(\mathcal L^{c_1}_{\mathrm{recon}} + \mathcal L^{c_2}_{\mathrm{recon}}\big) + \lambda_s \big(\mathcal L^{s_1}_{\mathrm{recon}} + \mathcal L^{s_2}_{\mathrm{recon}}\big), \end{aligned}$$

with $\lambda_x = 10$ and $\lambda_c = \lambda_s = 1$ (Appendix B.1). Each term has a job:

• Image reconstruction makes each pair $(E_i, G_i)$ an autoencoder of its own domain. It uses only images of one domain, so it needs no pairs.
• Content reconstruction asks that the translation keep the content: the content encoder of the target domain must read $c_1$ back from $x_{1 \to 2}$. This is what ties the output to the input.
• Style reconstruction asks that the translation use the style: $s_2$ must be readable from the output. Without it, the decoder could ignore $s_2$, and noise fed to a generator is usually ignored (Section 1).
• The adversarial loss makes translations look like real images of the target domain. MUNIT uses the least-squares form and three discriminators at different image scales, those of pix2pixHD below (Section 5.1).

At the optimum the encoded style distribution matches the Gaussian prior and the content distributions of the two domains coincide (Proposition 2): the content space becomes domain-invariant without any loss that asks for it explicitly. For high-resolution datasets ($512 \times 512$ and above) MUNIT adds a domain-invariant perceptual loss, a distance between VGG features of the input and of its translation taken after instance normalization of the features, which removes the domain-specific feature statistics so that the input can serve as the reference (Section 5.1, Appendix C).

Why MUNIT has no cycle loss#

A cycle loss forces a translation to be deterministic (Huang et al., 2018, Proposition 5). Suppose translation is random in both directions and the cycle holds exactly: every output of $x_1$, translated back, returns $x_1$, and every output of $x_2$, translated back, returns $x_2$. Assume also that translations of $X_1$ land in $X_2$ and vice versa (matched marginals). Let $a \neq b$ be two possible translations of one input $x^*_1$. Translating $a$ back gives $x^*_1$, by the forward cycle. Translating $x^*_1$ forward again must return $a$ with certainty, by the backward cycle from $a$. But $x^*_1$ can also produce $b$. So $a = b$: the translation has a single output. This is the formal version of the limit at the end of the CycleGAN note.

MUNIT keeps a weaker cycle that carries the style along (Proposition 4). Translate $x_1$ with a random $s_2$, then translate back with the style of the original, $E^s_1(x_1)$: the result is $x_1$. This style-augmented cycle holds at the optimum of the three reconstruction losses, and an explicit version of it can be added for some datasets (Eq. 6). StarGAN v2’s cycle, which returns with the input’s own style $E_y(x)$, is the same idea.

Results and the ablation#

On edges → shoes, diversity is the average LPIPS distance between pairs of outputs for the same input (100 inputs, 19 pairs each), and quality the share of human raters who prefer a method’s output to MUNIT’s (Huang et al., 2018, Table 1):

Method	Quality	Diversity
CycleGAN	36.0%	0.010
MUNIT without $\mathcal L^{x}_{\mathrm{recon}}$	6.0%	0.213
MUNIT without $\mathcal L^{c}_{\mathrm{recon}}$	20.7%	0.172
MUNIT without $\mathcal L^{s}_{\mathrm{recon}}$	28.6%	0.070
MUNIT	50.0%	0.109
BicycleGAN, trained on pairs	56.7%	0.104
real shoes	–	0.293

CycleGAN with dropout at test time barely varies its outputs. Without style reconstruction, many outputs are nearly identical, a partial collapse. The two most diverse rows are the worst: without image reconstruction the outputs are the most varied and are preferred in only 6.0% of comparisons, because artifacts make outputs differ and a diversity score counts them. The full model comes close to BicycleGAN, which needs paired data and is preferred to MUNIT in 56.7% of comparisons.

MUNIT and StarGAN v2#

StarGAN v2 (Choi et al., 2020) came two years later and compares itself with MUNIT under the protocol described in How translation models are measured. The two share the style machinery and differ in what they build around it:

	MUNIT	StarGAN v2
Domains per model	two, one autoencoder each	many, one generator
Content	an explicit code, $64 \times 64 \times 256$, reconstructed by $\mathcal L^{c}_{\mathrm{recon}}$	no separate code: $G$ encodes $x$ internally
Style	8 numbers drawn directly from $\mathcal N(0, I)$	64 numbers from a mapping network $F_y(z)$, a learned transform of a Gaussian per domain
Style injection	AdaIN, parameters from an MLP	AdaIN, an affine map per layer
Tie to the input	content reconstruction; image reconstruction within a domain	cycle with the input’s own style
Diversity	style reconstruction only	style reconstruction plus a diversity term
Discriminators	one per domain, three scales, least squares	one head per domain, non-saturating with R1

The comparison in the StarGAN note shows what StarGAN v2’s extras buy: configuration D of its ablation, noise fed to the generator with a latent reconstruction, already gives diverse outputs, and the mapping network and the diversity term lower FID further.

CUT: replacing the cycle with a contrastive loss#

Why drop the cycle?#

The cycle loss assumes that the two domains are in bijection, and many are not: a photo holds more information than its label map, and a horse can become many zebras (Park et al., 2020, Section 2). It also costs a second generator and a second discriminator that serve only to check the first. CUT keeps one generator, $\hat y = G(x)$, one discriminator, and replaces the cycle by a different way of saying that the output depends on its input: a patch of the output should tell us which patch of the input it came from. A zebra’s forehead should be traceable to the horse’s forehead, not to its legs or to the grass.

In information terms, the goal is to make the mutual information between corresponding input and output patches large. Since $I(X; \hat Y) = H(X) - H(X \mid \hat Y)$ and $H(X)$ does not depend on $G$, this is the same as making the input easy to infer from the output. Park et al. point out, citing earlier work, that the cycle loss bounds the conditional entropy $H(X \mid \hat Y)$ from above, so CycleGAN pursues the same goal through a second network (Section 3). CUT pursues it directly, with a contrastive estimate.

InfoNCE: pick the positive among negatives#

InfoNCE (van den Oord, Li and Vinyals, 2018, Section 2.3) turns “is $v^+$ related to $v$?” into a classification. A query $v$, its positive $v^+$ and $N$ negatives $v^-_1, \dots, v^-_N$ are mapped to unit vectors in $\mathbb R^K$, and a softmax over their dot products must pick the positive (Park et al., 2020, Eq. 2):

$$\ell(v, v^+, v^-) = -\log \frac{\exp(v \cdot v^+ / \tau)}{\exp(v \cdot v^+ / \tau) + \sum_{n=1}^{N} \exp(v \cdot v^-_n / \tau)},$$

a cross-entropy for an $(N + 1)$-way classification with temperature $\tau = 0.07$. Normalizing to the unit sphere keeps the embedding from collapsing to a point or growing without bound. The loss is the multi-class relative of noise-contrastive estimation: NCE trains a logistic regression to tell data from noise, InfoNCE a softmax to pick the one true partner among $N + 1$ candidates. In both, the optimal score is a log density ratio; for InfoNCE, $\exp(\text{score}) \propto p(v^+ \mid v)/p(v^+)$ (van den Oord et al., Section 2.3). The same loss, with the other passages of a batch as negatives, trains the retrieval encoders of semantic search.

Minimizing $\ell$ maximizes a lower bound on mutual information: $I \ge \log(N + 1) - \mathbb E\,\ell$, which tightens as $N$ grows (van den Oord et al., Section 2.3, with $N + 1$ samples in all). A loss of $\log(N + 1)$ means that the positive is no easier to find than a random candidate.

Worked example (toy numbers). A query has cosine $0.5$ with its positive and $0.45$, $0.2$ and $-0.1$ with three negatives. With $\tau = 1$ the softmax gives the positive a probability of only $0.309$, a loss of $1.176$, close to the chance level $\log 4 = 1.386$. With $\tau = 0.07$ the logits are divided by $0.07$, the gap of $0.05$ between the positive and the nearest negative becomes $0.71$, and the positive gets $0.665$, a loss of $0.408$. A small temperature makes the loss sensitive to small differences of similarity, and the hardest negative, the one most similar to the query, dominates the gradient.

PatchNCE: corresponding patches, negatives from the same image#

CUT applies InfoNCE to patches at several scales (Park et al., 2020, Section 3). The encoder half of the generator, $G_{\mathrm{enc}}$, already computes a stack of feature maps, and each position of a feature map at layer $l$ describes a patch of the input; deeper layers describe larger patches. CUT takes $L = 5$ layers, from the RGB pixels through the two downsampling convolutions to the first and the fifth residual block, so the patches range from single pixels to regions about a hundred pixels wide (Appendix C.1). A small two-layer MLP $H_l$ per layer, with 256 units, maps each feature vector to a unit vector, as in SimCLR.

The same encoder embeds the input and the output: $z_l = H_l(G^l_{\mathrm{enc}}(x))$ and $\hat z_l = H_l(G^l_{\mathrm{enc}}(\hat y))$. At 256 random positions $s$ per layer, the output feature $\hat z^s_l$ is the query, the input feature at the same position $z^s_l$ the positive, and the input features at the other 255 positions $z^{S \setminus s}_l$ the negatives (Eq. 3):

$$\mathcal L_{\mathrm{PatchNCE}}(G, H, X) = \mathbb E_{x}\, \sum_{l=1}^{L}\, \sum_{s=1}^{S_l}\, \ell\big(\hat z^s_l,\; z^s_l,\; z^{S \setminus s}_l\big).$$

The negatives come from the same input image. A zebra leg in the output must be closer to the horse leg at that position than to the horse’s back, the sky or the grass of the same photo. Park et al. also tried negatives from other images, kept in a large MoCo-style memory bank, and found them worse, alone or added to the internal ones (Section 4.2). Their explanation: with internal negatives the encoder need not model differences between images that are irrelevant to the translation, such as white against brown horses. Using only the last encoder layer also failed. Unlike representation learning, which embeds a fixed image as one vector, synthesis needs a dense signal at every scale.

What stops the generator from cheating? If the output kept no trace of the input, $\hat z^s_l$ would be as close to a negative as to the positive, and the loss would sit at about $\log 256 = 5.545$ per query. The loss is low only when each output patch is recognizably the transformed input patch. The encoder is free to learn what “corresponding” means across domains: it should match shapes and parts and ignore textures, the very things the translation changes. A fixed distance, such as an $L_1$ or a VGG loss between $x$ and $\hat y$, would penalize the stripes themselves.

The objective, and FastCUT#

The generator and the MLPs minimize (Park et al., 2020, Eq. 5)

$$\mathcal L_{\mathrm{GAN}}(G, D, X, Y) + \lambda_X\, \mathcal L_{\mathrm{PatchNCE}}(G, H, X) + \lambda_Y\, \mathcal L_{\mathrm{PatchNCE}}(G, H, Y),$$

and the discriminator plays the adversarial game. The last term feeds real images $y$ of the target domain through $G$ and asks that $G(y)$ correspond patch by patch to $y$: a learned, contrastive version of CycleGAN’s identity loss. Two settings are named:

• CUT: $\lambda_X = \lambda_Y = 1$, trained for 400 epochs.
• FastCUT: $\lambda_X = 10$, $\lambda_Y = 0$, trained for 200 epochs, with flip-equivariance augmentation: the input is flipped horizontally and the output features are flipped back before the loss.

Everything else follows CycleGAN: the ResNet generator with nine residual blocks, the PatchGAN discriminator, the least-squares GAN loss and a batch size of 1 (Appendix C.1). The identity term mainly stabilizes training. Without it, horse → zebra reached a better FID, but Cityscapes training could collapse (Section 4.2, Figure 6).

Results against CycleGAN#

Against CycleGAN, MUNIT and one-sided baselines, at $256 \times 256$ (Park et al., 2020, Table 1):

Method	Horse → zebra FID	Cat → dog FID	Cityscapes FID	Cityscapes mAP	s/iter	Memory, GB
CycleGAN	77.2	85.9	76.3	20.4	0.40	4.81
MUNIT	133.8	104.4	91.4	16.9	0.39	3.84
CUT	45.5	76.2	56.4	24.7	0.24	3.33
FastCUT	73.4	94.0	68.8	19.1	0.15	2.25

The Cityscapes mAP is measured by a segmentation network run on photos generated from label maps, so it checks that the output keeps the input’s layout. CUT needs 40% less time per iteration than CycleGAN and 31% less memory, FastCUT 63% less time and 53% less memory. CycleGAN, like FastCUT, trains for 200 epochs and CUT for 400, so a full CUT run is not shorter than a CycleGAN run. Without the cycle, CUT can also change more: on horse → zebra, zebras fill 36.8% of the pixels of the target training photos and horses 17.9% of the source photos. CUT enlarges the animals to 30.8% of the pixels, CycleGAN keeps them at 19.1%, and FastCUT, with its heavier contrastive weight, stays close to CycleGAN at 25.9% (Appendix C.4). Matching the target statistics this way helps for cat → dog, which needs changes of shape, and hurts when it paints stripes on the background.

Because the negatives come from within one image, CUT can even be trained on a single image per domain, for example a Monet painting and one photograph, using random $128 \times 128$ crops (Section 4.3).

Semantic image synthesis from a label map#

Semantic image synthesis turns a segmentation mask $m$, a class label for every pixel (sky, road, car, tree), into a photo. The training data are pairs of a photo and its mask, as for pix2pix, so the problem is supervised. Two things are new. Masks of street scenes come at $2048 \times 1024$, far beyond the $256 \times 256$ of pix2pix, and a mask is mostly large uniform regions, which, as we will see, the normalization layers of a standard generator erase.

pix2pixHD: multi-scale discriminators and feature matching#

pix2pixHD (Wang et al., 2018a, Section 3.2) scales pix2pix up in three ways:

• A coarse-to-fine generator. A global generator $G_1$, a residual network in the style of the CycleGAN generator with four downsampling steps and nine residual blocks, works at $1024 \times 512$. A local enhancer $G_2$ works at $2048 \times 1024$ and adds the last feature map of $G_1$ to its own before its residual blocks. $G_1$ is trained first, then $G_2$, then both together.
• Multi-scale discriminators. To judge a $2048 \times 1024$ image, a discriminator needs a large receptive field: a deeper network or larger kernels, both costly and prone to overfit. pix2pixHD uses three discriminators $D_1, D_2, D_3$ with identical architecture ($70 \times 70$ PatchGANs) on the image downsampled by factors of 1, 2 and 4. The same $70 \times 70$ window covers a four times wider region of the coarsest image, so $D_3$ judges global structure and $D_1$ fine detail, and the generator plays against all three: $\min_G \max_{D_1, D_2, D_3} \sum_{k} \mathcal L_{\mathrm{GAN}}(G, D_k)$. Without them, the authors observed repeated patterns in the images.
• A feature-matching loss. The generator must also match the discriminators’ internal features on the real photo and on its synthesis (Eq. 4):

$$\mathcal L_{\mathrm{FM}}(G, D_k) = \mathbb E_{(m, x)} \sum_{i=1}^{T} \frac{1}{N_i}\, \big\|D^{(i)}_k(m, x) - D^{(i)}_k\big(m, G(m)\big)\big\|_1 ,$$

where $D^{(i)}_k$ is the output of layer $i$ of discriminator $k$, with $N_i$ elements, and the discriminator receives the mask concatenated with the image. The full objective is $\min_G \big(\max_{D_1, D_2, D_3} \sum_k \mathcal L_{\mathrm{GAN}}(G, D_k) + \lambda \sum_k \mathcal L_{\mathrm{FM}}(G, D_k)\big)$ with $\lambda = 10$ and least-squares GAN losses; the discriminators only extract features for $\mathcal L_{\mathrm{FM}}$ and do not maximize it (Section 3.2; the weight in Section 4). The loss is a perceptual loss whose features are learned by the discriminators on this very data, at several depths and three scales. It needs the pairs: it compares the synthesis with the photo that belongs to the mask. A VGG perceptual loss with the same weight is optional and helps slightly.

On Cityscapes, a segmentation network recovers the input mask from pix2pixHD’s photos with a pixel accuracy of 83.78% and a mean IoU of 0.6389, against 78.34% and 0.3948 for pix2pix and 84.29% and 0.6857 on the real photos (Table 1). In the authors’ human study, adding the feature-matching loss to the GAN loss was the larger improvement and the VGG loss the smaller one, and 69.2% of raters preferred results with multi-scale discriminators to those with a single one (Section 4).

How normalization washes out the label map#

pix2pixHD feeds the mask, one-hot per class, to the first layer of its generator, and the generator follows every convolution with instance normalization. Normalization subtracts each channel’s mean over the positions and divides by its spread. A mask is piecewise constant, so after one convolution most of its information sits in exactly those statistics (Park et al., 2019, Section 3).

A uniform mask is erased. Let every pixel have the same label. Away from the borders, a convolution of a constant input is a constant: channel $c$ holds one value $a_c$ at every position, and with reflection padding at the borders too. Instance normalization divides by $\sqrt{\sigma^2 + \varepsilon}$ with a small $\varepsilon > 0$, here $\sigma = 0$, and turns the channel into $(a_c - a_c)/\sqrt{0 + \varepsilon} = 0$, whatever the label was. An all-grass mask and an all-sky mask produce the same activations, and the generator cannot know which to draw (Figure 3 of the paper shows pix2pixHD giving two identical outputs). This is the effect that made the StarGAN generator pad with zeros.

Worked example (toy numbers). A mask is sky on a fraction $p$ of the pixels and grass on the rest. After a $1 \times 1$ convolution, a channel holds $a_{\mathrm{sky}}$ on the sky and $a_{\mathrm{grass}}$ on the grass. Its mean is $p\,a_{\mathrm{sky}} + (1 - p)\,a_{\mathrm{grass}}$ and its standard deviation $\sqrt{p(1 - p)}\,|a_{\mathrm{sky}} - a_{\mathrm{grass}}|$, so instance normalization gives

$$\text{sky: } \pm\sqrt{\tfrac{1 - p}{p}}, \qquad \text{grass: } \mp\sqrt{\tfrac{p}{1 - p}},$$

with the upper signs when $a_{\mathrm{sky}} > a_{\mathrm{grass}}$. For $p = \tfrac14$ the values are $1.732$ and $-0.577$, for weights $(2, -1)$, $(0.3, 0.1)$ or $(5, 4)$ alike. Each channel keeps the layout, where the boundary is and which side is larger, and loses the size of the difference. A sea-and-sand mask with the same layout gives the same numbers in every channel where the order of the two values agrees. The more uniform the mask, the less survives, and nothing does for a single region.

Concatenating the mask to the input of every layer does not cure this, since each concatenated copy is again convolved and normalized. In the SPADE ablation on COCO-Stuff, the mean IoU of the segmentation check is 31.9 for a decoder with the mask concatenated at every layer and 35.2 for the same decoder with SPADE (Table 3).

SPADE: per-pixel scale and shift from the mask#

SPADE (SPatially-Adaptive DEnormalization; Park et al., 2019, Section 3) lets the mask in after the normalization, where nothing can remove it. The activations $h$ of a layer are normalized without learned parameters, with the per-channel mean $\mu_c$ and standard deviation $\sigma_c$ over the batch and all positions, as in batch normalization, and then scaled and shifted by maps computed from the mask (Eq. 1):

$$\mathrm{SPADE}(h, m)_{n, c, y, x} = \gamma_{c, y, x}(m)\, \frac{h_{n, c, y, x} - \mu_c}{\sigma_c} + \beta_{c, y, x}(m).$$

Unlike batch norm, whose $\gamma_c$ and $\beta_c$ are one learned number per channel, and unlike AdaIN, whose $\gamma_c(s)$ and $\beta_c(s)$ depend on a style vector but not on the position, SPADE’s scale and shift are tensors with spatial dimensions: a different value at every pixel, determined by the labels around it. They are computed by a small convolutional network: the mask, resized to the resolution of the layer by nearest-neighbour sampling, passes through a $3 \times 3$ convolution with 128 channels and a ReLU, and two further $3 \times 3$ convolutions output $\gamma$ and $\beta$ (Appendix A, Figure 10). Replacing the mask by a class label and making $\gamma, \beta$ constant over positions gives conditional batch normalization; replacing it by an image, making them constant and using a batch of one gives AdaIN (Section 3).

Figure

SPADE: the label map sets a scale and a shift at every position

1. Resize. The mask is downsampled to the resolution of the layer.
2. Modulation maps. A shared convolution with ReLU and two output convolutions turn it into $\gamma(x, y)$ and $\beta(x, y)$.
3. Normalize. The activations of each channel are standardized with their batch statistics.
4. Denormalize. The result is multiplied by $\gamma$ and shifted by $\beta$ element by element. In the toy figure, $h = 5$ in the sky becomes $\hat h = (5 - 2.25)/1.392 = 1.98$ and then $1.5 \cdot 1.98 + 0.8 = 3.76$; a uniform mask would now give a uniform but label-specific $\gamma$ and $\beta$, not zeros.

The generator. With the mask entering every normalization layer, the generator needs no encoder for it. It starts from a 256-dimensional random vector, maps it to a $4 \times 4 \times 1024$ tensor and applies seven residual blocks with SPADE in place of each normalization, separated by $2\times$ upsampling (Appendix A, Figures 11 and 12). The random vector gives multimodal synthesis. Trained together with an image encoder that outputs a mean and a variance, as in a VAE with a KL term of weight 0.05, the vector carries the style of a reference photo: one mask can be rendered in the style of any example (Section 4, Appendix A).

Training. The discriminator is pix2pixHD’s multi-scale PatchGAN with spectral normalization on all layers, and the loss is pix2pixHD’s, feature matching and VGG terms included, with the least-squares adversarial term replaced by the hinge loss. Spectral normalization is also applied to the generator. The learning rates are $10^{-4}$ for the generator and $4 \cdot 10^{-4}$ for the discriminator, with Adam and $\beta_1 = 0$, $\beta_2 = 0.999$ (Section 4).

Results. On COCO-Stuff (182 classes), the mean IoU of the segmentation check is 37.4 against 14.6 for pix2pixHD and 23.7 for the cascaded refinement network (CRN), and the FID 22.6 against 111.5 and 70.4. On ADE20K the mean IoU is 38.5 against 20.3 for pix2pixHD, and the FID 33.9 against 81.8. On Cityscapes the gap is small, 62.3 against 58.3 (Table 1). In the ablation on COCO-Stuff (Table 3), the encoder–decoder baseline pix2pixHD++ rises from 32.7 to 34.4 when its normalization layers become SPADE layers, and the SPADE generator described above, a decoder without an encoder, reaches 35.2. That generator is the one of Table 1, which lists 37.4; Section 4 quotes 35.2 for it, and the paper does not explain the difference. Which parameter-free normalization SPADE wraps matters less: in Table 4 the same generator scores 33.9 with instance normalization and 35.0 with synchronized batch normalization, its default, although the other rows of that table give the default configuration 35.2. What matters is where the mask enters.

Super-resolution: why the average looks blurry#

The MSE optimum is the conditional mean#

Single-image super-resolution recovers a high-resolution image from a downsampled one, usually $4\times$ along each side, so that 16 pixels must be produced for each pixel of the input. Training pairs are free: downsample any photo. The difficulty is the loss. A low-resolution image is consistent with many high-resolution ones, which differ in exactly the fine texture that downsampling removed.

The output that minimizes the expected squared error for a given input is the conditional mean, $\hat y = \mathbb E[x^{\mathrm{HR}} \mid x^{\mathrm{LR}}]$, and its expected error is the total conditional variance, $\operatorname{tr} \operatorname{Cov}(x^{\mathrm{HR}} \mid x^{\mathrm{LR}})$. PSNR, the standard score of super-resolution, is $10 \log_{10}(\text{peak}^2/\mathrm{MSE})$, a decreasing function of the MSE, so the PSNR-optimal output is the same mean. When the plausible answers differ in texture, their mean is a smooth image that matches none of them. This is the blur that the CycleGAN note met in pix2pix, and the reason a pixel distance rewards blur.

A perfect generator cannot fix the score. Let $x'$ be a sample from the true conditional distribution, drawn independently of the true image. Writing $\mu$ for the conditional mean,

$$\mathbb E\,\|x^{\mathrm{HR}} - x'\|^2 = \mathbb E\,\|(x^{\mathrm{HR}} - \mu) - (x' - \mu)\|^2 = 2 \operatorname{tr} \operatorname{Cov}(x^{\mathrm{HR}} \mid x^{\mathrm{LR}}),$$

because the two deviations are independent with mean zero, so the cross term vanishes. An ideal sampler has exactly twice the MSE of the mean, $10 \log_{10} 2 = 3.01$ dB less PSNR, on every dataset. A method that produces realistic texture must lose PSNR.

Figure

Why the MSE optimum is blurry and a sample is sharp

In the toy example, averaging $4 \times 4$ blocks erases a stripe pattern of period 4 entirely, so the four shifts of the stripes give the same input. The MSE optimum is a flat grey with MSE 0.25 (PSNR 6.02 dB); a sample is a sharp stripe pattern with expected MSE 0.50 (PSNR 3.01 dB). The $L_1$ loss does no better here: at each pixel half of the candidates are 0 and half are 1, so every value between 0 and 1 is a per-pixel median, the flat 0.5 included.

SRGAN: a VGG content loss and an adversarial loss#

SRGAN (Ledig et al., 2017, Section 2) replaces the squared error by a perceptual loss with two parts:

$$l^{\mathrm{SR}} = l^{\mathrm{SR}}_{\mathrm{VGG}/5.4} + 10^{-3}\, l^{\mathrm{SR}}_{\mathrm{Gen}} .$$

• The content loss is a squared distance between VGG19 features of the output and of the true image, $l^{\mathrm{SR}}_{\mathrm{VGG}/i.j} = \frac{1}{W_{i,j} H_{i,j}} \sum_{x, y} \big(\varphi_{i,j}(x^{\mathrm{HR}})_{x,y} - \varphi_{i,j}(G(x^{\mathrm{LR}}))_{x,y}\big)^2$, where $\varphi_{i,j}$ is the feature map after the activation of the $j$-th convolution before the $i$-th max-pooling (Eq. 5). The choice $\varphi_{5,4}$ is deep in the network, where features describe content and are less sensitive to exact pixel values than the image itself, following Johnson, Alahi and Fei-Fei (2016). The features are rescaled by $1/12.75$, which multiplies the loss by about 0.006 and brings it to the scale of the pixel MSE (Section 3.2).
• The adversarial loss is the non-saturating $-\log D(G(x^{\mathrm{LR}}))$, summed over the training images (Eq. 6). The content loss keeps the output close to the true image in feature space; the discriminator pushes it towards the set of natural images, where the average of several textures does not lie.

The generator, SRResNet, has 16 residual blocks of two $3 \times 3$ convolutions with 64 channels, batch normalization and PReLU, and two sub-pixel convolutions that each double the resolution. The discriminator follows the DCGAN guidelines: eight $3 \times 3$ convolutions from 64 to 512 channels with LeakyReLU (slope 0.2), strided convolutions instead of pooling, two dense layers and a sigmoid. Training uses 350,000 ImageNet images with $96 \times 96$ crops and batches of 16. The GAN starts from the generator trained with the MSE alone, to avoid poor local optima, and runs $10^5$ iterations at learning rate $10^{-4}$ and $10^5$ at $10^{-5}$ (Section 3.2).

On BSD100 at $4\times$, 26 raters scored images from 1 (bad) to 5 (excellent) (Table 2):

	bicubic	SRResNet (MSE)	SRGAN	original
PSNR, dB	25.94	27.58	25.16	∞
mean opinion score	1.47	2.29	3.56	4.46

SRGAN has the lowest PSNR, 2.42 dB below SRResNet and below even bicubic interpolation, and by far the best opinion score. The loss against SRResNet, 2.42 dB, is close to the $3.01$ dB that the argument predicts for an ideal sampler against the true mean. The paper concludes that PSNR and SSIM fail to measure quality as people judge it (Section 4); LPIPS was later fitted to human judgements to fill that gap.

ESRGAN: RRDB, a relativistic discriminator and features before activation#

ESRGAN (Wang et al., 2018b, Section 3) keeps SRGAN’s design and improves each of its three parts:

• The generator: RRDB without batch normalization. Batch normalization uses batch statistics in training and running averages at test time; when the two differ, it produced artifacts, more often in deep networks trained as GANs. ESRGAN removes it and replaces the residual block by the Residual-in-Residual Dense Block: dense blocks, in which each convolution receives the outputs of all previous ones, nested inside two levels of residual connections. Each residual branch is multiplied by a constant $\beta$ between 0 and 1 before it is added (residual scaling), and the weights are initialized with a smaller variance; both stabilize a very deep network. The large model has 23 RRDBs.
• The discriminator: relativistic average. SRGAN’s discriminator estimates the probability that an image is real, $D(x) = \sigma(C(x))$, with $C$ the raw output. The relativistic average discriminator of Jolicoeur-Martineau (2018) estimates the probability that a real image is more realistic than the fakes on average:

$$D_{\mathrm{Ra}}(x_r, x_f) = \sigma\big(C(x_r) - \mathbb E_{x_f}[C(x_f)]\big),$$

with the mean taken over the fakes of the mini-batch. The losses are symmetric (Eqs. 1–2):

$$\begin{aligned} \mathcal L^{\mathrm{Ra}}_D &= -\mathbb E_{x_r}\big[\log D_{\mathrm{Ra}}(x_r, x_f)\big] - \mathbb E_{x_f}\big[\log\big(1 - D_{\mathrm{Ra}}(x_f, x_r)\big)\big], \\ \mathcal L^{\mathrm{Ra}}_G &= -\mathbb E_{x_r}\big[\log\big(1 - D_{\mathrm{Ra}}(x_r, x_f)\big)\big] - \mathbb E_{x_f}\big[\log D_{\mathrm{Ra}}(x_f, x_r)\big], \end{aligned}$$

where $D_{\mathrm{Ra}}(x_f, x_r)$ compares a fake with the mean score of the real images. The generator’s loss now contains real images too, and its gradient flows through the term $\mathbb E_{x_f}[C(x_f)]$ inside $D_{\mathrm{Ra}}(x_r, x_f)$: as the fakes improve, the reals look less real by comparison, even though $C(x_r)$ has not changed. The authors credit this with sharper edges and more detailed textures.

• The perceptual loss: features before activation. After a ReLU deep in VGG19 most features are zero: for one test image, only 11.17% of the units at $\varphi_{5,4}$ are active. A loss on such sparse features gives weak supervision, and it also let the brightness of the output drift from the true image. ESRGAN takes the features before the activation.

The generator minimizes $\mathcal L_G = \mathcal L_{\mathrm{percep}} + \lambda\, \mathcal L^{\mathrm{Ra}}_G + \eta\, \mathcal L_1$ with $\lambda = 5 \cdot 10^{-3}$ and $\eta = 10^{-2}$, where $\mathcal L_1$ is the pixel distance to the true image (Eq. 3). It starts from a network trained with $\mathcal L_1$ alone, so that the discriminator sees reasonable images from the start (Section 4.1). Training uses DIV2K, Flickr2K and the OutdoorSceneTraining set, with $128 \times 128$ crops. Finally, network interpolation blends the two generators’ weights, $\theta = (1 - \alpha)\,\theta_{\mathrm{PSNR}} + \alpha\,\theta_{\mathrm{GAN}}$, which trades PSNR for texture continuously at test time without retraining and gave cleaner results than blending the output images (Section 3.4). A variant of ESRGAN won region 3 of the PIRM 2018 super-resolution challenge, the region where the error constraint was loosest and the perceptual index decided.

What changed#

The translation models of this and the two previous notes differ mostly in what keeps the output faithful to the input:

Method	Input → output	What ties the output to the input	Data	Outputs per input
pix2pix	image → image	$L_1$ to the true target; conditional $D(x, y)$	paired	one
CycleGAN	image → image, two domains	cycle loss through a second generator	unpaired	one
StarGAN	image + target label → image	cycle with the original label; domain classifier	unpaired, labelled domains	one per label
StarGAN v2	image + style code → image	cycle with the input’s style; style reconstruction	unpaired, one domain per image	many
MUNIT	image + style code → image, two domains	content and style reconstruction; image reconstruction within a domain	unpaired	many
CUT	image → image	PatchNCE between corresponding input and output patches	unpaired	one
pix2pixHD	label map → photo	conditional multi-scale $D$; feature matching to the paired photo	paired	one
SPADE	label map (+ style vector) → photo	the mask modulates every normalization layer; pix2pixHD losses	paired	many, with the image encoder
SRGAN	low-resolution → high-resolution image	VGG feature distance to the true image	paired (made by downsampling)	one

Limitations#

• MUNIT works on two domains, with a global style. One model per pair of domains, as for CycleGAN. The style is a single vector of 8 numbers for the whole image, so it cannot vary locally, and what counts as content or style is set by the architecture, a spatial map against a pooled vector, rather than by meaning.
• CUT gives one output per input. Its generator is deterministic. Its freedom to match the target statistics can make unwanted changes, such as zebra texture on the background, and an unfamiliar pose can make it texture the background instead of the animal (Figure 3 of the paper). Internal negatives also include patches that genuinely look alike, two patches of the same sky, which the loss nonetheless pushes apart.
• Semantic synthesis needs pixel-aligned masks. Annotation is expensive; for its Flickr landscapes SPADE used masks predicted by a segmentation network. The segmentation check measures layout, not realism, and the SIMS baseline on Cityscapes reached a better FID than SPADE by pasting real patches that do not fit the mask.
• Super-resolution GANs invent detail. The texture is plausible, not true: unsuitable where details carry evidence, as the SRGAN authors note for medical images and surveillance. PSNR drops by design, and the opinion scores, perceptual indices and learned distances that replace it each need a protocol of their own.
• Every number depends on the evaluation. FID changes with the number of samples, diversity scores count artifacts as diversity (MUNIT’s ablation shows it as clearly as StarGAN v2’s AFHQ results), and segmentation scores depend on the segmentation network.

Summary#

• MUNIT encodes an image into a spatial content code shared by two domains and an 8-dimensional style code per domain; it translates by decoding the content with a style drawn from $\mathcal N(0, I)$ or taken from an example, injected through AdaIN. Losses: image reconstruction, content and style reconstruction ($\lambda_x = 10$, $\lambda_c = \lambda_s = 1$) and a multi-scale least-squares GAN.
• An exact cycle loss with matched marginals forces a deterministic translation; MUNIT’s reconstruction losses imply only a style-augmented cycle, which is what StarGAN v2 uses.
• CUT is one-sided: a PatchNCE loss asks each output patch to pick its corresponding input patch among 255 other patches of the same input, at five layers of the generator’s encoder, with $\tau = 0.07$. InfoNCE is multi-class NCE and bounds mutual information from below. CUT beats CycleGAN on FID with less time and memory per iteration; FastCUT drops the identity term and needs less still.
• pix2pixHD reaches $2048 \times 1024$ with a coarse-to-fine generator, three discriminators at three scales and a discriminator feature-matching loss ($\lambda = 10$).
• Instance normalization after a convolution erases a uniform label map and keeps only the layout of a piecewise-constant one. SPADE normalizes the activations without parameters and then applies a per-pixel $\gamma(m)$ and $\beta(m)$ computed from the mask, so the labels enter after the normalization.
• The MSE-optimal super-resolution is the conditional mean, blurry when textures are uncertain; a perfect sampler has twice its MSE, 3.01 dB less PSNR. SRGAN trains on a VGG feature loss plus $10^{-3}$ times an adversarial loss; ESRGAN adds RRDBs without batch normalization, a relativistic average discriminator and features before activation.

Code#

A NumPy version of the InfoNCE and PatchNCE losses, checked on the worked example and on random features:

import numpy as np

def info_nce(q, pos, negs, tau=0.07):
    """One query: unit vectors q, pos [K] and negs [N, K]; returns the loss and p(positive)."""
    logits = np.r_[q @ pos, negs @ q] / tau
    logits -= logits.max()
    p = np.exp(logits) / np.exp(logits).sum()
    return -np.log(p[0]), p[0]

def patch_nce(f_out, f_in, tau=0.07):
    """f_out, f_in: [S, K] unit features at S sampled locations of the output and the input.
    Query s is f_out[s]; its positive is f_in[s]; the other S - 1 input locations are its negatives."""
    logits = f_out @ f_in.T / tau                  # [S, S]; the diagonal holds the positives
    logits -= logits.max(axis=1, keepdims=True)
    log_p = logits - np.log(np.exp(logits).sum(axis=1, keepdims=True))
    return -np.mean(np.diag(log_p))

unit = lambda a: a / np.linalg.norm(a, axis=-1, keepdims=True)

# the worked example: cosines 0.5 (positive) and 0.45, 0.2, -0.1 (negatives), in 2-D for readability
q = np.array([1.0, 0.0])
cosv = lambda c: np.array([c, np.sqrt(1 - c * c)])
pos, negs = cosv(0.5), np.stack([cosv(c) for c in (0.45, 0.2, -0.1)])
for tau in (1.0, 0.07):
    loss, p = info_nce(q, pos, negs, tau)
    print(f"tau={tau}: p(positive)={p:.3f}, loss={loss:.3f}")
# tau=1.0: p(positive)=0.309, loss=1.176
# tau=0.07: p(positive)=0.665, loss=0.408

# 256 locations with 256-dim features: a perfect copy, a noisy copy, unrelated features
rng = np.random.default_rng(0)
f_in = unit(rng.standard_normal((256, 256)))
for name, f_out in [("copy", f_in),
                    ("noisy copy", unit(f_in + 3.0 * unit(rng.standard_normal((256, 256))))),
                    ("unrelated", unit(rng.standard_normal((256, 256))))]:
    print(f"{name:>10}: PatchNCE = {patch_nce(f_out, f_in):.3f}")
print(f"log 256 = {np.log(256):.3f}")
#       copy: PatchNCE = 0.000
# noisy copy: PatchNCE = 1.725
#  unrelated: PatchNCE = 5.903
# log 256 = 5.545

The noisy copy has a cosine of about 0.31 with its positive. The positive is the most similar candidate for 255 of the 256 queries, yet the softmax gives it a probability of only about 0.2 on average, because 255 negatives with cosines near zero share the rest: the loss keeps pushing until the margin is large compared with $\tau$. Unrelated features score about $\log 256$ or slightly above: the positive is then no more similar to the query than any negative, and by chance some negatives are more similar.

Next#

Every model in this note was judged by numbers: FID for realism, LPIPS distances between outputs for diversity, segmentation scores for faithfulness to a label map, opinion scores for super-resolution. MUNIT’s ablation and the PSNR of SRGAN showed how easily such a number rewards the wrong thing. The next notes explain where these numbers come from and what they miss, starting with the Inception Score, which scores generated images with a pretrained classifier.

Where to see the images#

These models are best judged by their samples. Each line says where to find them; figure numbers refer to the linked arXiv version.

• MUNIT (Huang et al., 2018; arXiv v2): edges → shoes against the baselines in Figure 4; edges ↔ shoes and handbags in Figure 5, animals in Figure 6, street scenes in Figure 7, Yosemite summer ↔ winter in Figure 8; translation guided by an example style image in Figure 9; repository with the results video.
• CUT (Park et al., 2020; arXiv v3): horse → zebra, cat → dog and Cityscapes against the baselines in Figure 3, with randomly selected results in Figure 10; single-image painting → photo in Figure 9; project page with videos.
• pix2pixHD (Wang et al., 2018a; arXiv v2): $2048 \times 1024$ street scenes from label maps, with label and appearance editing, in Figure 1; edges → faces and cats in Figure 2; Cityscapes comparisons in Figures 9 and 10; ADE20K in Figure 11; project page with video and editing demos.
• SPADE (Park et al., 2019; arXiv v2): a label map combined with a style image in Figure 1; COCO-Stuff, ADE20K and Cityscapes against the baselines in Figures 5 and 6; Flickr landscapes in Figure 7; several outputs for one mask in Figure 9; project page.
• SRGAN (Ledig et al., 2017; arXiv v5): bicubic, an MSE-trained network and SRGAN side by side in Figure 2; the content-loss variants in Figure 6; Set5, Set14 and BSD100 results in Figures 11–15.
• ESRGAN (Wang et al., 2018b; arXiv v2): SRGAN against ESRGAN in Figures 1 and 7; the effect of each change in Figure 8; network interpolation in Figure 10 (main-text numbering; the supplementary restarts at 1); repository with result images.

References#

• X. Huang, M.-Y. Liu, S. Belongie, J. Kautz. Multimodal unsupervised image-to-image translation. ECCV, 2018. Section numbers refer to arXiv v2: the assumption and the model in Section 3, the losses in Eqs. 1–5, Propositions 2, 4 and 5 (Section 4 and Appendix A), the architecture and hyperparameters in Section 5.1 and Appendix B, the edges → shoes evaluation in Table 1.
• T. Park, A. A. Efros, R. Zhang, J.-Y. Zhu. Contrastive learning for unpaired image-to-image translation. ECCV, 2020. Section numbers refer to arXiv v3: InfoNCE and PatchNCE in Section 3 (Eqs. 2–5), the comparison in Table 1, the ablations in Section 4.2, training details and distribution matching in Appendices C.1 and C.4.
• A. van den Oord, Y. Li, O. Vinyals. Representation learning with contrastive predictive coding. arXiv:1807.03748, 2018. Section 2.3: the InfoNCE loss and the bound $I \ge \log N - \mathcal L_N$.
• T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, B. Catanzaro. High-resolution image synthesis and semantic manipulation with conditional GANs. CVPR, 2018 (cited as Wang et al., 2018a). Section 3.2: the coarse-to-fine generator, the multi-scale discriminators and the feature-matching loss (Eqs. 3–5); results in Section 4 and Table 1.
• T. Park, M.-Y. Liu, T.-C. Wang, J.-Y. Zhu. Semantic image synthesis with spatially-adaptive normalization. CVPR, 2019. Section 3: SPADE (Eq. 1) and why normalization washes out the mask; Section 4 and Tables 1, 3 and 4: results and ablations; Appendix A: the architectures and training details.
• C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, W. Shi. Photo-realistic single image super-resolution using a generative adversarial network. CVPR, 2017. Section 2: the networks and the perceptual loss (Eqs. 3–6); Section 3.2: training; Tables 1 and 2: PSNR and mean opinion scores.
• X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, C. C. Loy, Y. Qiao, X. Tang. ESRGAN: Enhanced super-resolution generative adversarial networks. ECCV Workshops, 2018 (cited as Wang et al., 2018b). Section 3: RRDB, the relativistic discriminator (Eqs. 1–2), the perceptual loss and the total loss (Eq. 3), network interpolation (Eq. 4); Section 4.1: training.
• A. Jolicoeur-Martineau. The relativistic discriminator: a key element missing from standard GAN. arXiv:1807.00734, 2018.
• J. Johnson, A. Alahi, L. Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. ECCV, 2016.
• Y. Choi, Y. Uh, J. Yoo, J.-W. Ha. StarGAN v2: Diverse image synthesis for multiple domains. CVPR, 2020. Section 2: the networks and losses, including the cycle with the input’s style.