Autoencoders

With the blocks of the previous note, convolutions, batch norm and LeakyReLU, we can build the first neural model of the course. It does not generate anything yet. An autoencoder is trained to reproduce its own input; training a network by back-propagation to pass its input through a narrow hidden layer and back already appears as the “encoding problem” of Rumelhart, Hinton and Williams (1986). The reconstruction is only the pretext. What we actually want is the representation: the compact code the network is forced to build on the way (Hinton and Salakhutdinov, 2006; Goodfellow, Bengio and Courville, 2016, chapter 14). This note trains a plain, a denoising and two sparse (KL and L1) convolutional autoencoders on MNIST, looks at what their codes capture and how to measure it, and ends with the gap that the variational autoencoder will close.

Notation. $x$ is an input image and $P$ the data distribution it comes from. The encoder $f_\varphi$, with parameters $\varphi$, maps $x$ to the latent code $z = f_\varphi(x)$. The decoder $g_\theta$, with parameters $\theta$, maps the code back to the reconstruction $\hat x = g_\theta(z)$. As in the VAE note, $\varphi$ belongs to the encoder and $\theta$ to the decoder. $\beta$ and $\lambda$ are the weights of the sparsity penalties below; they are unrelated to the noise schedule $\beta_t$ of the diffusion notes.

Architecture: downsample, bottleneck, upsample#

An encoder $f_\varphi$ maps an input $x$ to a latent code $z$, and a decoder $g_\theta$ maps the code back to input space:

$$z = f_\varphi(x), \qquad \hat{x} = g_\theta(z) = g_\theta\big(f_\varphi(x)\big).$$

The data path has a simple shape, from the input through the bottleneck $z$ to the reconstruction:

$$x \;\xrightarrow{\ \text{downsample}\ }\; z \;\xrightarrow{\ \text{upsample}\ }\; \hat{x}.$$

The bottleneck is smaller than the input, so the network cannot pass the pixels through unchanged. It has to keep what it needs to rebuild the input and drop the rest. This is the latent space idea: every input becomes a point in a lower-dimensional space that is learned from the data.

Input resolution: $28 \times 28 \to 64 \times 64$. The MNIST digits are resized to $64 \times 64$ (transforms.Resize((64, 64)), then ToTensor(), which already gives pixel values in $[0, 1]$). The larger image lets the model downsample step by step. The encoder halves the spatial size four times: a stride-2 convolution with kernel 3 and padding 1 maps $H$ to $\lfloor (H-1)/2 \rfloor + 1$ (output-size formula). The decoder doubles the size four times:

Side	Encoder, 4 halvings	Decoder, 4 doublings	Restores input?
$64$	$64 \to 32 \to 16 \to 8 \to 4$	$4 \to 8 \to 16 \to 32 \to 64$	yes
$28$	$28 \to 14 \to 7 \to 4 \to 2$	$2 \to 4 \to 8 \to 16 \to 32$	no ($32 \ne 28$)

With a side divisible by $2^4 = 16$, such as $64$, every stage halves exactly and the mirrored decoder returns to the input size. At $28 \times 28$ the same four stages would leave a $2 \times 2$ map and a $32 \times 32$ output.

Figure

The convolutional autoencoder, layer by layer

The model. Every encoder stage is a block of a $3\times 3$ convolution with stride 2, batch normalization and LeakyReLU with slope $0.2$ (all explained in the previous note). A last $3 \times 3$ convolution with stride 1 produces the code. Each decoder stage upsamples by a factor of 2 with bilinear interpolation and then applies a $3\times 3$ convolution, batch norm and LeakyReLU. A final convolution maps 16 channels to 1, followed by a sigmoid.

Tensor	Shape (channels × height × width)	Numbers
input $x$	$1 \times 64 \times 64$	$4096$
after encoder block 1	$16 \times 32 \times 32$	$16384$
after encoder block 2	$32 \times 16 \times 16$	$8192$
after encoder block 3	$32 \times 8 \times 8$	$2048$
after encoder block 4 and the last conv: code $z$	$32 \times 4 \times 4$	$512$
after decoder block 1	$32 \times 8 \times 8$	$2048$
after decoder block 2	$32 \times 16 \times 16$	$8192$
after decoder block 3	$32 \times 32 \times 32$	$32768$
after decoder block 4	$16 \times 64 \times 64$	$65536$
output $\hat x$	$1 \times 64 \times 64$	$4096$

Latent size. The bottleneck has shape $32 \times 4 \times 4$, so it holds $512$ numbers: the latent code is 512-dimensional when flattened. That is $4096 / 512 = 8$ times fewer numbers than the resized input, but only about $1.5$ times fewer than the original $28 \times 28 = 784$ pixels. The compression is mild; the point is the structure of the code, not its size.

How to read the encoder and decoder#

• The encoder works as a learned filter, not a fixed compression. Every downsampling step is a learned $3\times3$ convolution with stride 2. The filters extract features while the stride reduces the resolution, so the network itself decides what survives each halving, which a fixed resize could not do. The classic alternative is a stride-1 convolution followed by pooling, which halves the size by a fixed rule: max pooling keeps the largest value of each $2 \times 2$ window, average pooling their mean. In image classifiers, replacing max pooling by a convolution with stride 2 cost no accuracy in the experiments of Springenberg et al. (2015), and it makes the downsampling learned as well.
• The decoder is not an upscaler. It works as upsample + conv. The bilinear upsampling is a fixed operation with no parameters; the learned convolution after it fills in the detail.
• Compression comes from the architecture, not from an explicit squeeze. No layer is told to compress. The first encoder block even expands $4096$ input values to $16384$ activations. The code becomes small only because the shapes narrow towards the bottleneck.
• Layers learn features, not pixels. Early layers respond to local patterns such as edges and stroke fragments. Deeper layers see a larger part of the image through their receptive field and combine these into larger parts of a digit, so the channels encode features rather than individual pixel values.

Reconstruction loss#

The loss compares the output with the input. The standard choice is the mean squared error over all $n$ scalar components,

$$\mathrm{MSE}(\hat y, y) = \frac{1}{n}\sum_{i=1}^{n} \left(y_i - \hat{y}_i\right)^2,$$

averaged over the data:

$$\mathcal L_{\text{recon}}(\varphi, \theta) = \mathbb E_{x \sim P}\Big[\mathrm{MSE}\big(g_\theta(f_\varphi(x)),\ x\big)\Big].$$

For an autoencoder $y = x$ and $\hat y = \hat x$. PyTorch’s nn.MSELoss() averages over every pixel of the batch: $n = 64 \cdot 64 \cdot 64 = 262144$ for a batch of 64 images.

MSE or BCE. Pixel values are in $[0, 1]$, so binary cross-entropy is a natural alternative:

$$\begin{aligned} \mathrm{BCE}(\hat x, x) &= -\frac{1}{n}\sum_{i=1}^{n}\Big[x_i \log \hat x_i \\ &\qquad\quad + (1 - x_i)\log(1 - \hat x_i)\Big], \end{aligned}$$

averaged over pixels as nn.BCELoss does by default. It treats each pixel as the probability of being “on”, which suits nearly binary MNIST digits.

The two losses correspond to two decoder likelihoods. A Gaussian decoder with fixed variance $s^2$ on $D$ pixels has $-\log \mathcal N(x;\, \hat x,\, s^2 I) = \frac{1}{2s^2}\|x - \hat x\|^2 + \frac D2 \log(2\pi s^2)$, so MSE equals its negative log-likelihood up to a positive factor and an additive constant. BCE is the negative log-likelihood of a decoder that makes each pixel an independent Bernoulli variable with mean $\hat x_i$. Strictly, that holds only for binary pixels. The resized MNIST images have many grey values; for those, BCE is still minimized at $\hat x_i = x_i$, but it is no longer the negative log of a normalized density over $[0, 1]$ (Loaiza-Ganem and Cunningham, 2019). The VAE note builds on this reading of the loss as a likelihood.

With a sigmoid already applied in forward, BCE must be nn.BCELoss; nn.BCEWithLogitsLoss expects raw logits and applies the sigmoid itself, so combining it with a sigmoid output would apply the sigmoid twice. The numerically more stable option is to drop the sigmoid and use nn.BCEWithLogitsLoss on the logits.

The input range must suit both the output activation and the loss. MNIST is already in $[0, 1]$ after ToTensor(), so no extra normalization is needed here; with other data it usually is. A common choice for colour images, such as the $3 \times 32 \times 32$ images of CIFAR-10, is to map $[0, 1]$ to $[-1, 1]$ with $x \mapsto 2x - 1$, which puts the inputs in a range symmetric around zero, and to map outputs back with $x \mapsto (x + 1)/2$ for display. The decoder must then be able to output negative values (see the output activation below), and BCE no longer applies, since its targets must lie in $[0, 1]$; MSE does.

Emergent structure#

Nothing in the loss asks for an organized latent space, and no labels are used in training. Still, structure emerges. Projecting the 512-dimensional codes of the test set to 2-D with t-SNE (van der Maaten and Hinton, 2008) and colouring the points by digit shows clusters by digit that are more cleanly separated than before training. Most digits form one compact cluster; the 2s split into two groups and the 1s form an elongated band. The codes are organized by content, not by individual pixel values.

t-SNE draws neighbourhoods, not distances. It places the points so that each code’s nearest neighbours in the 512-dimensional space stay near it in the plane; the perplexity, 30 here, sets roughly how many neighbours count. The sizes of the clusters in the plot and the gaps between them carry little information (Wattenberg, Viégas and Johnson, 2016). What the plot is meant to show is whether the neighbours of a code are codes of the same digit, and even that is qualitative evidence: the embedding can tear or merge neighbourhoods (t-SNE and UMAP). To check it, measure in the original 512-dimensional space, for example the share of each code’s $k$ nearest neighbours that have its label, or the probe below.

Measuring a code with a classifier. A plot is qualitative. A number comes from a probe: freeze the encoder, compute the codes of a labelled set, train a simple classifier on them and report its accuracy on held-out images (Alain and Bengio, 2016, who use linear classifiers on the layers of a network). The classifier sees only the code, so its accuracy measures how much class information the code keeps in a form that is easy to read off. Because random convolutional features are already informative, the baseline is not chance but the same probe on the codes of an untrained encoder with the same architecture. Training has improved the representation only if the trained encoder clearly beats that baseline. The same codes can also stand in for Inception features in FID when the images are unlike ImageNet.

Two experiments probe the latent space directly:

• Interpolation. Encode a “7” and a “2” and decode $z_\alpha = (1 - \alpha) z_1 + \alpha z_2$ for 8 frames, $\alpha = 0, \tfrac17, \dots, 1$. The endpoints are clean digits, but the middle frames look like overlays of the two strokes rather than digits that morph into each other.
• Sampling. Fit an axis-aligned (diagonal) Gaussian per digit class to the codes of the training images, using the per-coordinate mean and standard deviation of $z$, so that the 512 coordinates are sampled independently. Draw one code per class and decode it. In the stored run six of the ten samples are recognisable digits, while the 2, 4, 5 and 8 are badly distorted. This is a single draw per class.

Both experiments show why a plain autoencoder is a representation learner but not yet a generative model. Nothing in training makes an arbitrary point of the latent space decode to a plausible image, and the codes do not follow any distribution we know how to sample from: a diagonal Gaussian ignores the correlations between coordinates, the class-wise sampling needed labels and per-class statistics, and the regions between clusters were never trained to decode to anything. Variational autoencoders close exactly this gap by making the code distribution part of the training objective.

Variants#

Denoising autoencoder#

A denoising autoencoder (Vincent et al., 2008) changes the task rather than the architecture:

1. Corrupt the input with a random corruption $q(\tilde x \mid x)$. The original paper sets a random fraction of the input components to zero; additive Gaussian noise is another common choice (Vincent et al., 2010).
2. Reconstruct the clean input from the corrupted one, $\hat x = g_\theta(f_\varphi(\tilde x))$, and compute the loss against $x$, not $\tilde x$.
3. The result is a robust representation: its features stay stable when part of the input is corrupted.

The network cannot succeed by learning the identity map, so it has to learn which structure in the data is signal and which is noise. Later this idea becomes central. With Gaussian corruption, training a denoiser is equivalent to score matching on a smoothed version of the data density (Vincent, 2011), and a diffusion model’s noise-prediction objective trains a denoiser at many noise levels.

Figure

Denoising autoencoder: corrupt the input, score against the clean image

Here the corruption is additive Gaussian noise,

$$\tilde x = x + 0.2\,\varepsilon, \qquad \varepsilon \sim \mathcal N(0, I),$$

without clipping, so corrupted pixels can leave $[0, 1]$. Every block of the encoder and decoder also applies Dropout2d(0.3), which drops whole feature maps at random during training: a second, internal kind of corruption. The loss is the same MSE against the clean image. After 5 epochs the denoising model’s training loss is $0.0102$, compared with $0.0021$ for the plain autoencoder. The numbers are not directly comparable: the denoising model solves a harder task, with noisy input and dropout active.

On test digits corrupted the same way, the trained model removes the noise completely: the background comes out uniformly black. The strokes come out slightly thicker and smoother than in the original, and small details, such as a short hook at the top of a 0, are lost. The codes behave like the plain model’s. Before training most digits already form clusters, with 4 and 9 overlapping; after training there are ten clusters, again with the 2s in two groups and the 1s in an elongated band. Class-wise sampling as above gives mostly recognisable but blurry digits in a single draw per class; the 4 and the 5 are badly distorted.

Sparse autoencoder#

A sparse autoencoder keeps the reconstruction loss and adds a sparsity penalty that encourages rare activations: each hidden unit should be active for only a small fraction of the inputs (Ng, 2011). Together with the reconstruction loss, this is meant to make each input use only a few strongly active units. The hidden layer can then be large, even larger than the input, without the network simply copying its input through.

Two variants are common.

KL penalty on intermediate layers. Hidden units use sigmoid activations after batch norm, so every activation lies in $(0, 1)$ and can be read as a firing rate. For each unit $j$, let $\hat\rho_j$ be its mean activation over the batch; Ng (2011) averages over the whole training set, and the batch mean is an estimate of that average. The penalty compares $\hat\rho_j$ with a target rate $\rho$ through the KL divergence between two Bernoulli distributions:

$$\mathcal L = \mathrm{MSE}(\hat x, x) + \beta \sum_{j} D_{\mathrm{KL}}(\rho \,\|\, \hat\rho_j),$$ $$D_{\mathrm{KL}}(\rho \,\|\, \hat\rho_j) = \rho \log\frac{\rho}{\hat\rho_j} + (1-\rho)\log\frac{1-\rho}{1-\hat\rho_j},$$

with $\rho = 0.05$ and $\beta = 0.001$ here.

Figure

Dense and sparse codes: what the sparsity penalty aims for

This is the KL divergence between a “fires with probability $\rho$” coin and a “fires with probability $\hat\rho_j$” coin. It is zero only when the average firing rate equals $\rho$. The penalty constrains only the average rate of each unit: a unit firing weakly at $0.05$ for every input satisfies it too. Sparsity for each input has to come from the interplay with the reconstruction loss. The sum runs over every channel and position of the four intermediate encoder layers: $16\cdot 32^2 + 32 \cdot 16^2 + 32 \cdot 8^2 + 32 \cdot 4^2 = 27136$ terms. This model’s decoder also differs from the others: it uses transposed convolutions (kernel 4, stride 2, padding 1) instead of upsample + conv. With dilation 1 and no output padding, the ConvTranspose2d output-size formula gives a doubled side:

$$H_{\text{out}} = (H - 1)\cdot 2 - 2\cdot 1 + 4 = 2H.$$

L1 penalty on the code. This is the plain architecture plus a penalty with weight $\lambda = 0.001$ on the mean absolute value of the 512 code entries, averaged over the batch as well:

$$\mathcal L = \mathrm{MSE}(\hat x, x) + \lambda \cdot \frac{1}{512}\sum_{k=1}^{512} |z_k|.$$

The (sub)gradient of $|z_k|$ is $\operatorname{sign}(z_k)$, of constant size, so the L1 penalty keeps pulling small entries towards zero, whereas the pull of a squared penalty vanishes near zero. The minimizer of an L1-penalized objective therefore typically has some entries exactly at zero: it soft-thresholds them (Goodfellow, Bengio and Courville, 2016, section 7.1.2). Plain gradient training with Adam, as here, only drives such entries close to zero, where they keep oscillating.

Neither variant works well here, and the reasons are instructive. If the hidden sigmoids start with batch means near $0.5$, as batch norm centres their inputs at initialization, the KL term starts at roughly $27136 \cdot 0.001 \cdot D_{\mathrm{KL}}(0.05\,\|\,0.5) \approx 13$. This is an order-of-magnitude estimate, since the per-unit means vary; because $D_{\mathrm{KL}}(\rho\,\|\,\cdot)$ is convex, a spread of the $\hat\rho_j$ around an average of $0.5$ only increases it (Jensen’s inequality).

Only the sum of the two terms was logged. It falls from $8.85$ in the first epoch to $0.214$ in the fifth, and it is still falling steeply ($0.788$ in the fourth epoch), so training has not converged. An all-black output would score an MSE equal to the mean squared pixel value, about $0.10$ for the resized MNIST images (computed from the MNIST test set with the same bilinear resize). The stored reconstructions of the KL model are worse than the plain model’s but far from blank, which suggests that much of the $0.214$ is penalty. It does not bound the split: the black output is one feasible baseline, not an upper bound on the reconstruction error of a model that has not converged, which can be larger. Logging the reconstruction term and the penalty separately would settle it. The scale of the penalty points the same way: with 27136 terms, $\beta = 0.001$ is too large a weight. The KL model’s reconstructions are visibly worse and its t-SNE plot is fragmented. Class-wise sampling from its codes fails completely: none of the ten decoded samples is a recognisable digit, only broken fragments of strokes.

The L1 variant has the opposite problem. With $\lambda = 0.001$ the penalty does not visibly change training: the loss ends at $0.0022$, essentially the plain model’s $0.0021$, and with one run each the gap is within noise. Everything else is similar to the plain model: ten clusters in the t-SNE plot and reconstructions as faithful as the plain model’s; the class-wise samples are somewhat worse, with the 0, 1, 6 and 7 clean and the other digits distorted. The penalty can also be bypassed. The first decoder block applies bilinear upsampling and a bias-free convolution, both linear, and then batch norm. Replacing $z$ by $r z$ for any $r > 0$ scales the convolution’s outputs by $r$, and batch norm divides this factor out again, as long as $r^2\sigma_B^2$ stays well above its $\varepsilon$ (batch norm ignores the scale of its input). The encoder could therefore lower the penalty by shrinking the overall scale of $z$, without making the code sparse. The scale of $z$ was not measured, so this is a possible bypass, not an observed one; comparing the mean of $\lvert z_k\rvert$ before and after training would show it. Sparsity penalties need their weight tuned against the reconstruction loss, and they need an architecture in which they cannot be bypassed.

Transposed convolution or upsample + conv?#

Both double the spatial size. A transposed convolution learns its own upsampling filter, but when the kernel size is not divisible by the stride, the kernel footprints overlap unevenly and produce checkerboard artifacts. Resizing followed by a convolution avoids the uneven overlap and is the usual safer default (Odena, Dumoulin and Olah, 2016). The same study adds two caveats. Its best results came with nearest-neighbour resizing, and the authors had difficulty making bilinear resizing work, which is what the plain model here uses. A kernel divisible by the stride, such as kernel 4 with stride 2 in the KL model, removes the uneven overlap, but the learned kernels can still produce artifacts. The two decoders here are not compared under equal conditions, so the choice is a design option rather than a measured result.

Why a sigmoid at the output?#

The convolutional blocks use LeakyReLU with slope $0.2$, $\max(0.2x,\ x)$ (Maas, Hannun and Ng, 2013). Why such non-saturating activations replace sigmoids inside deep networks is explained in the previous note. The sigmoid has a different job here. It squashes any real number into $(0, 1)$, the range of the pixel intensities, so it maps the unbounded output of the decoder’s last convolution onto the scale of the targets. It passes through $0.5$ at $x = 0$:

$$\sigma(x) = \frac{1}{1 + e^{-x}}, \qquad \sigma(0) = \tfrac12 .$$

Figure

LeakyReLU and sigmoid

The sigmoid is the output activation of every model here. In the KL-sparse model it is also the hidden activation, because the Bernoulli penalty needs activations that can be read as firing rates in $(0, 1)$.

For targets scaled to $[-1, 1]$ the sigmoid has the wrong range: it can never produce a negative value. The usual output is then $\tanh$, a sigmoid stretched to $(-1, 1)$:

$$\tanh(x) = \frac{e^{x} - e^{-x}}{e^{x} + e^{-x}} = 2\,\sigma(2x) - 1 .$$

The DCGAN generator, for example, ends with $\tanh$, and its training images are scaled to $[-1, 1]$ (Radford, Metz and Chintala, 2016, sections 3–4). With MSE a linear output, with no activation at all, also works; its values are unbounded and are clipped to the valid range for display.

Notebook#

The experiments follow the autoencoder notebook of the HSE-LAMBDA Deep Generative Models repository (pinned commit; cell numbers count every cell, markdown included, from 1). All loss values and observations about plots above come from its stored outputs; the all-black MSE is computed from the dataset.

• Cells 5–7: MNIST resized to $64 \times 64$ and loaded as tensors in $[0, 1]$, in batches of 64 with drop_last=True (cell 7).
• Cell 9: visualize_latent_space, which flattens the codes and projects them with t-SNE (perplexity 30); it is called before and after training in cells 15 and 20. It switches the model to eval(), so the plot before training uses batch norms that still hold their initial running statistics (mean 0, variance 1): they are effectively identity maps.
• Cell 10 writes the encoder as $f_\theta$ and the decoder as $g_\phi$, the opposite letters to this note. Cell 11: Block and AutoEncoder, with the shapes of the tables above. Cells 12–13 choose nn.MSELoss and Adam with learning rate $10^{-3}$; the markdown suggests nn.BCEWithLogitsLoss as an alternative, which, as explained above, would double the sigmoid. Cell 17 trains for 5 epochs.
• Cells 22–25: test reconstructions, latent interpolation between the first two test images and per-class diagonal-Gaussian sampling (compute_activation_stats, on the training codes).
• Cells 26–37: the denoising autoencoder (add_noise, default factor 0.1 but called with 0.2 both in training, cell 31, and for the test images, cell 35; DenoisingBlock with Dropout2d(0.3)), its t-SNE before and after training (cells 30 and 33), the original / noisy / denoised test images (cell 35) and class-wise sampling (cell 37).
• Cells 38–49: the KL-sparse autoencoder (kl_divergence_activations, SparseAE_KL with transposed-convolution decoder, $\rho = 0.05$, $\beta = 0.001$). Its decoder blocks also use sigmoids, and the code itself is passed through a sigmoid, so the class-wise Gaussians of cell 49 are fitted to codes in $(0, 1)$. The sparse-activation picture in cell 38 comes from Jordan (2018), an illustrated introduction to autoencoders that the repository recommends.
• Cells 50–59: the L1-sparse autoencoder ($\lambda = 0.001$): training, t-SNE (cell 56), reconstructions (cell 57) and class-wise sampling (cell 59).
• The KL and L1 training loops (cells 43 and 53) run the encoder a second time after the forward pass to get the activations for the penalty. This doubles the encoder’s compute, and in training mode it updates batch norm’s running averages twice per step.
• Saved training losses per epoch (average over the 937 batches of an epoch):

Model	Cell	Epoch 1	2	3	4	5
plain	17	0.0103	0.0034	0.0026	0.0023	0.0021
denoising	31	0.0313	0.0137	0.0116	0.0107	0.0102
KL-sparse (sum of both terms)	43	8.8531	4.4430	2.3315	0.7877	0.2141
L1-sparse (sum of both terms)	53	0.0095	0.0034	0.0028	0.0024	0.0022

Homework 1 of the same repository (same cell convention) trains an autoencoder on CIFAR-10 and then uses its encoder for FID; the FID part is described in the FID note. The notebook has no saved outputs and its solution cells are empty, so it documents a procedure only.

• Cell 3: set_seed(42) seeds Python’s random, NumPy and PyTorch on the CPU and all GPUs, and sets cudnn.deterministic = True and cudnn.benchmark = False (what each does).
• Cells 6–13: CIFAR-10 (50 000 training and 10 000 validation images of $3 \times 32 \times 32$, ten classes) with the transform ToTensor() followed by x * 2 - 1, and denormalize_image, $(x + 1)/2$, for display. Batches of 128 for training; the validation loader uses batches of one image.
• Cells 15–19: the student writes Encoder and Decoder from any of Conv2d, MaxPool2d, AvgPool2d, Upsample and ConvTranspose2d. Two checks: the flattened code must have fewer than $3 \cdot 32 \cdot 32 = 3072$ numbers, and the decoder must return a $3 \times 32 \times 32$ image. The inputs lie in $[-1, 1]$, so the decoder’s last layer needs a $\tanh$ or a linear output, not a sigmoid.
• Cells 20–22 and 32: classification_score, the probe described above. It puts the encoder in eval() mode, computes the flattened code of each image under torch.no_grad(), trains a GradientBoostingClassifier with n_estimators=50 and max_depth=5 on the codes of 5000 random training images and returns its accuracy on all 10 000 validation images. Cell 22 scores an untrained Encoder(); cell 32 requires the trained encoder to score more than 1.05 times as much.
• Cells 23–31: train (MSE, the mean of the batch losses over one pass through the training set) and eval (the same on the validation set, in eval() mode and under torch.no_grad()); full_train trains with AdamW at its default settings (learning rate $10^{-3}$, weight decay $10^{-2}$) for 30 epochs, plots training and validation MSE per epoch, and is to be extended so that it loads saved weights instead of training again.
• Cell 33: ten validation images and their reconstructions.
• Notebook issues. classification_score draws a new random subset of 5000 training images at every call, so the untrained and the trained encoder are probed on different subsets; fixing the subset would make the 5% comparison less noisy. The function named eval in cell 27 shadows Python’s built-in eval. Cell 17 tests the encoder on torch.rand(1, 3, 32, 32) - 1, values in $[-1, 0)$ rather than $[-1, 1]$; only the shape is checked, so this does no harm.

Next#

A plain autoencoder learns useful codes, but on its own it gives no way to sample new images: it specifies no prior over codes and never trains arbitrary latent points to decode to plausible images. A density fitted to the codes afterwards makes a generative pipeline of sorts, but the class-wise Gaussians above needed labels and still gave distorted digits. The next note makes the distribution of the codes part of the model and trains it with a lower bound on the log-likelihood, which turns the autoencoder into a generator.

References#

• D. E. Rumelhart, G. E. Hinton, R. J. Williams. Learning internal representations by error propagation. In Parallel Distributed Processing, vol. 1, chapter 8, MIT Press, 1986; also ICS Report 8506, 1985. Section “The encoding problem”.
• G. E. Hinton, R. R. Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313(5786), 2006.
• P. Vincent, H. Larochelle, Y. Bengio, P.-A. Manzagol. Extracting and composing robust features with denoising autoencoders. ICML, 2008.
• P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, P.-A. Manzagol. Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. JMLR, 11, 2010.
• P. Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23(7), 2011.
• A. Ng. Sparse autoencoder. CS294A lecture notes, Stanford, 2011.
• I. Goodfellow, Y. Bengio, A. Courville. Deep Learning. MIT Press, 2016. Section 7.1.2: L1 regularization; chapter 14: autoencoders.
• G. Loaiza-Ganem, J. P. Cunningham. The continuous Bernoulli: fixing a pervasive error in variational autoencoders. NeurIPS, 2019.
• L. van der Maaten, G. Hinton. Visualizing data using t-SNE. JMLR, 9, 2008.
• M. Wattenberg, F. Viégas, I. Johnson. How to use t-SNE effectively. Distill, 2016. Sections “Cluster sizes in a t-SNE plot mean nothing” and “Distances between clusters might not mean anything”.
• A. Saxe, P. W. Koh, Z. Chen, M. Bhand, B. Suresh, A. Ng. On random weights and unsupervised feature learning. ICML, 2011.
• G. Alain, Y. Bengio. Understanding intermediate layers using linear classifier probes. arXiv:1610.01644, 2016.
• J. T. Springenberg, A. Dosovitskiy, T. Brox, M. Riedmiller. Striving for simplicity: the all convolutional net. ICLR Workshop, 2015.
• A. Odena, V. Dumoulin, C. Olah. Deconvolution and checkerboard artifacts. Distill, 2016.
• A. L. Maas, A. Y. Hannun, A. Y. Ng. Rectifier nonlinearities improve neural network acoustic models. ICML Workshop on Deep Learning for Audio, Speech and Language Processing, 2013.
• A. Radford, L. Metz, S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. ICLR, 2016. Section 3: $\tanh$ output of the generator; section 4: images scaled to $[-1, 1]$.
• J. Jordan. Introduction to autoencoders. Blog post, 2018.
• PyTorch documentation: ConvTranspose2d.
• HSE-LAMBDA, Deep Generative Models course repository: autoencoders.ipynb and 01_hw_AE-and-FID.ipynb.