Batch normalization, gradients and convolutions

The previous note fitted a model with a handful of parameters. From here on the models are neural networks. The autoencoders of the next note are built from blocks of convolution → batch normalization → activation, and every later model is trained by backpropagating gradients through many such layers. The evaluation metrics later in the course also run images through pretrained convolutional networks. This note collects that background: what batch normalization computes and how instance normalization differs, why gradients can vanish or explode, how a convolution layer is shaped, and the routine around a training run.

Batch normalization#

Batch normalization (Ioffe and Szegedy, 2015) standardizes the activations of a layer using statistics of the current mini-batch. Take one feature (in a convolutional network, one channel) and a mini-batch of $n$ values $x_1, \dots, x_n$ of it. The layer first computes the batch mean and batch variance

$$\mu_B = \frac1n\sum_{i=1}^{n} x_i, \qquad \sigma_B^2 = \frac1n\sum_{i=1}^{n} \left(x_i - \mu_B\right)^2,$$

and then standardizes every value:

$$\hat x_i = \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \varepsilon}} .$$

The small constant $\varepsilon > 0$ (PyTorch uses $10^{-5}$) keeps the division safe when a batch has almost no spread. After this step the batch has mean $0$ and variance $\sigma_B^2/(\sigma_B^2 + \varepsilon) \approx 1$.

Figure

Batch normalization of one feature

A small worked example. For the batch $x = (1, 2, 3, 6)$ with $\varepsilon = 10^{-5}$:

	first	second	third	fourth
$x_i$	$1$	$2$	$3$	$6$
$x_i - \mu_B$	$-2$	$-1$	$0$	$3$
$\hat x_i$	$-1.069$	$-0.535$	$0$	$1.604$

Here $\mu_B = 12/4 = 3$ and $\sigma_B^2 = (4 + 1 + 0 + 9)/4 = 3.5$, so every centred value is divided by $\sqrt{3.5 + 10^{-5}} \approx 1.871$. The variance divides by $n$, not $n - 1$: it is the plain variance of the batch.

The rest of the layer#

A full batch-norm layer has two learnable parameters per feature, a scale $\gamma$ and a shift $\beta$, and outputs

$$y_i = \gamma\,\hat x_i + \beta .$$

The parameters $\gamma$ and $\beta$ are fixed numbers, while $\mu_B$ and $\sigma_B^2$ change from batch to batch, so the identity cannot be matched batch by batch. It can be matched against the population statistics of the feature: with $\gamma = \sqrt{\operatorname{Var}[x] + \varepsilon}$ and $\beta = \mathbb E[x]$ the layer reproduces its input at inference, where running averages that estimate these population statistics (described below) replace $\mu_B$ and $\sigma_B^2$. In training it does so only approximately, up to the batch-to-batch fluctuation of $\mu_B$ and $\sigma_B^2$ (Ioffe and Szegedy, 2015, section 3). Standardization therefore does not restrict what the layer can express. It changes the parametrization, so that the scale of each feature is controlled by one parameter.

Training and inference behave differently. During training the layer uses the batch statistics $\mu_B, \sigma_B^2$ and also keeps running averages of them. In PyTorch, with the default momentum $0.1$, these are $\mu_{\text{run}} \leftarrow 0.9\,\mu_{\text{run}} + 0.1\,\mu_B$ and $\sigma^2_{\text{run}} \leftarrow 0.9\,\sigma^2_{\text{run}} + 0.1\,\tfrac{n}{n-1}\sigma_B^2$; the running variance stores the unbiased estimate. At inference, when a “batch” may be a single image, the running averages are used instead, so the output of one example no longer depends on the other examples in the batch. In PyTorch the switch is model.eval(). Turning off gradients with torch.no_grad() does not switch batch norm to its running statistics.

In a convolutional network, BatchNorm2d normalizes each channel over all values of that channel in the batch: batch size × height × width of them. A convolution followed by batch norm is usually created with bias=False, because subtracting $\mu_B$ cancels any constant bias, and $\beta$ takes over its role.

For a similar reason, batch norm does not see the scale of what comes before it. If a bias-free convolution with weights $W$ computes the channel values $x_i$ from an input $u$, then scaling $u$ or $W$ by a factor $r > 0$ turns every $x_i$ into $r\,x_i$, because the convolution is linear. Then $\mu_B$ becomes $r\,\mu_B$ and $\sigma_B^2$ becomes $r^2\sigma_B^2$, and

$$\frac{r x_i - r\mu_B}{\sqrt{r^2\sigma_B^2 + \varepsilon}} = \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \varepsilon/r^2}} .$$

Up to the small change of $\varepsilon$ to $\varepsilon/r^2$, the output is unchanged (Ioffe and Szegedy, 2015, section 3.3). A bias-free convolution followed by batch norm is therefore insensitive to the overall scale of its input and of its weights.

Why does it help? The original paper motivated batch norm by reducing “internal covariate shift”, the drift of each layer’s input distribution during training. Santurkar et al. (2018), sections 2–3, argue that the gain does not come from reducing internal covariate shift. In their analysis batch norm makes the loss and its gradients smoother, which tolerates larger learning rates; why batch norm works is still debated. Either way, it keeps the scale of activations under control, which is exactly the issue of the next section.

Instance normalization#

Batch norm pools each channel over the whole batch, so the output for one image depends on the other images in it. Instance normalization (Ulyanov, Vedaldi and Lempitsky, 2016, Eq. 3) standardizes each image on its own. For image $n$ and channel $c$ of an $H \times W$ feature map $x$,

$$\begin{gathered} \mu_{nc} = \frac{1}{HW}\sum_{h,w} x_{nchw}, \qquad \sigma_{nc}^2 = \frac{1}{HW}\sum_{h,w} \left(x_{nchw} - \mu_{nc}\right)^2, \\ y_{nchw} = \gamma_c\,\frac{x_{nchw} - \mu_{nc}}{\sqrt{\sigma_{nc}^2 + \varepsilon}} + \beta_c , \end{gathered}$$

with a learnable scale $\gamma_c$ and shift $\beta_c$ per channel, as in batch norm (PyTorch’s InstanceNorm2d includes them only with affine=True). The layers differ only in which values share a mean and a variance:

Layer	Statistics computed over	One mean and variance per
batch norm	the $N$ images and all $H \times W$ positions	channel
instance norm	the $H \times W$ positions of one image	image and channel
layer norm, image (CNN)	all channels and positions of one image	image
layer norm, Transformer	the $d$ features of one token	token

Layer norm normalizes over whatever trailing axes it is configured with (normalized_shape in PyTorch). The CNN row is the configuration that pools a whole image; in a Transformer the same layer normalizes the feature vector of each token separately.

If the four values $(1, 2, 3, 6)$ of the worked example above are the $2 \times 2$ map of one channel of one image, instance norm (with $\gamma_c = 1$, $\beta_c = 0$) turns them into $(-1.069, -0.535, 0, 1.604)$, whatever the other images in the batch contain. This has three consequences:

• No dependence on the batch. Training and inference compute the same function, no running averages are needed, and a batch of a single image works.
• Per-image contrast is removed. The mean and the spread of every feature map of every image are discarded; $\gamma_c$ and $\beta_c$ are the same for all images and cannot restore them. This is what fast style transfer needs: the stylized image should not depend on the contrast of the content image (Ulyanov et al., section 2). Where the overall brightness or contrast of an image carries information, the layer throws it away.
• It needs spatial extent. On a $1 \times 1$ map each channel has a single value, its variance is zero, and the output is $\beta_c$ whatever the input.

Three uses of instance normalization:

• Style transfer. Replacing batch norm by instance norm in a feed-forward stylization network, and keeping it at test time, improved the stylized images markedly (Ulyanov et al., section 3).
• CycleGAN. Both the generators and the discriminators use it, and training runs with batches of one image (Zhu et al., 2017, section 4 and appendix 7.2; see the networks of CycleGAN).
• The critic of WGAN-GP. The gradient penalty constrains the gradient of the critic’s output for each input separately, which is meaningful only if each output depends on its own input alone. Batch norm breaks this, so WGAN-GP omits it from the critic. Any normalization without interactions between examples is allowed; Gulrajani et al. (2017), section 4, recommend layer normalization, and instance normalization qualifies as well.

Vanishing and exploding gradients#

We train by computing the gradient of the loss with respect to every weight (backpropagation) and stepping against it. By the chain rule, the gradient that reaches an early layer is a product of one factor per layer, or per time step, that it passed through. With one unit per layer, $h_i = \varphi(a_i)$ with pre-activation $a_i = w_i h_{i-1} + b_i$ and activation function $\varphi$, each factor is

$$\frac{\partial h_i}{\partial h_{i-1}} = w_i\,\varphi'(a_i),$$

a weight times the slope of the activation. For layers with many units the factors are Jacobian matrices, and what matters is their size: the product of many Jacobians tends to grow or shrink geometrically with depth, at a rate set by their largest singular values (Pascanu, Mikolov and Bengio, 2013, section 2.1; Goodfellow, Bengio and Courville, 2016, sections 8.2.5 and 10.7). So if the per-layer factors are mostly

• larger than $1$ in size, the product grows geometrically: exploding gradients;
• smaller than $1$ in size, the product shrinks geometrically: vanishing gradients.

The further the factors are from $1$, the faster this happens, but the departure need not be large (see the table below). The problem was first analysed for recurrent networks (Hochreiter, 1991; Bengio, Simard and Frasconi, 1994), where it is easiest to see.

Example: an RNN#

A recurrent network applies the same weight at every time step. Take a single recurrent unit with a ReLU, $h_t = \operatorname{ReLU}(w\,h_{t-1} + u\,x_t + b)$, and recurrent weight $w = 2$.

Figure

Unrolled RNN with recurrent weight w = 2

Every step along the recurrent path multiplies by the same $w = 2$, so the sensitivity of a later state to an earlier one is $2$ after one step, $4$ after two, $8$ after three, and so on. Formally, let the loss be computed only from the last state $h_{k+\tau}$, $\tau$ steps after $h_k$ (as in the unrolled RNN above, where only the last output is used). While the ReLU is in its active region ($\operatorname{ReLU}' = 1$), each per-step factor $w\,\varphi'(a_t)$ equals $w$, and the chain rule gives

$$\begin{aligned} \frac{\partial h_t}{\partial h_{t-1}} &= w \quad \text{for every } t, \\ \frac{\partial h_{k+\tau}}{\partial h_k} &= \prod_{s=k+1}^{k+\tau} \frac{\partial h_s}{\partial h_{s-1}} = w^{\tau}, \\ \frac{\partial L}{\partial h_k} &= \frac{\partial L}{\partial h_{k+\tau}}\, w^{\tau} . \end{aligned}$$

The gradient sent back $\tau$ steps carries the factor $w^\tau$:

weight	after 3 steps	after 10 steps	after 50 steps
$w = 2$	$8$	$1024$	$\approx 1.1\cdot 10^{15}$
$w = 1.1$	$1.33$	$2.59$	$117$
$w = 1$	$1$	$1$	$1$
$w = 0.9$	$0.73$	$0.35$	$0.0052$
$w = 0.5$	$0.125$	$\approx 9.8\cdot 10^{-4}$	$\approx 8.9\cdot 10^{-16}$

Even a weight only 10 % away from $1$ changes the gradient by two orders of magnitude over 50 steps. Nothing in the table is specific to recurrence: a stack of $\tau$ layers with the same factor $w$ each gives the same $w^\tau$, and that depth-wise reading is all the autoencoders of the next notes need. Recurrent units and their gates are developed in the NLP course (recurrent networks, LSTM and GRU).

Figure

Repeated multiplication by w

What this does to training#

A gradient-descent step is $\theta \leftarrow \theta - \eta\,\nabla_\theta L$, so the size of the step is proportional to the gradient.

• Exploding: the gradient arrives multiplied by a huge factor, so the update takes a very large step in the loss landscape. Each update overshoots, and the loss jumps around or diverges.
• Vanishing: the gradient that reaches the early layers of a deep stack carries one factor per layer. If the factors are small, the product goes to zero and the steps become tiny: the early weights barely move, and the training budget runs out long before the minimum is reached.

Figure

What a huge or tiny gradient does to the training steps

Common remedies#

The rest of the course relies on the standard fixes:

• Initialization that keeps the scale of signals near 1: Xavier/Glorot initialization for saturating activations (Glorot and Bengio, 2010) and He initialization for ReLUs (He et al., 2015), described in the next section.
• Normalization layers such as batch norm above.
• Non-saturating activations instead of sigmoids. The sigmoid’s derivative is at most $\tfrac14$ (reached at $0$), so the factor $w_i\,\varphi'(a_i)$ of a sigmoid layer is at most $|w_i|/4$ in size: each sigmoid layer shrinks the gradient by at least a factor of 4 even when $|w_i| = 1$. ReLU, $\operatorname{ReLU}(x) = \max(0, x)$ (Nair and Hinton, 2010), has slope exactly $1$ wherever the unit is active. LeakyReLU, $\max(\alpha x,\ x)$ with a small slope $\alpha$ such as $0.2$ (Maas, Hannun and Ng, 2013), also keeps slope $\alpha$ instead of $0$ for negative inputs, so an inactive unit still passes some gradient.
• Gradient clipping against explosions: rescale the gradient when its norm exceeds a threshold (Pascanu, Mikolov and Bengio, 2013, section 3.2).
• Gated recurrent cells. In the LSTM (Hochreiter and Schmidhuber, 1997) the cell state is updated additively. In the original cell its factor per step is exactly $1$. The cell used today adds a forget gate (Gers, Schmidhuber and Cummins, 2000), and the factor equals that gate: a number in $(0, 1)$ that the network can keep near $1$. The GRU (Cho et al., 2014) works the same way: it interpolates between the old state and a candidate, so the direct path carries the factor of its update gate.
• Residual connections (He et al., 2016a), $x_{l+1} = x_l + F(x_l)$. With a pure identity skip, as in pre-activation ResNets (He et al., 2016b, eq. 5), the Jacobian of a block is $I + \partial F/\partial x_l$, so the gradient always has a path whose factor is exactly $1$.

A convolution layer#

How a 2-D convolution layer (nn.Conv2d) is parametrized and trained (Goodfellow, Bengio and Courville, 2016, chapter 9; Dumoulin and Visin, 2016):

• The kernels are trained. All entries of the weight tensor and the bias are learned by gradient descent:

  $$W \in \mathbb R^{\text{out}_c \times \text{in}_c \times k_H \times k_W}, \qquad b \in \mathbb R^{\text{out}_c} .$$

• Notation. $\text{in}[c]$ is input channel number $c$, an $H_{\text{in}} \times W_{\text{in}}$ plane (subscripted $W_{\text{in}}, W_{\text{out}}$ are widths; $W$, $W_j$, $W_{j,c}$ are always weights). $W_{j,c}$ is the $k_H \times k_W$ kernel that output channel $j$ applies to input channel $c$. Collected over $c$, $W_j \in \mathbb R^{\text{in}_c \times k_H \times k_W}$ is the filter for output channel $j$.

• Output channel $j$, for $j = 1, \dots, \text{out}_c$, sums the per-channel results and adds one scalar bias:

  $$\text{out}[j] = \sum_{c=1}^{\text{in}_c} W_{j,c} \star \text{in}[c] + b_j .$$

  Each $\text{out}[j]$ is an $H_{\text{out}} \times W_{\text{out}}$ plane. There is one bias per output channel, so the bias vector has $\text{out}_c$ entries.

Figure

Shapes in a convolution layer: one output channel

Written out entry by entry for stride $s$ and dilation $d$, the operation $\star$ is a sliding dot product. Deep-learning libraries compute a cross-correlation and do not flip the kernel, unlike the convolution of signal processing; since the kernel is learned, the difference does not matter.

$$\begin{aligned} \text{out}[j]_{y,x} = b_j &+ \sum_{c=1}^{\text{in}_c}\sum_{m=0}^{k_H-1}\sum_{l=0}^{k_W-1} W_{j,c,m,l} \\ &\quad\cdot\ \text{in}[c]_{\,s y + d m,\; s x + d l} , \end{aligned}$$

where $y = 0, \dots, H_{\text{out}} - 1$, $x = 0, \dots, W_{\text{out}} - 1$, and $\text{in}[c]$ denotes the input after zero-padding by $p$ on each side, indexed from $0$. The formula assumes one group, so every output channel sees every input channel; grouped convolutions restrict these connections.

• Hyperparameters, not trained: stride, padding, kernel size and dilation are fixed when the layer is built. They determine the output size (PyTorch documentation):

  $$\begin{aligned} H_{\text{out}} &= \left\lfloor \frac{N_H}{s} \right\rfloor + 1, \\ N_H &= H_{\text{in}} + 2p - d\,(k_H - 1) - 1 \end{aligned}$$

  for the height, where $N_H$ is the padded height minus the extent $d\,(k_H - 1) + 1$ of the dilated kernel. The same holds for $W_{\text{out}}$ with $W_{\text{in}}$ and $k_W$. In the autoencoder of the next note each of the four encoder blocks has $k = 3,\ p = 1,\ s = 2,\ d = 1$, so $H_{\text{out}} = \lfloor (H_{\text{in}} - 1)/2 \rfloor + 1$ halves even sizes: $64 \to 32 \to 16 \to 8 \to 4$.

• Initialization. The kernels start random. With $\text{fan}_{\text{in}} = \text{in}_c\, k_H k_W$ and $\text{fan}_{\text{out}} = \text{out}_c\, k_H k_W$, Xavier/Glorot initialization draws weights with variance $2/(\text{fan}_{\text{in}} + \text{fan}_{\text{out}})$ (Glorot and Bengio, 2010), and He initialization with variance $2/\text{fan}_{\text{in}}$ for layers followed by a ReLU (He et al., 2015, section 2.2). Both are designed for the goal of the previous section: keep signals and gradients at a constant scale from layer to layer. PyTorch’s default for Conv2d is a simpler fan-in rule, a uniform distribution on $\pm 1/\sqrt{\text{fan}_{\text{in}}}$ (implemented as kaiming_uniform_ with $a = \sqrt5$). A uniform distribution on $\pm b$ has variance $b^2/3$, so this default has variance $1/(3\,\text{fan}_{\text{in}})$, six times smaller than the He variance. Without normalization, the signal variance of a deep ReLU stack initialized this way shrinks by about a factor of 6 per layer. In the networks of the next notes, batch norm after the convolutions restores the scale.

• What the weights learn. Kernels in the first layers typically become small detectors for edges, colour blobs and simple textures. Deeper layers combine them into parts and objects over larger receptive fields (Zeiler and Fergus, 2014, section 4 and figure 2). This is why pretrained convolutional networks are useful as feature extractors for the evaluation metrics of the course.

For scale: a layer with $\text{in}_c = 16$, $\text{out}_c = 32$ and $3 \times 3$ kernels has $32 \cdot 16 \cdot 3 \cdot 3 = 4608$ weights plus $32$ biases.

A training run in practice#

Around the model and its loss, every training run in the course follows the same routine.

• Optimizer. Most models use Adam (Kingma and Ba, 2015), which divides each coordinate’s step by a running estimate of its gradient’s magnitude (see the fitting note). A common variant is AdamW (Loshchilov and Hutter, 2019, section 2). An L2 penalty $\tfrac\lambda2\|\theta\|^2$ added to the loss enters Adam through the gradient and is divided by the same per-coordinate scale, so weights with historically large gradients are regularized less than intended. AdamW instead shrinks the weights directly, outside the adaptive step:

  $$\theta \leftarrow \theta - \eta\lambda\,\theta - \eta\,\frac{\hat m}{\sqrt{\hat v} + \epsilon},$$

  where $\hat m$ and $\hat v$ are Adam’s running estimates of the gradient’s mean and squared magnitude. For plain SGD, where $\hat m/(\sqrt{\hat v} + \epsilon)$ is replaced by the gradient itself, the two forms give the same update; for Adam they differ. PyTorch’s AdamW defaults to $\eta = 10^{-3}$ and $\lambda = 10^{-2}$.

• Validation. After each epoch the loss is also computed on held-out data, with the model in evaluation mode (model.eval(), which switches batch norm to its running averages and turns dropout off) and without gradient tracking (torch.no_grad(), which saves memory and time and changes nothing else). Plotting both losses per epoch shows overfitting as a validation curve that levels off or rises while the training curve keeps falling.

• Checkpoints. The trained parameters (the model’s state_dict) are saved to disk, so that the model can be reloaded and evaluated later without training it again.

• Reproducibility. Every random number generator involved is seeded: Python’s random, NumPy, and PyTorch, whose torch.manual_seed covers the CPU and all GPUs. On GPUs two more sources of variation remain. cuDNN can time several convolution algorithms and keep the fastest, which may differ between runs, and some algorithms are not deterministic; torch.backends.cudnn.benchmark = False and torch.backends.cudnn.deterministic = True turn both off, at some cost in speed. Even then, results are reproducible only on the same hardware and software versions (PyTorch documentation, “Reproducibility”).

Notebook#

Several notebooks of the HSE-LAMBDA Deep Generative Models repository (pinned commit; cell numbers count every cell, markdown included, from 1) use these pieces.

• The autoencoder notebook, cell 11, defines Block: Conv2d(..., bias=False) → BatchNorm2d → LeakyReLU(0.2), kernel 3, padding (kernel - 1) // 2 and stride 2 in the encoder. Four blocks map a $1 \times 64 \times 64$ MNIST image to $32 \times 4 \times 4$, and a final stride-1 $3\times 3$ convolution keeps that size, giving a 512-dimensional latent. Cells 17 and 22 switch explicitly between ae.train() and ae.eval(), so batch norm uses batch statistics while training and running averages while reconstructing test images.
• The VAE notebook, cells 10 and 12, orders its blocks as convolution → LeakyReLU → BatchNorm. Its generation code in cells 18 and 23 wraps sampling in torch.no_grad() without calling eval(), so batch norm still normalizes with the statistics of the generated batch and keeps updating its running buffers. This is the train/inference distinction from above in practice. The VAE notebook has no saved outputs, so it documents procedure only.
• The WGAN notebook, cell 12, normalizes the critic’s second to fourth convolutions with InstanceNorm2d(..., affine=True) (on $16 \times 16$, $8 \times 8$ and $4 \times 4$ maps for $64 \times 64$ images), and cell 23 does the same in the blocks of its spectrally normalized critic. The generators keep batch norm.
• Homework 1 follows the routine of the last section: set_seed in cell 3 sets all the seeds and both cuDNN flags, cells 23–31 train an autoencoder with AdamW at its defaults for 30 epochs and plot training and validation MSE per epoch, with the validation pass in eval() mode under torch.no_grad(), and the training function is to load saved weights when they exist. The notebook has no saved outputs.

Next#

With these pieces we can build the first neural model of the course. The next note stacks convolution → batch norm → LeakyReLU blocks into an encoder and a decoder and trains them to reconstruct MNIST digits through a 512-dimensional code. The questions it raises are what such a code captures, and whether new images can be generated from it.

References#

• S. Ioffe, C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. ICML, 2015.
• S. Santurkar, D. Tsipras, A. Ilyas, A. Mądry. How does batch normalization help optimization? NeurIPS, 2018.
• D. Ulyanov, A. Vedaldi, V. Lempitsky. Instance normalization: the missing ingredient for fast stylization. arXiv:1607.08022, 2016. Section 2 and Eq. (3): instance normalization and the contrast argument; section 3: its effect on stylization networks.
• J. L. Ba, J. R. Kiros, G. E. Hinton. Layer normalization. arXiv:1607.06450, 2016.
• J.-Y. Zhu, T. Park, P. Isola, A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. ICCV, 2017. Section 4 and appendix 7.2: instance normalization in the generators and discriminators, batch size 1.
• I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, A. Courville. Improved training of Wasserstein GANs. NeurIPS, 2017. Section 4, “No critic batch normalization”.
• D. P. Kingma, J. Ba. Adam: A method for stochastic optimization. ICLR, 2015.
• I. Loshchilov, F. Hutter. Decoupled weight decay regularization. ICLR, 2019. Section 2: L2 regularization and weight decay differ for Adam; AdamW.
• S. Hochreiter. Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, TU München, 1991.
• Y. Bengio, P. Simard, P. Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2), 1994.
• R. Pascanu, T. Mikolov, Y. Bengio. On the difficulty of training recurrent neural networks. ICML, 2013. Section 2.1 (conditions on the largest singular value) and section 3.2 (gradient-norm clipping).
• S. Hochreiter, J. Schmidhuber. Long short-term memory. Neural Computation, 9(8), 1997.
• F. A. Gers, J. Schmidhuber, F. Cummins. Learning to forget: Continual prediction with LSTM. Neural Computation, 12(10), 2000.
• K. Cho, B. van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, Y. Bengio. Learning phrase representations using RNN encoder–decoder for statistical machine translation. EMNLP, 2014.
• V. Nair, G. E. Hinton. Rectified linear units improve restricted Boltzmann machines. ICML, 2010.
• A. L. Maas, A. Y. Hannun, A. Y. Ng. Rectifier nonlinearities improve neural network acoustic models. ICML Workshop on Deep Learning for Audio, Speech and Language Processing, 2013.
• X. Glorot, Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. AISTATS, 2010.
• K. He, X. Zhang, S. Ren, J. Sun. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. ICCV, 2015.
• K. He, X. Zhang, S. Ren, J. Sun. Deep residual learning for image recognition. CVPR, 2016 (cited as 2016a).
• K. He, X. Zhang, S. Ren, J. Sun. Identity mappings in deep residual networks. ECCV, 2016 (cited as 2016b).
• M. D. Zeiler, R. Fergus. Visualizing and understanding convolutional networks. ECCV, 2014.
• V. Dumoulin, F. Visin. A guide to convolution arithmetic for deep learning. arXiv:1603.07285, 2016.
• I. Goodfellow, Y. Bengio, A. Courville. Deep Learning. MIT Press, 2016. Sections 8.2.5 and 10.7 (long-term dependencies) and chapter 9 (convolutional networks).
• PyTorch documentation: BatchNorm2d, InstanceNorm2d, Conv2d, AdamW and Reproducibility.