LPIPS

LPIPS (Learned Perceptual Image Patch Similarity; Zhang et al., 2018) measures how different two images look. Inception Score and FID score a whole set of generated images. LPIPS instead compares a pair of images, usually a generated or distorted image $x'$ and a reference $x$.

The idea in one line: take outputs from several internal layers of a pretrained network, find the differences between the feature maps, then normalise, weigh and average.

Why not compare pixels?#

The obvious distance between two images is pixelwise: the $L_2$ distance or the mean squared error over all $C\times H\times W$ values,

$$\mathrm{MSE}(x, x') = \frac{1}{CHW}\sum_{c,h,w}\big(x_{chw} - x'_{chw}\big)^2 .$$

For perception it is a bad distance, and it fails in both directions:

• Shift the image and MSE becomes large. Moving the picture by one pixel changes almost nothing for a human. Every pixel on an edge, however, now meets a different value, so the error is large.
• MSE considers blur to be fine. A blurred image replaces each pixel by an average of its neighbours. Each value moves only a little, so MSE stays small, even though the blur is clearly visible.

A small computed example makes this concrete. Take an $8\times 8$ image of vertical stripes with a period of 4 pixels, so every row reads 0 0 1 1 0 0 1 1. Compare it with two copies:

• shifted right by 1 px (wrapping around), so every row becomes 1 0 0 1 1 0 0 1;
• blurred with the kernel $\tfrac14[1\;2\;1]$ (again wrapping around at the edges), so every row becomes .25 .25 .75 .75 .25 .25 .75 .75.

Copy	Per-pixel squared error	MSE	To a human
Shifted	$1$ on half the pixels, $0$ elsewhere	$0.5$	looks the same
Blurred	$0.25^2 = 0.0625$ on every pixel	$0.0625$	looks worse

Pixel MSE calls the blurred copy $0.5 / 0.0625 = 8$ times closer to the original, which is the opposite of what we see.

Figure

Pixel MSE punishes a shift and forgives a blur

A related effect explains why networks trained with a pure MSE loss tend to produce blurry outputs (Mathieu, Couprie and LeCun, 2016). The prediction that minimises the expected squared error is the conditional mean, so when several sharp answers are plausible, the best guess under MSE is their average, which is a blur. The same averaging is one cause of blurry VAE samples. A perceptual distance should instead be robust to small, imperceptible shifts and sensitive to the loss of structure. Deep features get much closer to this. A convolutional network responds to edges, textures and parts, and its deeper layers summarise each neighbourhood on a coarse grid, so a one-pixel shift moves their responses only slightly. Early layers still register the shift, so the features change less than raw pixels but are not invariant: LPIPS itself reacts to shifts too small for people to notice (Ghildyal and Liu, 2022).

The algorithm#

1. Extract features $\varphi_l(x)$ and $\varphi_l(x')$ at several layers $l = 1,\dots,L$ of a pretrained network such as AlexNet (Krizhevsky, Sutskever and Hinton, 2012) or VGG (Simonyan and Zisserman, 2015). The same frozen network, with shared weights, processes both images.
2. Normalise channel-wise. At every spatial position $(h, w)$, divide the feature vector by its length across the channels.
3. Subtract the two normalised feature maps.
4. Weigh the difference channel by channel with a vector $w_l$, take its squared length across the channels at each position and average spatially over all positions. This gives one number $d_l$ per layer.
5. Sum over layers to get LPIPS.

The weights $w_l$ are learned from human perceptual judgements while the network stays frozen (the lin configuration of Zhang et al., 2018, §3). They are fitted on the BAPPS dataset of two-alternative forced choices (2AFC), in which people saw a reference patch of $64\times 64$ pixels and two distorted versions and picked the one that looked closer (§2).

Figure

How LPIPS compares two images

The formula#

Notation: layer $l$ produces a feature map $\varphi_l(x) \in \mathbb R^{C_l \times H_l \times W_l}$ with $C_l$ channels on an $H_l \times W_l$ grid. The feature vector at position $(h, w)$ is $\varphi_l(x)_{hw} \in \mathbb R^{C_l}$, and the hat marks its unit-normalised version:

$$\hat\varphi_l(x)_{hw} = \frac{\varphi_l(x)_{hw}}{\big\|\varphi_l(x)_{hw}\big\|_2} .$$

A post-ReLU feature vector can be zero. The reference implementation divides by $\|\varphi_l(x)_{hw}\|_2 + \varepsilon$ with $\varepsilon = 10^{-10}$, so a zero vector stays zero (Zhang, 2021).

Layer $l$, with a weight vector $w_l \in \mathbb R^{C_l}$ and the difference $\Delta_{l,hw}$ of the normalised features at each position (Zhang et al., 2018, §3, Eq. 1):

$$\begin{aligned} \Delta_{l,hw} &= \hat\varphi_l(x)_{hw} - \hat\varphi_l(x')_{hw} , \\ d_l(x, x') &= \frac{1}{H_l W_l}\sum_{h,w}\big\|\, w_l \odot \Delta_{l,hw}\big\|_2^2 . \end{aligned}$$

Here $\odot$ is the element-wise product, so $w_l$ scales each channel of the difference. The reference lpips package (Zhang, 2021) instead applies weights $v_{l,c} \ge 0$ to the squared channel differences, $\sum_c v_{l,c}\,\Delta_{l,hw,c}^2$, where $\Delta_{l,hw,c}$ is channel $c$ of $\Delta_{l,hw}$; the paper also restricts the weights to be non-negative (Zhang et al., 2018, Appendix A, Figure 10). Since $\|w_l \odot \Delta_{l,hw}\|_2^2 = \sum_c w_{l,c}^2\,\Delta_{l,hw,c}^2$, this is the formula above with $w_{l,c} = \sqrt{v_{l,c}}$, so the two forms are equivalent.

Final:

$$\mathrm{LPIPS}(x, x') = \sum_{l} d_l(x, x') .$$

Each piece has a job:

• Normalisation removes the overall magnitude of the activations. Only the direction of the feature vector, meaning which channels respond and in what proportion, is compared. Layers or positions with large activations therefore do not dominate. Without the weights the term is a cosine distance in disguise: for unit vectors with angle $\theta$ between them, $\|\hat u - \hat v\|_2^2 = \|\hat u\|_2^2 + \|\hat v\|_2^2 - 2\,\hat u^\top\hat v = 2 - 2\cos\theta$ (Zhang et al., 2018, §3).
• The weights $w_l$ decide which channels matter for perception and turn down the ones that do not.
• The spatial average $\frac{1}{H_l W_l}\sum_{h,w}$ makes layers with different grid sizes comparable. It also forgives a small local disagreement, provided the rest of the map agrees. Without the average, the terms $\|w_l \odot \Delta_{l,hw}\|_2^2$ form a map that shows where two images differ. The reference package returns such a map, upsampled to the input resolution and summed over layers, when it is created with spatial=True (Zhang, 2021).

The toy layer below has $C_l = 2$ channels on a $3 \times 3$ grid, so each feature vector can be drawn as an arrow. The weights are $w_l = (1, 0.6)$. Two positions show the effect of each step:

• Bottom right. The feature vectors point along channel 1 and channel 2. They differ by $90^\circ$, the largest angle possible for features taken after a ReLU, as in the reference implementation (Zhang, 2021), since such features are non-negative. Unweighted, the term is $2 - 2\cos 90^\circ = 2$. The difference is $(1, -1)$, and the weights turn it into $(1, -0.6)$, so the term is $1 + 0.36 = 1.36$.
• Centre. The directions differ by $40^\circ$. Unweighted, the term would be $2 - 2\cos 40^\circ \approx 0.47$. The weight $0.6$ on channel 2 lowers it to $0.34$.

All the other positions nearly agree. Averaging the nine terms gives $d_l \approx 0.20$.

Figure

Inside one layer: from two feature maps to the number dₗ

Interpretation#

• Low LPIPS: the images look perceptually similar.
• High LPIPS: the images differ in visible structure (edges, textures, shapes).

In a search through about 1000 randomly drawn FashionMNIST images, the nine nearest images to an ankle boot are all ankle boots. That search fed the network raw grayscale values $0$–$255$ instead of the $[-1, 1]$ range it expects, so it measured a mis-scaled variant, not LPIPS as defined. Rescaling the input changes the nonlinear features, and with them possibly the order of the neighbours. The result is a diagnostic of that setup, not evidence that LPIPS ranks this query by class; the claim needs a rerun with the reference preprocessing.

Pros and cons#

Pros

• Better agreement with humans than pixelwise metrics and classic measures such as SSIM (Wang et al., 2004). Most of the gain comes from the deep features themselves. Unweighted, the features of supervised networks agree with human 2AFC judgements 67.0–68.9% of the time, against 63.2% for $L_2$ and 63.1% for SSIM, while a human scores 73.9% (Zhang et al., 2018, §4.1). The learned weights help most on distortion types like those they were trained on: for AlexNet they raise the score from 76.8% to 78.7% on those test sets, but only from 65.0% to 65.3% on the outputs of real algorithms (Appendix A, Table 5).
• Less sensitive to small shifts than raw pixels: deeper features change little under them, although the metric is not shift-invariant (Ghildyal and Liu, 2022).

Cons

• Depends on the backbone CNN. AlexNet and VGG give different values, so scores are comparable only when they use the same backbone and the same version of the learned weights (Zhang, 2021).
• Slower than pixelwise metrics: every comparison needs a forward pass through a deep network.
• Still not perfect. It is a learned approximation of human perception, fitted to judgements on small patches. Small adversarial perturbations of an image can make LPIPS contradict human judgement dramatically (Kettunen, Härkönen and Lehtinen, 2019).

Where it is used#

• Comparing the outputs of GANs and diffusion models. It measures how close a generated image is to a reference. In conditional generation, the average LPIPS between pairs of outputs sampled for the same input serves as a measure of diversity (Zhu et al., 2017, §5.2, Figure 6).
• Nearest-neighbour checks for memorisation. Searching the training set for the LPIPS-nearest images of each generated sample can surface candidate copies, which FID cannot see (FID drawbacks). It is a screening tool: a small distance is not proof of memorisation, and a search that finds nothing is not an exhaustive audit. Inspect the candidates and report which part of the training set was searched, the preprocessing and the threshold.
• Evaluating image restoration and editing: super-resolution, inpainting and style transfer, where a pixel metric would reward blurry outputs.
• As a “perceptual loss” in training networks. LPIPS is differentiable, so it can replace or complement MSE as a training loss, in the spirit of earlier feature-space losses (Johnson, Alahi and Fei-Fei, 2016). This targets exactly the blur problem above.

Compared with IS and FID#

This extends the IS-versus-FID table with a column for LPIPS.

	Inception Score	FID	LPIPS
Aim	confidence and diversity of generated images	closeness of generated images to real ones	perceptual similarity of two images
Data used	generated images only	real and generated images	a pair of images, e.g. generated vs. reference
What is compared	each $p(y \mid x)$ with the marginal $p(y)$	Gaussian fit to real features vs. Gaussian fit to generated features	normalised deep features of the two images, position by position
Distance	KL divergence	squared Fréchet (2-Wasserstein) distance	learned, weighted squared $L_2$ distance in feature space
Better is	higher	lower	lower (more similar)

Notebook#

The metrics notebook of the HSE-LAMBDA Deep Generative Models repository (pinned commit; cell numbers count every cell, markdown included, from 1) contains the hands-on part of the evaluation block.

• Cell 7 gives the LPIPS formula, with the layer sum outside and the notation $\hat x^l_{ij}$, $\hat y^l_{ij}$ for the normalized features.
• Cells 10–13 load FashionMNIST, show the query image (index 0, an ankle boot) and compute LPIPS with lpips.LPIPS(net='alex') against 1000 indices drawn with replacement (np.random.seed(10)), after resizing each image to $512\times512$. Each image enters as a $512\times512$ grayscale tensor with values $0$–$255$. Broadcasting in the package’s input scaling layer turns it into a three-channel batch of one, without the rescaling to $[-1, 1]$ that the package expects (Zhang, 2021). The saved run is therefore an input-contract failure: its distances are not comparable with published LPIPS scores, and its ranking is not validated either, since correct scaling changes the nonlinear features and can reorder the neighbours. The numbers below are a diagnostic of this setup until they are recomputed with inputs in $[-1, 1]$ (for example normalize=True on images in $[0, 1]$). The query tensor is rebuilt in every iteration and gradients are tracked, which costs time but does not change the distances. When the model loads, the package prints spatial [off]: with the default spatial=False it returns one number per pair rather than the distance map described above.
• Cells 14–15 plot a histogram of distances per class and show the nine closest images. In the saved run, with the unscaled inputs, the nine nearest distances are 0.168, 0.181, 0.183, 0.191, 0.213, 0.214, 0.215, 0.217 and 0.220, all ankle boots. All ankle boots in the sample lie at about 0.17–0.44. The closest images of other classes, a few sandals and sneakers, lie at about 0.25–0.28, and almost all other images lie above 0.3.
• Cell 16 contains two automatic checks: the smallest distance must be below 0.2, and more than 90% of the nine nearest neighbours must share the query’s label, which with nine neighbours means all nine.

Next#

IS and FID judge sets of generated images, LPIPS judges pairs. None of them can say whether a model fails by producing implausible images or by leaving out part of the data. The next note separates the two with precision and recall.

References#

• R. Zhang, P. Isola, A. A. Efros, E. Shechtman, O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. CVPR, 2018. §2: the BAPPS 2AFC dataset of 64 × 64 patches; §3 and Eq. (1): unit normalisation over channels, channel weights $w_l$, cosine distance for $w_l = 1$, the lin configuration with frozen features; §4.1: 2AFC agreement of deep features vs. $L_2$ and SSIM; Appendix A, Table 5: full 2AFC results; Appendix A, Figure 10: non-negative learned weights, AlexNet channel counts.
• R. Zhang. PerceptualSimilarity, reference implementation of LPIPS (lpips package, release v0.1.4, weights version 0.1), 2021. lpips/lpips.py: input scaling for images in $[-1, 1]$, normalisation with $\varepsilon = 10^{-10}$, weights on squared channel differences; lpips/pretrained_networks.py: features taken after each ReLU.
• A. Krizhevsky, I. Sutskever, G. E. Hinton. ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems 25 (NIPS), 2012. The AlexNet backbone.
• K. Simonyan, A. Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015. The VGG backbone.
• Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 2004. The SSIM index.
• M. Mathieu, C. Couprie, Y. LeCun. Deep multi-scale video prediction beyond mean square error. ICLR, 2016. Blurry predictions under an MSE loss.
• A. Ghildyal, F. Liu. Shift-tolerant perceptual similarity metric. ECCV, 2022. Sensitivity of LPIPS to imperceptible shifts.
• M. Kettunen, E. Härkönen, J. Lehtinen. E-LPIPS: robust perceptual image similarity via random transformation ensembles. arXiv, 2019. Adversarial attacks on LPIPS.
• J.-Y. Zhu, R. Zhang, D. Pathak, T. Darrell, A. A. Efros, O. Wang, E. Shechtman. Toward multimodal image-to-image translation. NeurIPS, 2017. §5.2 and Figure 6: average pairwise LPIPS as a diversity score.
• J. Johnson, A. Alahi, L. Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. ECCV, 2016.
• HSE-LAMBDA, Deep Generative Models course repository: metrics.ipynb.