Variational Autoencoders

A variational autoencoder is a latent-variable model with a learned approximate posterior. It turns otherwise difficult posterior inference into a differentiable optimization problem.

Evidence lower bound

For observation $x$ and latent variable $z$, introduce $q_\phi(z\mid x)$ as an approximation to $p_\theta(z\mid x)$. Then

$$\log p_\theta(x) \ge \mathbb{E}_{q_\phi(z\mid x)}[\log p_\theta(x\mid z)] - D_{\mathrm{KL}}(q_\phi(z\mid x)\,\|\,p(z)).$$

The first term rewards reconstruction under the decoder. The second regularizes the approximate posterior toward the prior. Their balance shapes both fidelity and latent-space organization.

Reparameterization

For a diagonal Gaussian posterior,

$$z = \mu_\phi(x) + \sigma_\phi(x) \odot \epsilon, \qquad \epsilon \sim \mathcal{N}(0,I).$$

Randomness is isolated in $\epsilon$, leaving a differentiable path through $\mu_\phi$ and $\sigma_\phi$. This is the reparameterization trick.