Course / 01
Probability Theory
Reasoning exactly about outcomes we cannot predict: from counting to the limit theorems, and how they become the losses, estimates and guarantees of machine learning.
Before you start: school algebra, sums and limits. Derivatives and integrals are used from Continuous distributions on; matrix products, inverses and eigenvectors from The multivariate Gaussian on. No probability is assumed. Proof-heavy detours inside notes, such as the Vitali set or the characteristic-function proof of the central limit theorem, are marked as optional depth and can be skipped on a first reading.
- 01
Foundations
What probability is, how to count it, and how information changes it.
After this section: Tell an event from a model and from evidence, and update a probability with Bayes' formula.
4 of 4 notes published - 02
Random Variables
Numbers produced by chance, and how to summarize them.
After this section: Pick a distribution for a quantity, find the distribution of a function of it, and compute its mean and variance.
4 of 4 notes published - 03
Several Random Variables
How random quantities depend on each other.
After this section: Work with joint and conditional distributions and the Gaussian in several dimensions; explain why least squares predicts a conditional mean and why zero covariance is not independence in general.
3 of 3 notes published - 04
Limit Theorems
Why averages work, how fast, and why the Gaussian is everywhere.
After this section: Say how far an average can be from its expectation, exactly, by a finite-sample bound or asymptotically, and under which conditions.
3 of 3 notes published - 05
From Probability to Statistics
From data back to the distribution that produced it.
After this section: Estimate a parameter by maximum likelihood or Bayes, attach an interval to it and run a test, keeping effect size and decision error apart.
5 of 5 notes published - 06
Random Processes and Sampling
Random systems that evolve, and sampling from distributions we cannot write down.
After this section: A change of task: the distribution is known, perhaps only up to a constant, but its integrals cannot be computed and it cannot be sampled directly. Compute its predictions with Markov chains, Monte Carlo and MCMC, and check whether a chain has converged.
3 of 3 notes published