Conditional probability and Bayes' formula

The axioms assign probabilities to all events before anything is known. Then information arrives: a test comes back positive, the first card is an ace, a user has already clicked on two links. The opening note handled one such case by drawing a tree of 1,000 people and counting. This note turns that tree into formulas that work for any probability: the conditional probability, the chain rule, the law of total probability and Bayes’ formula. It ends with independence, the assumption behind almost every product of probabilities in the course, and with what happens when it is assumed wrongly.

Notation. $P(A \mid B)$ is the probability of $A$ given $B$. Events $B_1, \dots, B_n$ form a partition of $\Omega$ if they are disjoint, together make up $\Omega$, and each has $P(B_i) > 0$. The odds of an event are $O(A) = P(A) / P(A^c)$, so odds of $1 : 99$ mean a probability of $0.01$.

Conditioning shrinks the sample space#

Learning that $B$ happened rules out every outcome outside $B$. What remains is $B$ itself, and the probabilities inside it are rescaled so that they add up to 1 again:

$$P(A \mid B) = \frac{P(A \cap B)}{P(B)}, \qquad P(B) > 0 .$$

With two dice, let $A$ be “the first die shows 3” and $B$ “the sum is 8”:

Figure

Conditioning shrinks the sample space

Five equally likely outcomes give the sum 8, and one of them has a 3 on the first die, so $P(A \mid B) = 1/5$. Six outcomes have a 3 on the first die, and one of them sums to 8, so $P(B \mid A) = 1/6$. The two conditional probabilities share the numerator $P(A \cap B) = 1/36$ and differ in the denominator. Confusing $P(A \mid B)$ with $P(B \mid A)$ is the most common error in reasoning with evidence; the base-rate example was exactly that error, with $P(+ \mid \text{sick}) = 0.9$ read as $P(\text{sick} \mid +)$.

For a fixed $B$, the function $A \mapsto P(A \mid B)$ satisfies the three axioms, so it is a probability in its own right, and every rule of the previous note holds conditionally: $P(A^c \mid B) = 1 - P(A \mid B)$, the union formula, the union bound.

The chain rule#

Multiplying the definition out gives the probability of “both” as a sequence of steps:

$$P(A \cap B) = P(A)\, P(B \mid A), \qquad P(A_1 \cap \dots \cap A_n) = P(A_1)\, P(A_2 \mid A_1) \cdots P(A_n \mid A_1 \cap \dots \cap A_{n-1}) .$$

The probability that the first two cards from a shuffled deck are both aces is $\frac{4}{52} \cdot \frac{3}{51} = \frac{1}{221} \approx 0.0045$: the second factor is the probability of an ace given that one ace is gone. The birthday product of the counting note is the same rule, each factor the probability that the next birthday is new given that the previous ones were all different. And a language model is the chain rule applied to text: $P(w_1, \dots, w_n) = \prod_t P(w_t \mid w_1, \dots, w_{t-1})$, with a model for each conditional, as in the n-gram model.

The law of total probability#

A probability that is hard to compute directly often becomes easy after splitting on what one would like to know. For a partition $B_1, \dots, B_n$,

$$P(A) = \sum_{i} P(A \mid B_i)\, P(B_i) .$$

The share of positive tests splits on the patient’s status: $P(+) = 0.9 \cdot 0.01 + 0.09 \cdot 0.99 = 0.0981$.

Monty Hall. A car is behind one of three doors, goats behind the other two. A player picks door 1. The host, who knows where the car is, opens one of the other two doors with a goat behind it and offers the player a switch. Splitting on where the car is: with probability $1/3$ it is behind door 1, and switching loses; with probability $2/3$ it is behind one of the other two, the host has to open the other one, and switching wins. So switching wins with probability $2/3$. Most people answer $1/2$: they condition only on the fact that the opened door hides a goat and forget how that information was produced. If the host opened one of the other doors at random and it happened to show a goat, switching would indeed win with probability only $1/2$. The rule by which information is revealed is part of the model.

Bayes’ formula#

Writing $P(A \cap B)$ in both orders of the chain rule and dividing gives the rule for reversing a conditional:

$$P(B \mid A) = \frac{P(A \mid B)\, P(B)}{P(A)} , \qquad P(A) = \sum_i P(A \mid B_i)\, P(B_i) .$$

$P(B)$ is the prior, $P(A \mid B)$ the likelihood of the evidence, $P(B \mid A)$ the posterior. For the test, $P(\text{sick} \mid +) = 0.9 \cdot 0.01 / 0.0981 = 0.0917$, the 9 in 98 of the opening note (whose tree rounded 89.1 false positives to 89).

With only two hypotheses, dividing Bayes’ formula for $B$ by the same formula for $B^c$ cancels the denominator and leaves the odds form:

$$O(B \mid A) = O(B) \cdot \frac{P(A \mid B)}{P(A \mid B^c)} .$$

Posterior odds are prior odds times the likelihood ratio of the evidence. For a positive test the ratio is $0.9 / 0.09 = 10$, so odds of $1 : 99$ become $10 : 99$, a probability of $10/109 = 0.0917$. A second positive test, independent of the first given the patient’s status, multiplies the odds by 10 again:

Figure

Every positive test is the same step in log-odds

On the logarithmic scale every independent piece of evidence adds its log likelihood ratio. This is how a naive Bayes classifier combines words and why a logistic regression outputs log-odds. It also gives a practical rule, under one assumption. Suppose the base rate changes between training and deployment, from 1% positives to 5%, and nothing else does: the inputs of each class are distributed as before, $P_{\text{new}}(x \mid y) = P_{\text{old}}(x \mid y)$ (label shift), with the same labels and no inputs outside the training range. Then the likelihood ratio of every input is unchanged, and a classifier calibrated on the training data has log-odds that shift by the same constant, $\log(5/95) - \log(1/99) \approx 1.65$, for every input. A change in prevalence alone does not justify the shift. If the cases in the new population are milder, so that the test detects only half of them, sensitivity drops to $0.5$, the likelihood ratio of a positive falls from 10 to $5.6$, and a positive means $P(\text{sick} \mid +) = 0.23$, not the $0.34$ that the constant shift predicts.

Independence#

Events $A$ and $B$ are independent if

$$P(A \cap B) = P(A)\, P(B) ,$$

which for $P(B) > 0$ says $P(A \mid B) = P(A)$: learning $B$ does not change the probability of $A$. Two rolls of a die are independent by construction, so $P(\text{3, then 5}) = \frac16 \cdot \frac16$. Independence is not disjointness: two disjoint events with positive probabilities are strongly dependent, since learning that one happened makes the other impossible.

Several events. $A_1, \dots, A_n$ are independent if the product rule holds for every subcollection, not only for pairs. Toss two fair coins and let $A$ be “the first is heads”, $B$ “the second is heads”, $C$ “the two agree”. Every pair is independent: $P(A \cap C) = P(\mathrm{HH}) = 1/4 = P(A) P(C)$, and the same for the other pairs. But $P(A \cap B \cap C) = 1/4$, not $1/8$: any two of the events determine the third.

Conditional independence. Events $A$ and $B$ are conditionally independent given $C$, with $P(C) > 0$, if $P(A \cap B \mid C) = P(A \mid C)\, P(B \mid C)$. Two runs of the test are independent given each health status, sick and healthy, and still not independent: $P(\text{second} + \mid \text{first} +) = 0.164$, against $P(+) = 0.098$, because a first positive makes the disease more likely, and with it a second positive. Conditional independence given a hidden cause and dependence without it is the normal situation in data; the naive Bayes classifier assumes that the words of a document are conditionally independent given its class, which is false and often good enough.

When independence is assumed wrongly. In 1999 Sally Clark was convicted in England of murdering her two infant sons. An expert witness told the court that the probability of two sudden infant deaths in a family like hers was 1 in 73 million, obtained by squaring his estimate of 1 in 8,543 for one death. Two errors are in that number. The squaring assumes the two deaths are independent, while genetic and environmental factors shared by a family make a second death more likely after a first. And the number is $P(\text{two deaths} \mid \text{innocent})$, which the jury could read as $P(\text{innocent} \mid \text{two deaths})$: the relevant comparison is between the two explanations, and a double murder of infants is also extremely rare. The Royal Statistical Society published a statement on the misuse of statistics in the case in 2001, and the conviction was overturned in 2003.

What changed#

	Without conditioning	With conditioning
Probability after evidence	recount a frequency tree	$P(A \mid B) = P(A \cap B) / P(B)$
Probability of a sequence	count ordered outcomes	chain rule
A probability that is hard to compute	—	total probability: split on the unknown
Reversing a conditional	—	Bayes’ formula; odds × likelihood ratio
Multiplying probabilities	assumed	justified by independence, or wrong

Limitations#

• Conditioning on an event of probability zero is not defined here. “Given that the measured value is exactly 0.3” has probability 0 for a continuous quantity; it needs conditional densities, introduced with joint distributions and conditional expectation.
• The conditioning event must be what was actually learned, including how. Monty Hall gives $2/3$ or $1/2$ depending on the host’s rule.
• Independence is usually an assumption. It can rarely be checked from a few observed events, and a wrong independence assumption produces small probabilities that are off by orders of magnitude.

What you should remember#

• $P(A \mid B) = P(A \cap B) / P(B)$: restrict to $B$ and rescale. $P(A \mid B)$ and $P(B \mid A)$ are different numbers.
• Chain rule: $P(A_1 \cap \dots \cap A_n) = \prod_k P(A_k \mid A_1 \cap \dots \cap A_{k-1})$.
• Total probability: $P(A) = \sum_i P(A \mid B_i) P(B_i)$ over a partition.
• Bayes: posterior ∝ likelihood × prior; in odds, posterior odds = prior odds × likelihood ratio, and independent pieces of evidence add in log-odds.
• Independence means $P(A \cap B) = P(A) P(B)$; for several events it must hold for every subcollection, and conditional independence does not imply independence.

Code#

The numbers of the note, checked exactly with fractions or by simulation:

import numpy as np
from fractions import Fraction

# two dice: P(first is 3 | sum is 8) and P(sum is 8 | first is 3)
dice = [(a, b) for a in range(1, 7) for b in range(1, 7)]
sum8 = [d for d in dice if sum(d) == 8]
first3 = [d for d in dice if d[0] == 3]
print(Fraction(sum(d[0] == 3 for d in sum8), len(sum8)), Fraction(sum(sum(d) == 8 for d in first3), len(first3)))   # 1/5 1/6

# the test: total probability, Bayes, the odds form for 1, 2, 3 positive tests
prior, sens, fpr = 0.01, 0.9, 0.09
p_pos = prior * sens + (1 - prior) * fpr
print(round(p_pos, 4), round(prior * sens / p_pos, 4))            # 0.0981 0.0917
odds = prior / (1 - prior)
for k in range(1, 4):
    odds *= sens / fpr
    print(k, round(odds / (1 + odds), 3))                          # 1 0.092, 2 0.503, 3 0.91
p_both = prior * sens**2 + (1 - prior) * fpr**2
print(round(p_both / p_pos, 3))                                    # 0.164 = P(second + | first +)

# pairwise but not mutually independent: two coins, "first is H", "second is H", "they agree"
coins = [(x, y) for x in 'HT' for y in 'HT']
P = lambda event: Fraction(sum(map(event, coins)), len(coins))
first = lambda c: c[0] == 'H'; second = lambda c: c[1] == 'H'; agree = lambda c: c[0] == c[1]
print(P(lambda c: first(c) and agree(c)), P(first) * P(agree))                                 # 1/4 1/4
print(P(lambda c: first(c) and second(c) and agree(c)), P(first) * P(second) * P(agree))       # 1/4 1/8

# Monty Hall: the host knows where the car is and always opens a door with a goat
rng = np.random.default_rng(0)
car, pick = rng.integers(0, 3, (2, 100_000))
print(round(float((car == pick).mean()), 3), round(float((car != pick).mean()), 3))   # stay wins 0.332, switch wins 0.668

Next#

So far events have been yes-or-no statements. Most random quantities are numbers: the number of heads in 10 flips, the number of clicks in an hour, the waiting time until a failure. The next note introduces random variables and the discrete distributions that appear everywhere: Bernoulli, binomial, geometric, Poisson and the hypergeometric count of the card example.

References#

• J. K. Blitzstein, J. Hwang. Introduction to Probability, 2nd ed. CRC Press, 2019. Chapter 2 (conditional probability, Bayes’ rule, the law of total probability, independence, the Monty Hall problem, Simpson’s paradox).
• W. Feller. An Introduction to Probability Theory and Its Applications, Vol. 1, 3rd ed. Wiley, 1968. Chapter V (conditional probability, stochastic independence).
• S. Selvin. A Problem in Probability. Letter to the editor, The American Statistician, 29(1), 1975. The Monty Hall problem.
• G. Gigerenzer, U. Hoffrage. How to Improve Bayesian Reasoning Without Instruction: Frequency Formats. Psychological Review, 102(4), 1995.
• Royal Statistical Society. Statement on the use of statistics in the Sally Clark case, October 2001.
• R. Hill. Multiple sudden infant deaths – coincidence or beyond coincidence? Paediatric and Perinatal Epidemiology, 18(5), 2004.