Grade 12

Conditional probability and Bayes' theorem

How the probability of an event changes once we know another event has occurred.

IntuitionIntuition: new information changes the odds

Roll two fair six-sided dice and let $A$ be the event that the sum of the two dice equals $8$ , so ordinarily $P(A) = \frac{5}{36}$ , since exactly the five pairs $(2,6), (3,5), (4,4), (5,3), (6,2)$ out of $36$ equally likely pairs give that sum. Now suppose someone peeks at the first die and reports that it shows a $5$ ; call this event $B$ . Knowing $B$ shrinks the sample space from all $36$ pairs down to only the $6$ pairs where the first die is $5$ , and among those $6$ pairs only $(5,3)$ has sum $8$ . The extra information changes the probability of $A$ from $\frac{5}{36}$ to $\frac{1}{6}$ — this updated value, written $P(A \mid B)$ , is called the conditional probability of $A$ given $B$ .

A probability tree for a two-stage experiment: the first branching splits the sample space by which condition holds (for example, which urn is picked or whether a patient is sick), and each second-level branch is labeled with a conditional probability such as

P(A \mid B)

. Multiplying along a path gives

P(A \cap B)

, and summing over all paths that end in

A

gives the total probability

P(A)

SchoolThe definition of conditional probability

Definition: Conditional probability

For two events $A$ and $B$ in the same sample space with $P(B) > 0$ , the conditional probability of $A$ given $B$ is $P(A \mid B) = \frac{P(A \cap B)}{P(B)}$ . Intuitively, once we know $B$ has occurred, the sample space is restricted to $B$ , and $P(A \mid B)$ measures what fraction of that restricted space is also covered by $A$ .

P(A \mid B) = \frac{P(A \cap B)}{P(B)}

Here $P(A \cap B)$ is the probability that both $A$ and $B$ happen, and $P(B)$ is the probability of the conditioning event, which must be positive since dividing by $P(B) = 0$ is undefined. Rearranging the definition gives the multiplication rule $P(A \cap B) = P(B) \cdot P(A \mid B)$ , and by symmetry also $P(A \cap B) = P(A) \cdot P(B \mid A)$ whenever $P(A) > 0$ ; this rule extends naturally to chains of several events.

P(A \cap B) = P(B) \cdot P(A \mid B) = P(A) \cdot P(B \mid A)

A special case worth naming is independence: $A$ and $B$ are independent exactly when learning $B$ gives no new information about $A$ , that is $P(A \mid B) = P(A)$ , which is equivalent to $P(A \cap B) = P(A) \cdot P(B)$ . When $A$ and $B$ are not independent, the two conditional probabilities $P(A \mid B)$ and $P(B \mid A)$ are generally different numbers, and confusing them is one of the most common mistakes in probability.

Conditional probability in special cases
Relationship between $A$ and $B$	Value of $P(A \mid B)$	Reason
Independent	$P(A \mid B) = P(A)$	$B$ carries no information about $A$
$A$ implies $B$ (that is $A \subseteq B$ )	$P(A \mid B) = \frac{P(A)}{P(B)}$	Since $A \cap B = A$
$A$ and $B$ mutually exclusive	$P(A \mid B) = 0$	Since $A \cap B = \varnothing$

UndergraduateThe law of total probability and Bayes' theorem

Theorem: The law of total probability

Let $B_1, B_2, \ldots, B_n$ be a partition of the sample space $\Omega$ , meaning $B_i \cap B_j = \varnothing$ for $i \neq j$ , $B_1 \cup B_2 \cup \cdots \cup B_n = \Omega$ , and $P(B_i) > 0$ for every $i$ . Then for any event $A$ , $P(A) = \sum_{i=1}^{n} P(B_i) \cdot P(A \mid B_i)$ .

Why is it true?

Since the $B_i$ partition $\Omega$ , event $A$ is automatically chopped into $n$ disjoint pieces, one inside each $B_i$ ; adding up the probability of each piece, expressed via the multiplication rule, recovers all of $P(A)$ without double-counting or missing anything.

Proof

Step 1 (partition $A$ using the $B_i$ ): define $A_i = A \cap B_i$ for each $i = 1, \ldots, n$ . Since the $B_i$ are pairwise disjoint, so are the $A_i$ , and since $B_1 \cup \cdots \cup B_n = \Omega \supseteq A$ , every outcome of $A$ lies in exactly one $A_i$ , so $A = A_1 \cup A_2 \cup \cdots \cup A_n$ with the union disjoint.

Step 2 (add probabilities of disjoint pieces): because probability is additive over pairwise disjoint events, $P(A) = P(A_1) + P(A_2) + \cdots + P(A_n) = \sum_{i=1}^{n} P(A \cap B_i)$ .

Step 3 (rewrite each term with the multiplication rule): for each $i$ , the multiplication rule gives $P(A \cap B_i) = P(B_i) \cdot P(A \mid B_i)$ , which is well defined since $P(B_i) > 0$ .

Step 4 (substitute back): replacing every $P(A \cap B_i)$ in Step 2 by $P(B_i) \cdot P(A \mid B_i)$ gives exactly $P(A) = \sum_{i=1}^{n} P(B_i) \cdot P(A \mid B_i)$ , as claimed.

Theorem: Bayes' theorem

Let $B_1, B_2, \ldots, B_n$ be a partition of $\Omega$ with $P(B_i) > 0$ for every $i$ , and let $A$ be an event with $P(A) > 0$ . Then for each $i$ , $P(B_i \mid A) = \frac{P(B_i) \cdot P(A \mid B_i)}{\sum_{j=1}^{n} P(B_j) \cdot P(A \mid B_j)}$ .

Why is it true?

Bayes' theorem reverses the direction of conditioning: it starts from how likely the evidence $A$ is under each hypothesis $B_i$ (often easy to know or measure) and returns how likely each hypothesis is once the evidence $A$ has actually been observed (usually what we really want to know), by re-weighting each prior probability $P(B_i)$ according to how well it explains $A$ .

Proof

Step 1 (write $P(B_i \cap A)$ two ways): by the multiplication rule applied in both directions, $P(B_i \cap A) = P(B_i) \cdot P(A \mid B_i)$ and also $P(B_i \cap A) = P(A) \cdot P(B_i \mid A)$ , since $P(A) > 0$ .

Step 2 (equate the two expressions and solve): setting the two expressions for $P(B_i \cap A)$ equal gives $P(A) \cdot P(B_i \mid A) = P(B_i) \cdot P(A \mid B_i)$ , and dividing both sides by $P(A)$ yields $P(B_i \mid A) = \frac{P(B_i) \cdot P(A \mid B_i)}{P(A)}$ .

Step 3 (expand the denominator with the law of total probability): since $B_1, \ldots, B_n$ partition $\Omega$ , the law of total probability gives $P(A) = \sum_{j=1}^{n} P(B_j) \cdot P(A \mid B_j)$ .

Step 4 (substitute to finish): replacing $P(A)$ in Step 2 by this sum gives exactly $P(B_i \mid A) = \frac{P(B_i) \cdot P(A \mid B_i)}{\sum_{j=1}^{n} P(B_j) \cdot P(A \mid B_j)}$ , which is Bayes' theorem.

UndergraduateReal-World Applications and Worked Examples

Bayes' theorem is the mathematical core of medical diagnosis (updating the chance of a disease after a test result), spam filtering (updating the chance an email is spam after seeing its words), legal reasoning (updating the chance of guilt after new evidence), and machine learning classifiers (updating the chance of a label after observing features). The two examples below apply the law of total probability and Bayes' theorem to concrete numbers, including the classic paradox of false positives in medical testing.

Example: Which urn was it drawn from?

There are two urns. Urn I has $3$ red balls and $7$ blue balls; urn II has $6$ red balls and $4$ blue balls. An urn is chosen at random with $P(\text{Urn I}) = 0.4$ and $P(\text{Urn II}) = 0.6$ , and then one ball is drawn from the chosen urn. The ball turns out to be red. Find the probability that it came from Urn I.

Solution

Step 1: let $R$ be the event the drawn ball is red. Within Urn I, $P(R \mid \text{Urn I}) = \frac{3}{10}$ , and within Urn II, $P(R \mid \text{Urn II}) = \frac{6}{10}$ .

Step 2: since Urn I and Urn II partition the choice of urn, the law of total probability gives $P(R) = P(\text{Urn I}) \cdot P(R \mid \text{Urn I}) + P(\text{Urn II}) \cdot P(R \mid \text{Urn II}) = 0.4 \cdot 0.3 + 0.6 \cdot 0.6 = 0.12 + 0.36 = 0.48$ .

Step 3: Bayes' theorem gives $P(\text{Urn I} \mid R) = \frac{P(\text{Urn I}) \cdot P(R \mid \text{Urn I})}{P(R)} = \frac{0.12}{0.48} = \frac{1}{4}$ , so despite Urn I being chosen less often, seeing red still leaves only a $25\%$ chance it came from Urn I, because Urn II produces red balls twice as reliably.

Example: The false-positive paradox in medical testing

A disease affects $1\%$ of a population, so $P(D) = 0.01$ for a randomly chosen person. A test for the disease has sensitivity $99\%$ , meaning $P(+ \mid D) = 0.99$ , and specificity $95\%$ , meaning $P(- \mid D^c) = 0.95$ (so the false-positive rate is $P(+ \mid D^c) = 0.05$ ). A random person tests positive. Find $P(D \mid +)$ , the probability that they actually have the disease.

Solution

Step 1: $D$ and $D^c$ (has the disease, does not have the disease) partition the population, with $P(D) = 0.01$ and $P(D^c) = 0.99$ .

Step 2: apply the law of total probability to the event of testing positive, $P(+) = P(D) \cdot P(+ \mid D) + P(D^c) \cdot P(+ \mid D^c) = 0.01 \cdot 0.99 + 0.99 \cdot 0.05 = 0.0099 + 0.0495 = 0.0594$ .

Step 3: Bayes' theorem gives $P(D \mid +) = \frac{P(D) \cdot P(+ \mid D)}{P(+)} = \frac{0.0099}{0.0594} = \frac{1}{6}$ , only about $16.7\%$ .

Step 4: this is the false-positive paradox: even with a test that is $99\%$ sensitive and $95\%$ specific, most positive results are still false alarms, because the disease is rare so the $99\%$ of healthy people contribute far more false positives ( $0.0495$ ) than the $1\%$ of sick people contribute true positives ( $0.0099$ ).

Given $P(A) = 0.4$ , $P(B) = 0.5$ , and $P(A \cap B) = 0.2$ , find $P(A \mid B)$ .

A screening test has sensitivity $99\%$ and specificity $99\%$ . In a population where $1\%$ of people have the disease, a randomly chosen person tests positive. What is $P(D \mid +)$ , rounded to the nearest whole percent?

Let $B_1, B_2, \ldots, B_n$ be a partition of the sample space $\Omega$ with $P(B_i) > 0$ for every $i$ . Which formula correctly gives $P(A)$ for any event $A$ ?

Events $A$ and $B$ satisfy $P(B) > 0$ . Which equality is exactly the definition of $A$ and $B$ being independent?