Total Probability Law and Bayes’ Rule.
Example: A ridiculous game...
Box 1 (B1) has two gold coins and one penny. Box 2 (B2) has one gold coin and two pennies.
Box 3 (B3) has four gold coins and one penny.
• Player 1 rolls a fair 6-sided die. Call the outcome D. Player 1 picks a box according to the
outcome of the die roll as follows:


1, 2 → pick B1
D = 3, 4, 5 → pick B2


6 → pick B3.
Then, player 1 selects a coin at random from the chosen box and tells player 2 whether the
coin is a gold coin or a penny.
• Player 2 then guesses which box the coin came from.
• If player 2 guesses correctly, then player 2 keeps the selected coin. Otherwise, player 1
keeps the chosen coin.
a.) What is the probability that player 1 selects a gold coin?
b.) What box will player 2 guess if player 1 selects a gold coin?
c.) What is the probability that player 2 guesses the correct box?
d.) Would you prefer to be player 1 or player 2?
a.) A tree diagram shows all possible outcomes of the two-step procedure.
- There are 3 distinct ways to get a gold coin:
E1 = (B1, G),
E2 = (B2, G),
and E3 = (B3, G).
- E1 , E2 , and E3 are mutually disjoint.
- E1 ∪ E2 ∪ E2 = G
- Axiom (iii) → P (G) = P (E1 ∪ E2 ∪ E2 ) = P (E1 ) + P (E2 ) + P (E3 )
- By definition of conditional probability,
P (E1 ) = P (B1 and G)
= P (G|B1)P (B1)
2 1
2
= ( )( ) =
3 3
9
- Likewise,
1 1
P (E2 ) = P (G|B2)P (B2) = ( )( )
3 2
4 1
P (E3 ) = P (G|B3)P (B3) = ( )( )
5 6
- Then, P (G) =
2
9
+ 16 +
4
30
≈ .522.
*** We just used the Law of Total Probability to compute the probability of a gold coin.
Definition. A collection of events B1 , . . . Bk is called a cover or partition of Ω if
(i) the events are disjoint (Bi ∩ Bj = ∅ for i 6= j), and
(ii) the union of the events is Ω (∪ki=1 Bi = Ω).
• If we represent a multi-step procedure with a tree diagram, then the branches of the tree
are a cover.
• We can also represent a cover using a Venn diagram.
Theorem.
Law of Total Probability: If the collection of events B 1 , . . . , Bk is a cover of Ω, and A is an
event, then
P (A) =
k
X
P (A|Bi )P (Bi ).
i=1
Proof of the Law of Total Probability:
• By definition of conditional probability P (A|Bi )P (Bi ) = P (A ∩ Bi )
• Because B1 , . . . , Bk partition Ω, the events A ∩ B1 , . . . A ∩ Bk are disjoint, and ∪ki=1 Ai = A.
P
P
• By Axiom (iii), P (A) = ki=1 P (A ∩ Bi ) = ki=1 P (A|Bi )P (Bi ).
b.) I tell you that I got a gold coin. Which box do you think it came from?
We want to compute P (Bj |G), j = 1, 2, 3 and pick the highest one.
By definition of conditional probability,
P (Bj ∩ G)
P (G)
P (G|Bj )P (Bj )
=
P (G)
P (Bj |G) =
=
P (G|Bj )P (Bj )
P (G|B1 )P (B1 ) + P (G|B2 )P (B2 ) + P (G|B3 )P (B3 )
Specifically,
P (B1 |G) =
P (B2 |G) =
P (B3 |G) =
To figure out these probabilities, we used Bayes’ rule.
Theorem.
Bayes’ Rule: If B1 , . . . , Bk is a cover or partition of Ω, and A is an event, then
P (A|Bj )P (Bj )
P (Bj |A) = Pk
.
j=1 P (A|Bj )P (Bj )
Proof:
P (Bj |A) = P (Bj ∩ A)
P (A|Bj )P (Bj )
=
P (A)
P (A|Bj )P (Bj )
= Pk
.
j=1 P (A|Bj )P (Bj )
We can represent Bayes’ rule with tree diagrams and Venn diagrams as well.
Read Section 1.7 (Hofmann notes).
Example 1.7.3: (Hofmann notes)
A given lot of chips contains 2% defective chips. Each chip is tested before delivery.
However, the tester is not wholly reliable:
P ( “tester says chip is good” | “chip is good” ) = 0.95
P ( “tester says chip is defective” | “chip is defective” ) = 0.94
If the test device says the chip is defective, what is the probability that the chip actually is
defective?
P ( chip is defective
{z
}
|
|
:=Cd
tester says it’s defective ) = P (Cd |Td )
|
{z
}
Bayes’ Rule, use Cd ,C̄d as cover
:=Td
P (Td |Cd )P (Cd )
=
P (Td |Cd )P (Cd ) + P (Td |C̄d )P (C̄d )
0.94 · 0.02
=
= 0.28.
0.94 · 0.02 + (1 − P (T¯d |C̄d ) ·0.98
|
{z
}
=
0.05
=