# Student Probability Notes ```Lecture 5: Basic Probability Theory
Donglei Du
(ddu@unb.edu)
E3B 9Y2
Donglei Du (UNB)
1 / 55
1
Probability Theory
What is probability?
Random Experiment (RE)
Set/events operations
How to Interpret Probabilities?
Rules of Counting
Donglei Du (UNB)
2 / 55
Layout
1
Probability Theory
What is probability?
Random Experiment (RE)
Set/events operations
How to Interpret Probabilities?
Rules of Counting
Donglei Du (UNB)
3 / 55
What is probability?
Probability is a measure of the likelihood that an event in the future
will happen.
It can only assume a value between 0 and 1;
A value near zero means the event is not likely to happen;
A value near one means it is likely to happen.
Donglei Du (UNB)
4 / 55
Random Experiment (RE)
RE is a process that satisfies the following propositions:
the process can be repeated as many trials as you want
the outcome of any trial is uncertain
well-defined set of possible outcomes
each outcome has a probability associated with it
Donglei Du (UNB)
5 / 55
An example
Toss a coin once:
This process can be repeated as many times as you want (repetitive
nature)
Nobody knows whether Head or Tail will appear in any particular
tossing (uncertain outcome)
However, we know either Head or Tail must appear (Well-defined
sample space)
Now what is the chance you will get a Head if you toss a coin now?
Donglei Du (UNB)
6 / 55
Some Terminologies
Outcome: is a particular result of a random experiment
Sample Space: is the collection or set of all the possible outcomes of
a random experiment
Event: is the collection of one or more outcomes of an experiment
Donglei Du (UNB)
7 / 55
Example
Toss one coin in very trial
The sample space is S = {H, T}.
An event is the occurrence of head: E = {h}.
Flip two coins in very trial
The sample space has two outcomes
S = {(H,H), (H,T), (T,H), (T,T)}.
An event is the occurrence of a head in the first coin:
E = {(H, H), (H, T )}
Roll one die in very trial
The sample space contains 6 outcomes S = {1, . . . , 6}.
An event is the occurrence of an even number: E = {2, 4, 6}.
Roll two dice in very trial
The sample space contains the 36 outcomes S = {(1, 1), . . . , (6, 6)}.
An event is the occurrence of two dice with sum equal to 4:
E = {(1, 3), (2, 2), (3, 1)}
Donglei Du (UNB)
8 / 55
Venn Diagram
A Venn diagram or set diagram is a diagram that shows all possible
logical relations between a finite collection of sets.
The Venn diagram for the above die example.
Donglei Du (UNB)
1
3
5
2
4
6
9 / 55
An example
Toss two coins in very trial:
This process can be repeated as many times as you want (repetitive
nature)
Every trial contains the outcome of two coins
Origin
First
Flip
p
H
T
Donglei Du (UNB)
Second
Flip
p
H
HH
T
H
HT
TH
T
TT
10 / 55
Set operations
Union
Intersection:
Complement:
Donglei Du (UNB)
11 / 55
Union
Sample Space
A B
A
B
Union: The union event of two events A and B is denoted as A ∪ B,
which consists of all outcomes in either A or B or in both. Namely
event A ∪ B occurs if either A or B occurs.
Example: Roll a die: E1 = {1, 3, 5} and E2 = {1, 2, 3}. Then
E1 ∪ E2 = {1, 2, 3, 5}
Donglei Du (UNB)
12 / 55
Intersection
Sample Space
A
A B
B
Intersection: The intersection event of two events A and B is denoted
as A ∩ B, which consists of all outcomes in both A and B or in both.
Namely event A ∩ B occurs if both A and B occur.
Example: Roll a die: E1 = {1, 3, 5} and E2 = {1, 2, 3}. Then
E1 ∩ E2 = {1, 3}
Two events A and B are mutually exclusive if A ∩ B = ∅.
Donglei Du (UNB)
13 / 55
Complement
Sample Space
A
A
Complement: The complement event of an event A is denoted as Ā,
which consists of all outcomes that are not in A. Namely event Ā
occurs if A does not occur.
Example: Roll a die: E1 = {1, 3, 5}. Then Ē1 = {2, 4, 6}
Donglei Du (UNB)
14 / 55
Probabilities defined on events
For any random experiment with sample space S, the probability of
any event is P (E) satisfying
(i) 0 ≤ P (E) ≤ 1.
(ii) P (S) = 1.
(iii) (special addition rule) For any pair-wise mutually exclusive events
E1 , E2 , . . .
!
∞
∞
[
X
P
En =
P (En ).
n=1
Donglei Du (UNB)
i=1
15 / 55
Examples
Toss one fair coin in very trial: If we assume a head is equally likely to
appear as a tail, then
1
2
Toss one biased coin in very trial: If we assume a head is twice as
likely to appear as a tail, then
P (H) = P (T ) =
1
2
P (H) = , P (T ) = .
3
3
Toss one fair die in very trial: If we assume all sides are equally likely
to appear, then
1
P (i) = , i = 1, . . . , 6.
6
From (iii) in the previous slide, we must have
1
P ({1, 3, 5}) = P (1) + P (2) + P (3) = .
2
Donglei Du (UNB)
16 / 55
Probability Distributions defined on sample space
A (discrete) probability distribution on a given sample space is a table
of all disjoint outcomes and their associated probabilities.
Example: Toss two fair coins in very trial: The table below is the
probability distribution for the sample space
S = {HH, HT, T H, T T }:
Outcome
HH
HT
TH
TT
Donglei Du (UNB)
Prob.
0.25
0.25
0.25
0.25
17 / 55
Rule of Probability
Complement Rule
Multiplication Rule
Donglei Du (UNB)
18 / 55
Rule of Complement
For any event A:
P (Ā) = 1 − P (A)
Donglei Du (UNB)
19 / 55
For any two events A and B:
P (A ∪ B) = P (A) + P (B) − P (A ∩ B).
If events A and B are mutually exclusive:
P (A ∪ B) = P (A) + P (B).
The above can be extended to any number of mutually exclusive
events A1 , . . . , An :
P (A1 ∪ . . . ∪ An ) = P (A1 ) + . . . + P (An ).
Donglei Du (UNB)
20 / 55
Toss two fair coins in very trial and assume that the four possible
outcomes S = {HH, HT, T H, T T } are equally likely to happen.
Let A = {HH, HT } be the event that the first coin falls head, and
B = {HH, T H} be the event that the second coin falls head.
Problem: Find the probability that either the first coin falls head or
Solution:
P (A ∪ B) = P (A) + P (B) − P (A ∩ B) =
Donglei Du (UNB)
3
1 1 1
+ − = .
2 2 4
4
21 / 55
Rule of Multiplication
Joint probability
Conditional probability
Donglei Du (UNB)
22 / 55
Joint probability
Joint probability: For two events A and B defined on the same
sample space, the joint probability of events A and B is P (A ∩ B).
Example: Roll one die in very trial. Let A = {1, 2, 3} and
B = {1, 3, 5} be two events. Then the joint probability of A and B is
P (A ∩ B) = P ({1, 3}) =
Donglei Du (UNB)
2
1
= .
6
3
23 / 55
Represent Joint probability as contingency table
Joint probability distribution is given in a tabular form, called
contingency table. The probability distribution for each variable is
called the marginal distribution.
Example: A survey of undergraduate students in the Faculty of
Business Management at UNB revealed the following regarding the
gender and majors of the students:
Male
Female
Marginal
Accoun.
150/750
175/750
325/750
IB
150/750
160/750
310/750
HR
50/750
65/750
115/750
Marginal
350/750
400/750
1
Example: What is the Probability of selecting a Female Accounting
student?
Solution:
P (F ∩ A) = 175/750 = 23.33%
Donglei Du (UNB)
24 / 55
Conditional probability
Conditional probability: For any event A and B, the probability of
event A, given the occurrence of event B:
P (A|B) =
P (A ∩ B)
.
P (B)
Example: What is the probability of selecting a Female, given that
the person selected is an International Business major?
Solution:
P (F |I) = 160/310 = 51.6%
Compare to the unconditional probability:
P (F ) = 400/750 = 50%
Donglei Du (UNB)
25 / 55
Independent events: two events
Two events A and B are independent if and only if
P (A|B) = P (A)
or equivalently
P (A ∩ B) = P (A) &times; P (B).
Example: the events F = {(F, A), (F, IB), (F, HR)} and
IB = {(F, IB), (M, IB)} are not independent, since
P (F |IB) = 160/310 = 51.6% 6= 50% = 400/750 = P (F )
Donglei Du (UNB)
26 / 55
Independent events: more than two events
n events A1 , . . . , An are independent if and only if for any r ≤ n:
P (A1 ∩ . . . ∩ Ar ) = P (A1 ) &times; . . . &times; P (Ar ).
Example: (Pairwise independent events may not be independent
overall): Roll a four-faced die once. Let A1 = {1, 2}, A2 = {1, 3} and
A3 = {1, 4} be three events. Then
1
4
1
P (A1 ∩ A3 ) = P (A1 ) &times; P (A3 ) =
4
1
P (A2 ∩ A3 ) = P (A2 ) &times; P (A3 ) =
4
1
1
P (A1 ∩ A2 ∩ A3 ) =
6= = P (A1 ) &times; P (A2 ) &times; P (A3 )
4
8
P (A1 ∩ A2 ) = P (A1 ) &times; P (A2 ) =
Donglei Du (UNB)
27 / 55
Rule of Multiplication
For any two events A and B:
P (A ∩ B) = P (A|B)P (B).
The above can be extended to any number of events A1 , . . . , An :
P (A1 ∩ . . . ∩ An ) = P (A1 )P (A2 |A1 ) . . . P (An |A1 ∩ . . . ∩ An−1 ).
If events A and B are independent:
P (A ∩ B) = P (A)P (B).
The above can be extended to any number of independent events
A1 , . . . , A n :
P (A1 ∩ . . . ∩ An ) = P (A1 ) . . . P (An ).
Donglei Du (UNB)
28 / 55
Rule of Multiplication: Example
Example: Draw three cards with replacement i.e., draw one card,
look at it, put it back, and repeat twice more.
Problem: Find the probability of drawing 3 Queens in a row:
Solution: Let Qi (i = 1, 2, 3) be the event that ith draw gives you a
Queen. Then the three events are independent of each other.
P (Q1 ∩ Q2 ∩ Q3 ) = P (Q1 )P (Q2 )P (Q3 ) =
Donglei Du (UNB)
4
4
4
&times;
&times;
≈ 0.00046.
52 52 52
29 / 55
Rule of Multiplication: Example
Example: Draw three cards without replacement i.e., draw one card,
look at it, keep it, and repeat twice more.
Problem: Find the probability of drawing 3 Queens in a row:
Solution: Let Qi (i = 1, 2, 3) be the event that ith draw gives you a
Queen. Then the three events are dependent of each other.
P (Q1 ∩ Q2 ∩ Q3 ) = P (Q1 )P (Q2 |Q1 )P (Q3 |Q1 ∩ Q2 )
4
3
2
=
&times;
&times;
≈ 0.00018.
52 51 50
Donglei Du (UNB)
30 / 55
More examples on rules of probability
Example: Draw one card from a deck of 52 cards
Problem: What is the probability of getting a red card or a Queen?
Solution:
P (R ∪ Q) = P (R) + P (Q) − P (R ∩ Q) =
Donglei Du (UNB)
26 + 4 − 2
7
=
≈ 0.538
52
13
31 / 55
More examples on rules of probability
Example: Two men throw their identical hats into the center of the
room at a party. Then the hats are mixed up and each man randomly
selects a hat.
Problem: What is the probability that none of them selects his own
hat.
Solution: Let Ai (i = 1, 2) be the event that the ith man selects his
own hat. Then the desired probability can be calculated via the
complement rule. The complement event is that at least one man
selects his own hat:
P (A1 ∪ A2 ) = P (A1 ) + P (A2 ) − P (A1 ∩ A2 )
= 0.5 + 0.5 − P (A1 )P (A2 |A1 ) = 1 − 0.5(1) = 0.5
Therefore none of them selects his own hat is
P A1 ∪ A2 = 1 − P (A1 ∪ A2 ) = 1 − 0.5 = 0.5
Donglei Du (UNB)
32 / 55
Another method for solving the above problem
Solution: Let Ai (i = 1, 2) be the event that the ith man selects his
own hat. Therefore none of them selects his own hat is
P Ā1 ∩ Ā2 = P (Ā1 )P (Ā2 |Ā1 ) = 0.5(1) = 0.5.
Donglei Du (UNB)
33 / 55
Bayes’ Theorem
Sample Space
A1  B
A1
An  B
A2  B
A2
……
An
Given an event B, and a set of mutually exclusive and exhaustive
events A1 , . . . , An , and we know the prior probabilities
P (A1 ), . . . , P (An ). Then the posterior is:
P (A1 |B) =
Donglei Du (UNB)
P (B|A1 )P (A1 )
P (B|A1 )P (A1 ) + . . . + P (B|An )P (An )
34 / 55
Bayes’ Theorem: Example
Example: 25 percent of residents in an area leaves their garage doors
open when they left their home. The chances of being raided are 5
percent and 1 percent for those who leave their doors open, and who
do not leave their doors open, respectively.
Problem: Suppose one resident was robbed, what is the probability
that he or she originally left his or her door open?
Solution: Let B = robbed residents, A1 = open doors residents, A2
= close door residents. We know that P (A1 ) = 0.25, P (A2 ) = 0.75,
P (B|A1 ) = 0.05, P (B|A2 ) = 0.01.
Donglei Du (UNB)
35 / 55
Bayes’ Theorem: Tree diagram
B  A1
0.05
B | A1
0.25 A
1
0 75
0.75
A2
B | A1
0.95
0.01
B  A1
B  A2
B | A2
B | A2
0.99
Donglei Du (UNB)
B  A2
36 / 55
Bayes’ Theorem: Example
Therefore
P (A1 |B) =
=
P (B|A1 )P (A1 )
P (B|A1 )P (A1 ) + P (B|A2 )P (A2 )
0.0125
0.05(0.25)
=
= 0.625.
0.05(0.25) + 0.01(0.75)
0.02
Similarly we can calculate
P (A2 |B) =
Donglei Du (UNB)
0.01(0.75)
0.0075
=
= 0.375.
0.05(0.25) + 0.01(0.75)
0.02
37 / 55
Bayes’ Theorem: Example
Example: An insurance company classifies drivers as good, medium,
or poor risks. Drivers who apply to for insurance fall into these three
groups in the proportions: 30%, 50%, and 20%, respectively. The
probabilities of a good-risk, medium-risk, and poor-risk drivers will
have an accident are 0.01, 0.03, and 0.10, respectively.
Problem: Suppose the company sells Mr. Brophy an insurance policy
and he has an accident. What is the probability that Mr. Brophy was
a good-risk driver?
Solution: Let B = accident drivers, A1 = good drivers, A2 =
medium drivers, and A3 = poor drivers. We know that P (A1 ) = 0.3,
P (A2 ) = 0.5, P (A3 ) = 0.2; P (B|A1) = 0.01, P (B|A2 ) = 0.03,
P (B|A3 ) = 0.10.
Donglei Du (UNB)
38 / 55
Bayes’ Theorem: Tree diagram
B  A1
0.01
B | A1
B | A1
0.99
0.3
A1
A2
0.5
02
0.2
B  A1
B  A2
0 03
0.03
B | A2
B | A2
0.97
A3
B  A2
B  A3
0.10
B | A3
B | A3
0.90
Donglei Du (UNB)
2013/9/17
B  A3
Donglei Du: Lecture 1
39 / 55
Bayes’ Theorem: Example
Therefore
P (A1 |B) =
=
P (B|A1 )P (A1 )
P (B|A1 )P (A1 ) + P (B|A2 )P (A2 ) + P (B|A3 )P (A3 )
0.01(0.3)
0.03
=
≈ 0.079
0.01(0.3) + 0.03(0.5) + 01.0(0.2)
0.038
Similarly we can calculate
P (A2 |B) =
P (A3 |B) =
0.03(0.5)
0.015
=
≈ 0.395
0.038
0.038
0.10(0.2)
0.02
=
≈ 0.526
0.038
0.038
Now we know that Mr. Brophy is mostly likely a poor-risk driver.
Donglei Du (UNB)
40 / 55
How to Interpret Probabilities
There are two ways to Interpret probability:
Objectivists
Subjectivists
Donglei Du (UNB)
41 / 55
How to Interpret Probabilities: Objectivists
Objectivists: assign numbers to describe some objective or physical
state of affairs.
The most popular version of objective probability is frequentist
probability, which claims that the probability of a random event
denotes the relative frequency of occurrence of an experiment’s
outcome, when repeating the experiment. This interpretation
considers probability to be the relative frequency ”in the long run” of
outcomes.
Donglei Du (UNB)
42 / 55
How to Interpret Probabilities: Subjectivists
Subjectivists: assign numbers per subjective probability, i.e., as a
degree of belief.
The most popular version of subjective probability is Bayesian
probability, which includes expert knowledge as well as experimental
data to produce probabilities. The expert knowledge is represented by
some (subjective) prior probability distribution. The data is
incorporated in a likelihood function. The product of the prior and
the likelihood, normalized, results in a posterior probability
distribution that incorporates all the information known to date.
Donglei Du (UNB)
43 / 55
Rules of Counting
Multiplication Rule
Permutation Rule
Combination Rule
Donglei Du (UNB)
44 / 55
Multiplication Rule
If one thing can be done in M ways, and if after this is done,
something else can be done in N ways, then both things can be done
in a total of M*N different ways in that stated order!
Donglei Du (UNB)
45 / 55
Permutation Rule
A counting technique that is used when order is important
A permutation of r objects chosen from n objects is a group of any r
objects, when order is important
Pnr =
Donglei Du (UNB)
n!
(n − r)!
46 / 55
Example
Example: You are assigned the task of choosing 2 of your 6
classmates to serve on a task force. One will act as the Chair of the
task force, and the other will be the Secretary.
Problem: In how many ways can you make this assignment?
Solution:
P62 =
Donglei Du (UNB)
6!
= 30
(6 − 2)!
47 / 55
Combination Rule
A counting technique that is used when order is NOT important
A combination of r objects chosen from n objects is a group of any r
objects, when order is not important
Cnr =
Donglei Du (UNB)
n!
r!(n − r)!
48 / 55
Example
Example: You are assigned the task of choosing 2 of your 6
classmates to serve on a task force. Responsibilities are evenly shared.
Problem: In how many ways can you make this assignment?
Solution:
C62 =
Donglei Du (UNB)
6!
= 15
2!(6 − 2)!
49 / 55
The Twin Paradox (Probabilistic Pigeonhole Principle)
Example: The Statistics Professor wants to play a little game with
his students. “I bet that there are two of you who have the same
birthday! What do you think”.
Several students reply immediately: “There are 366 possible
birthdays, so you could only conclude this if there were at least 367 of
us in the class! But there are only 50 of us, and so you would lose the
bet for sure!”
Problem: Let us calculate the probability of winning by the professor.
Donglei Du (UNB)
50 / 55
The Twin Paradox (Probabilistic Pigeonhole Principle)
Solution: Let A be the event of winning by the professor. Then its
complement is the event of wining by students, which is equivalent to
the event that all n = 50 students having different birthdays.
P (Ā) =
=
number of different birthdays for 50 students
number of birthdays for 50 students
50
C366
366!
366 &times; . . . &times; 316
=
=
≈ 0.03.
50
50
366 /50!
(366 − 50)!366
36650
Therefore the teacher has a much higher chance of wining:
P (A) = 1 − 0.03 = 97%.!
Donglei Du (UNB)
51 / 55
The Twin Paradox (Probabilistic Pigeonhole Principle)
Here are the probability of wining by the professor for different n, the
number of students.
n Prob.
10
12
20
41
30
71
40
81
50
97
60
99
Donglei Du (UNB)
52 / 55
Case: conditional probability and market direction
prediction
Probability of consecutive up or down price moves
G*#2)354&quot;?+67&quot;@#/3)&quot;J;3*I6&quot;
Data: S&amp;P 500 from
Jan 1950 to May 2015
K&quot;#
K&quot;#\$&quot;%&quot;
#\$&quot;!
!B&quot;
K&quot;#\$&quot;!B&quot;
@1A&quot;BCD&quot;BE!F&quot;
Figure: Probability of consecutive up and down price movements
Donglei Du (UNB)
53 / 55
Case: conditional probability and market direction
prediction
Probability of 4th up day (A), given 3 consecutive up days (B) is
P (A|B) =
Donglei Du (UNB)
P (A ∩ B)
P (A)
0.086
=
=
= 53.1%
P (B)
P (B)
0.162
54 / 55
Case: conditional probability and market direction
prediction
G*#212+)+034&quot;#\$&quot;@1*:36&quot;@#;34&quot;
G*#2)354&quot;?+67&quot;@#/3)&quot;J;3*I6&quot;
The probabilities
are all roughly
53% – which is
the same as the
probability of
having a
‘normal’ up day.
Q&quot;#
Q&quot;#\$&quot;%&quot;
#\$&quot;!
!B&quot;
Q&quot;#\$&quot;!B&quot;
@1A&quot;BCD&quot;BE!F&quot;
Figure: Probability of consecutive up and down price movements
Donglei Du (UNB)