Lecture 5: Basic Probability Theory
Donglei Du
(ddu@unb.edu)
Faculty of Business Administration, University of New Brunswick, NB Canada Fredericton
E3B 9Y2
Donglei Du (UNB)
ADM 2623: Business Statistics
1 / 52
Table of contents
1
Probability Theory
What is probability?
Random Experiment (RE)
Set/events operations
How to Interpret Probabilities?
Rules of Counting
Donglei Du (UNB)
ADM 2623: Business Statistics
2 / 52
Layout
1
Probability Theory
What is probability?
Random Experiment (RE)
Set/events operations
How to Interpret Probabilities?
Rules of Counting
Donglei Du (UNB)
ADM 2623: Business Statistics
3 / 52
What is probability?
Probability is a measure of the likelihood that an event in the future
will happen.
It can only assume a value between 0 and 1;
A value near zero means the event is not likely to happen;
A value near one means it is likely to happen.
Donglei Du (UNB)
ADM 2623: Business Statistics
4 / 52
Random Experiment (RE)
RE is a process that satisfies the following propositions:
the process can be repeated as many trials as you want
the outcome of any trial is uncertain
well-defined set of possible outcomes
each outcome has a probability associated with it
Donglei Du (UNB)
ADM 2623: Business Statistics
5 / 52
An example
Toss a coin once:
This process can be repeated as many times as you want (repetitive
nature)
Nobody knows whether Head or Tail will appear in any particular
tossing (uncertain outcome)
However, we know either Head or Tail must appear (Well-defined
sample space)
Now what is the chance you will get a Head if you toss a coin now?
Donglei Du (UNB)
ADM 2623: Business Statistics
6 / 52
Some Terminologies
Outcome: is a particular result of a random experiment
Sample Space: is the collection or set of all the possible outcomes of
a random experiment
Event: is the collection of one or more outcomes of an experiment
Donglei Du (UNB)
ADM 2623: Business Statistics
7 / 52
Example
Toss a coin once
The sample space is S = {H, T}.
An event is the occurrence of head: E = {h}.
Flip two coins
The sample space has two outcomes
S = {(H,H), (H,T), (T,H), (T,T)}.
An event is the occurrence of a head in the first coin:
E = {(H, H), (H, T )}
Roll a die once
The sample space contains 6 outcomes S = {1, . . . , 6}.
An event is the occurrence of an even number: E = {2, 4, 6}.
Roll two dice.
The sample space contains the 36 outcomes S = {(1, 1), . . . , (6, 6)}.
An event is the occurrence of two dice with sum equal to 4:
E = {(1, 3), (2, 2), (3, 1)}
Donglei Du (UNB)
ADM 2623: Business Statistics
8 / 52
Venn Diagram
A Venn diagram or set diagram is a diagram that shows all possible
logical relations between a finite collection of sets.
The Venn diagram for the above Die example.
Donglei Du (UNB)
1
3
5
2
4
6
ADM 2623: Business Statistics
9 / 52
An example
Toss a coin twice:
This process can be repeated as many times as you want (repetitive
nature)
Every trial contains the outcome of two coins
Origin
First
Flip
p
H
T
Donglei Du (UNB)
Second
Flip
p
H
HH
T
H
HT
TH
T
TT
ADM 2623: Business Statistics
10 / 52
Set operations
Union
Intersection:
Complement:
Donglei Du (UNB)
ADM 2623: Business Statistics
11 / 52
Union
Sample Space
A B
A
B
Union: The union event of two events A and B is denoted as A ∪ B,
which consists of all outcomes in either A or B or in both. Namely
event A ∪ B occurs if either A or B occurs.
Example: Roll a die: E1 = {1, 3, 5} and E2 = {1, 2, 3}. Then
E1 ∪ E2 = {1, 2, 3, 5}
Donglei Du (UNB)
ADM 2623: Business Statistics
12 / 52
Intersection
Sample Space
A
A B
B
Intersection: The intersection event of two events A and B is denoted
as A ∩ B, which consists of all outcomes in both A and B or in both.
Namely event A ∩ B occurs if both A and B occur.
Example: Roll a die: E1 = {1, 3, 5} and E2 = {1, 2, 3}. Then
E1 ∩ E2 = {1, 3}
Two events A and B are mutually exclusive if A ∩ B = ∅.
Donglei Du (UNB)
ADM 2623: Business Statistics
13 / 52
Complement
Sample Space
A
A
Complement: The complement event of an events A is denoted as Ā,
which consists of all outcomes that are not in A. Namely event Ā
occurs if A does not occur.
Example: Roll a die: E1 = {1, 3, 5}. Then Ē1 = {2, 4, 6}
Donglei Du (UNB)
ADM 2623: Business Statistics
14 / 52
Probabilities defined on events
For any random experiment with sample space S, the probability of
any event is P (E) such that
(i) 0 ≤ P (E) ≤ 1.
(ii) P (S) = 1.
(iii) (special addition rule) For any pair-wise mutually exclusive events
E1 , E2 , . . .
!
∞
∞
[
X
P
En =
P (En ).
n=1
Donglei Du (UNB)
i=1
ADM 2623: Business Statistics
15 / 52
Examples
Toss a fair coin once: If we assume a head is equally likely to appear
as a tail, then
1
P (H) = P (T ) =
2
Toss a biased coin once: If we assume a head is twice as likely to
appear as a tail, then
2
1
P (H) = , P (T ) = .
3
3
Toss a fair die once: If we assume all sides are equally likely to
appear, then
1
P (i) = , i = 1, . . . , 6.
6
From (iii), we must have
1
P ({1, 3, 5}) = P (1) + P (2) + P (3) = .
2
Donglei Du (UNB)
ADM 2623: Business Statistics
16 / 52
Probability Distributions defined on sample space
A (discrete) probability distribution on a given sample space is a table
of all disjoint outcomes and their associated probabilities.
Example: Toss two fair coins: The table below is the probability
distribution for the sample space S = {HH, HT, T H, T T }:
Outcome
HH
HT
TH
TT
Donglei Du (UNB)
Prob.
0.25
0.25
0.25
0.25
ADM 2623: Business Statistics
17 / 52
Rule of Probability
Complement Rule
Addition Rule
Multiplication Rule
Donglei Du (UNB)
ADM 2623: Business Statistics
18 / 52
Rule of Complement
For any event A:
P (Ā) = 1 − P (A)
Donglei Du (UNB)
ADM 2623: Business Statistics
19 / 52
Rule of Addition
For any event A and B:
P (A ∪ B) = P (A) + P (B) − P (A ∩ B).
If events A and B are mutually exclusive:
P (A ∪ B) = P (A) + P (B).
The above can be extended to any number of mutually exclusive
events A1 , . . . , An :
P (A1 ∪ . . . ∪ An ) = P (A1 ) + . . . + P (An ).
Donglei Du (UNB)
ADM 2623: Business Statistics
20 / 52
Rule of Addition: Example
Toss two coins and assume that the four possible outcomes
S = {HH, HT, T H, T T } are equally likely to happen.
Let A = {HH, HT } be the event that the first coin falls head, and
B = {HH, T H} be the event that the second coin falls head.
Problem: Find the probability that either the first coin falls head or
the second coin falls head?
Solution:
P (A ∪ B) = P (A) + P (B) − P (A ∩ B) =
Donglei Du (UNB)
ADM 2623: Business Statistics
3
1 1 1
+ − = .
2 2 4
4
21 / 52
Rule of Multiplication
Joint probability
Conditional probability
Donglei Du (UNB)
ADM 2623: Business Statistics
22 / 52
Joint probability
Joint probability: For two events A and B defined on the same
sample space, the joint probability of events A and B is P (A ∩ B).
Example: Roll a die once. Let A = {1, 2, 3} and B = {1, 3, 5} be two
events. Then the joint probability of A and B is
P (A ∩ B) = P ({1, 3}) =
Donglei Du (UNB)
ADM 2623: Business Statistics
1
2
= .
6
3
23 / 52
Represent Joint probability as contingency table
Joint probability distribution is given in a tabular form, called
contingency table. The probability distribution for each variable is
called the marginal distribution.
Example: A survey of undergraduate students in the Faculty of
Business Management at UNB revealed the following regarding the
gender and majors of the students:
Male
Female
Marginal
Accoun.
150/750
175/750
325/750
IB
150/750
160/750
310/750
HR
50/750
65/750
115/750
Marginal
350/750
400/750
1
Example: What is the Probability of selecting a Female Accounting
student?
Solution:
P (F ∩ A) = 175/750 = 23.33%
Donglei Du (UNB)
ADM 2623: Business Statistics
24 / 52
Conditional probability
Conditional probability: For any event A and B, the probability of
event A, given the occurrence of event B:
P (A|B) =
P (A ∩ B)
.
P (B)
Example: What is the probability of selecting a Female, given that
the person selected is an International Business major?
Solution:
P (F |I) = 160/310 = 51.6%
Compare to the unconditional probability:
P (F ) = 400/750 = 50%
Donglei Du (UNB)
ADM 2623: Business Statistics
25 / 52
Independent events: two events
Two events A and B are independent if and only if
P (A|B) = P (A)
or equivalently
P (A ∩ B) = P (A) × P (B).
Example: the events F = {(F, A), (F, IB), (F, HR)} and
IB = {(F, IB), (M, IB)} are not independent, since
P (F |IB) = 160/310 = 51.6% 6= 50% = 400/750 = P (F )
Donglei Du (UNB)
ADM 2623: Business Statistics
26 / 52
Independent events: more than two events
n events A1 , . . . , An are independent if and only if for any r ≤ n:
P (A1 ∩ . . . ∩ Ar ) = P (A1 ) × . . . × P (Ar ).
Example: (Pairwise independent events may not be independent
overall): Roll a four-faced die once. Let A1 = {1, 2}, A2 = {1, 3} and
A3 = {1, 4} be three events. Then
1
4
1
P (A1 ∩ A3 ) = P (A1 ) × P (A3 ) =
4
1
P (A2 ∩ A3 ) = P (A2 ) × P (A3 ) =
4
1
1
P (A1 ∩ A2 ∩ A3 ) =
6= = P (A1 ) × P (A2 ) × P (A3 )
4
8
P (A1 ∩ A2 ) = P (A1 ) × P (A2 ) =
Donglei Du (UNB)
ADM 2623: Business Statistics
27 / 52
Rule of Multiplication
For any two events A and B:
P (A ∩ B) = P (A|B)P (B).
The above can be extended to any number of events A1 , . . . , An :
P (A1 ∩ . . . ∩ An ) = P (A1 )P (A2 |A1 ) . . . P (An |A1 ∩ . . . ∩ An−1 ).
If events A and B are independent:
P (A ∩ B) = P (A)P (B).
The above can be extended to any number of independent events
A1 , . . . , A n :
P (A1 ∩ . . . ∩ An ) = P (A1 ) . . . P (An ).
Donglei Du (UNB)
ADM 2623: Business Statistics
28 / 52
Rule of Multiplication: Example
Example: Draw three cards with replacement i.e., draw one card,
look at it, put it back, and repeat twice more.
Problem: Find the probability of drawing 3 Queens in a row:
Solution: Let Qi (i = 1, 2, 3) be the event that ith draw gives you a
Queen. Then the three events are independent of each other.
P (Q1 ∩ Q2 ∩ Q3 ) = P (Q1 )P (Q2 )P (Q3 ) =
Donglei Du (UNB)
ADM 2623: Business Statistics
4
4
4
×
×
≈ 0.00046.
52 52 52
29 / 52
Rule of Multiplication: Example
Example: Draw three cards without replacement i.e., draw one card,
look at it, keep it, and repeat twice more.
Problem: Find the probability of drawing 3 Queens in a row:
Solution: Let Qi (i = 1, 2, 3) be the event that ith draw gives you a
Queen. Then the three events are dependent of each other.
P (Q1 ∩ Q2 ∩ Q3 ) = P (Q1 )P (Q2 |Q1 )P (Q3 |Q1 ∩ Q2 )
4
3
2
=
×
×
≈ 0.00018.
52 51 50
Donglei Du (UNB)
ADM 2623: Business Statistics
30 / 52
More examples on rules of probability
Example: Draw one card from a deck of 52 cards
Problem: What is the probability of getting a red card or a Queen?
Solution:
P (R ∪ Q) = P (R) + P (Q) − P (R ∩ Q) =
Donglei Du (UNB)
ADM 2623: Business Statistics
26 + 4 − 2
7
=
≈ 0.538
52
13
31 / 52
More examples on rules of probability
Example: Two men throw their identical hats into the center of the
room at a party. Then the hats are mixed up and each man randomly
selects a hat.
Problem: What is the probability that none of them selects his own
hat.
Solution: Let Ai (i = 1, 2) be the event that the ith man selects his
own hat. Then the desired probability can be calculated via the
complement rule. The complement event is that at least one man
selects his own hat:
P (A1 ∪ A2 ) = P (A1 ) + P (A2 ) − P (A1 ∩ A2 )
= 0.5 + 0.5 − P (A1 )P (A2 |A1 ) = 1 − 0.5(1) = 0.5
Therefore none of them selects his own hat is
P A1 ∪ A2 = 1 − P (A1 ∪ A2 ) = 1 − 0.5 = 0.5
Donglei Du (UNB)
ADM 2623: Business Statistics
32 / 52
Another method for solving the above problem
Solution: Let Ai (i = 1, 2) be the event that the ith man selects his
own hat. Therefore none of them selects his own hat is
P Ā1 ∩ Ā2 = P (Ā1 )P (Ā2 |Ā1 ) = 0.5(1) = 0.5.
Donglei Du (UNB)
ADM 2623: Business Statistics
33 / 52
Bayes’ Theorem
Sample Space
Sample Space
A1  B
A1
An  B
A2  B
A2
……
An
Given an event B, and a set of mutually exclusive and exhaustive
events A1 , . . . , An , and we know the prior probabilities
P (A1 ), . . . , P (An ). Then the posterior is:
P (A1 |B) =
Donglei Du (UNB)
P (B|A1 )P (A1 )
P (B|A1 )P (A1 ) + . . . + P (B|An )P (An )
ADM 2623: Business Statistics
34 / 52
Bayes’ Theorem: Example
Example: 25 percent of residents in an area leaves their garage doors
open when they left their home. The chances of being raided are 5
percent and 1 percent for those who leave their doors open, and who
do not leave their doors open, respectively.
Problem: Suppose one resident was robbed, what is the probability
that he or she originally left his or her door open?
Solution: Let B = robbed residents, A1 = open doors residents, A2
= close door residents. We know that P (A1 ) = 0.25, P (A2 ) = 0.75,
P (B|A1 ) = 0.05, P (B|A2 ) = 0.01.
Donglei Du (UNB)
ADM 2623: Business Statistics
35 / 52
Bayes’ Theorem: Tree diagram
B  A1
0.05
B | A1
0.25 A
1
0 75
0.75
A2
B | A1
0.95
0.01
B  A1
B  A2
B | A2
B | A2
0.99
Donglei Du (UNB)
ADM 2623: Business Statistics
B  A2
36 / 52
Bayes’ Theorem: Example
Therefore
P (A1 |B) =
=
P (B|A1 )P (A1 )
P (B|A1 )P (A1 ) + P (B|A2 )P (A2 )
0.0125
0.05(0.25)
=
= 0.625.
0.05(0.25) + 0.01(0.75)
0.02
Similarly we can calculate
P (A2 |B) =
Donglei Du (UNB)
0.01(0.75)
0.0075
=
= 0.375.
0.05(0.25) + 0.01(0.75)
0.02
ADM 2623: Business Statistics
37 / 52
Bayes’ Theorem: Example
Example: An insurance company classifies drivers as good, medium,
or poor risks. Drivers who apply to for insurance fall into these three
groups in the proportions: 30%, 50%, and 20%, respectively. The
probabilities of a good-risk, medium-risk, and poor-risk drivers will
have an accident are 0.01, 0.03, and 0.10, respectively.
Problem: Suppose the company sells Mr. Brophy an insurance policy
and he has an accident. What is the probability that Mr. Brophy was
a good-risk driver?
Solution: Let B = accident drivers, A1 = good drivers, A2 =
medium drivers, and A3 = poor drivers. We know that P (A1 ) = 0.3,
P (A2 ) = 0.5, P (A3 ) = 0.2; P (B|A1) = 0.01, P (B|A2 ) = 0.03,
P (B|A3 ) = 0.10.
Donglei Du (UNB)
ADM 2623: Business Statistics
38 / 52
Bayes’ Theorem: Tree diagram
B  A1
0.01
B | A1
B | A1
0.99
0.3
A1
A2
0.5
02
0.2
B  A1
B  A2
0 03
0.03
B | A2
B | A2
0.97
A3
B  A2
B  A3
0.10
B | A3
B | A3
0.90
Donglei Du (UNB)
2013/9/17
B  A3
Donglei Du: Lecture 1
ADM 2623: Business Statistics
39 / 52
Bayes’ Theorem: Example
Therefore
P (A1 |B) =
=
P (B|A1 )P (A1 )
P (B|A1 )P (A1 ) + P (B|A2 )P (A2 ) + P (B|A3 )P (A3 )
0.01(0.3)
0.03
=
≈ 0.079
0.01(0.3) + 0.03(0.5) + 01.0(0.2)
0.038
Similarly we can calculate
P (A2 |B) =
P (A3 |B) =
0.03(0.5)
0.015
=
≈ 0.395
0.038
0.038
0.10(0.2)
0.02
=
≈ 0.526
0.038
0.038
Now we know that Mr. Brophy is mostly likely a poor-risk driver.
Donglei Du (UNB)
ADM 2623: Business Statistics
40 / 52
How to Interpret Probabilities
There are two ways to Interpret probability:
Objectivists
Subjectivists
Donglei Du (UNB)
ADM 2623: Business Statistics
41 / 52
How to Interpret Probabilities: Objectivists
Objectivists: assign numbers to describe some objective or physical
state of affairs.
The most popular version of objective probability is frequentist
probability, which claims that the probability of a random event
denotes the relative frequency of occurrence of an experiment’s
outcome, when repeating the experiment. This interpretation
considers probability to be the relative frequency ”in the long run” of
outcomes.
Donglei Du (UNB)
ADM 2623: Business Statistics
42 / 52
How to Interpret Probabilities: Subjectivists
Subjectivists: assign numbers per subjective probability, i.e., as a
degree of belief. The degree of belief has been interpreted as, ”the
price at which you would buy or sell a bet that pays 1 unit of utility if
E, 0 if not E.”
The most popular version of subjective probability is Bayesian
probability, which includes expert knowledge as well as experimental
data to produce probabilities. The expert knowledge is represented by
some (subjective) prior probability distribution. The data is
incorporated in a likelihood function. The product of the prior and
the likelihood, normalized, results in a posterior probability
distribution that incorporates all the information known to date.
Donglei Du (UNB)
ADM 2623: Business Statistics
43 / 52
Rules of Counting
Multiplication Rule
Permutation Rule
Combination Rule
Donglei Du (UNB)
ADM 2623: Business Statistics
44 / 52
Multiplication Rule
If one thing can be done in M ways, and if after this is done,
something else can be done in N ways, then both things can be done
in a total of M*N different ways in that stated order!
Donglei Du (UNB)
ADM 2623: Business Statistics
45 / 52
Permutation Rule
A counting technique that is used when order is important
A permutation of r objects chosen from n objects is a group of any r
objects, when order is important
Pnr =
Donglei Du (UNB)
n!
(n − r)!
ADM 2623: Business Statistics
46 / 52
Example
Example: You are assigned the task of choosing 2 of your 6
classmates to serve on a task force. One will act as the Chair of the
task force, and the other will be the Secretary.
Problem: In how many ways can you make this assignment?
Solution:
P62 =
Donglei Du (UNB)
6!
= 30
(6 − 2)!
ADM 2623: Business Statistics
47 / 52
Combination Rule
A counting technique that is used when order is NOT important
A combination of r objects chosen from n objects is a group of any r
objects, when order is not important
Cnr =
Donglei Du (UNB)
n!
r!(n − r)!
ADM 2623: Business Statistics
48 / 52
Example
Example: You are assigned the task of choosing 2 of your 6
classmates to serve on a task force. Responsibilities are evenly shared.
Problem: In how many ways can you make this assignment?
Solution:
C62 =
Donglei Du (UNB)
6!
= 15
2!(6 − 2)!
ADM 2623: Business Statistics
49 / 52
The Twin Paradox (Probabilistic Pigeonhole Principle)
Example: The Statistics Professor wants to play a little game with
his students. “I bet that there are two of you who have the same
birthday! What do you think”.
Several students reply immediately: “There are 366 possible
birthdays, so you could only conclude this if there were at least 367 of
us in the class! But there are only 50 of us, and so you would lose the
bet for sure!”
Problem: Let us calculate his probability of winning by both parties.
Donglei Du (UNB)
ADM 2623: Business Statistics
50 / 52
The Twin Paradox (Probabilistic Pigeonhole Principle)
Solution: Let A be the event of winning by the professor. Then its
complement is the event of wining by students, which is equivalent to
the event that all n = 50 students having different birthdays.
P (Ā) =
=
number of different birthdays for 50 students
number of birthdays for 50 students
50
C366
366!
366 × . . . × 316
=
=
≈ 0.03.
50
50
366 /50!
(366 − 50)!366
36650
Therefore the teacher has a much higher chance of wining:
P (A) = 1 − 0.03 = 97%.!
Donglei Du (UNB)
ADM 2623: Business Statistics
51 / 52
The Twin Paradox (Probabilistic Pigeonhole Principle)
Here are the probability of wining by the professor for different n, the
number of students.
n Prob.
10
12
20
41
30
71
40
81
50
97
60
99
Donglei Du (UNB)
ADM 2623: Business Statistics
52 / 52