PROBABILITY AND
BAYES THEOREM
1
PROBABILITY
SAMPLE
POPULATION
STATISTICAL
INFERENCE
2
• PROBABILITY: A numerical value
expressing the degree of uncertainty
regarding the occurrence of an event. A
measure of uncertainty.
• STATISTICAL INFERENCE: The science
of drawing inferences about the population
based only on a part of the population,
sample.
3
PROBABILITY
• CLASSICAL INTERPRETATION
If a random experiment is repeated an infinite
number of times, the relative frequency for any given
outcome is the probability of this outcome.
Probability of an event: Relative frequency of the
occurrence of the event in the long run.
– Example: Probability of observing a head in a fair coin toss is 0.5 (if coin is
tossed long enough).
• SUBJECTIVE INTERPRETATION
The assignment of probabilities to event of interest is
subjective
– Example: I am guessing there is 50% chance of rain today.
4
PROBABILITY
• Random experiment
– a random experiment is a process or course of action,
whose outcome is uncertain.
• Examples
Experiment
• Flip a coin
• Record a statistics test marks
• Measure the time to assemble
a computer
Outcomes
Heads and Tails
Numbers between 0 and 100
Numbers from zero and above
5
PROBABILITY
• Performing the same random experiment
repeatedly, may result in different
outcomes, therefore, the best we can do is
consider the probability of occurrence of
a certain outcome.
• To determine the probabilities, first we
need to define and list the possible
outcomes
6
Sample Space
• Determining the outcomes.
– Build an exhaustive list of all possible
outcomes.
– Make sure the listed outcomes are mutually
exclusive.
• The set of all possible outcomes of an
experiment is called a sample space and
denoted by S.
7
Sample Space
Countable
Uncountable
(Continuous )
Finite number
of elements
Infinite number of
elements
8
EXAMPLES
• Countable sample space examples:
– Tossing a coin experiment
S : {Head, Tail}
– Rolling a dice experiment
S : {1, 2, 3, 4, 5, 6}
– Determination of the sex of a newborn child
S : {girl, boy}
• Uncountable sample space examples:
– Life time of a light bulb
S : [0, ∞)
– Closing daily prices of a stock
S : [0, ∞)
9
Sample Space
• Multiple sample spaces for the same
experiment are possible
• E.g. with 5 coin tosses we can take:
S={HHHHH, HHHHT, …} or if we are only
interested in the number of heads we can
take S*={0,1,2,3,4,5}
10
EXAMPLES
• Examine 3 fuses in sequence and note the
results of each experiment, then an
outcome for the entire experiment is any
sequence of N’s (non-defectives) and D’s
(defectives) of length 3. Hence, the
sample space is
S : { NNN, NND, NDN, DNN, NDD,
DND, DDN, DDD}
11
Assigning Probabilities
– Given a sample space S ={O1,O2,…,Ok}, the
following characteristics for the probability P(Oi)
of the simple event Oi must hold:
1.
0  POi   1 for each i
k
2.
 PO   1
i
i 1
– Probability of an event: The probability P(A),
of event A is the sum of the probabilities
assigned to the simple events contained in A.
12
Assigning Probabilities
• P(A) is the proportion of times the event A
is observed.
total outcomes in A
P( A) 
total outcomes in S
13
Set theory: Definitions
• Set: a set A is a collection of elements (or
outcomes)
• Membership: x A (x is in A), or x  A (x is not in A)
C
• Complement: A  {x : x  A}
• Union: A  B  {x : x  A or x  B}
• Intersection: A  B  {x : x  A and x  B}
• Difference: A \ B  ABC  {x : x  A and x  B}
• Subset: A  B ( x  A  x  B) A is contained in B
• Equality: A  B if A  B and B  A
• Symmetric difference: AB  {x : x  A or x  B, but not both}
14
Algebraic laws
• commutative: A ∪ B = B ∪ A
A∩B=B∩A
• associative: (A ∪ B) ∪ C = A ∪ (B ∪ C)
A ∩ (B ∩ C) = (A ∩ B) ∩ C
• distributive: A ∪ (B ∩ C) = (A ∪ B) ∩ (A ∪ C)
A ∩ (B ∪ C) = (A ∩ B) ∪ (A ∩ C)
• DeMorgan’s: (A ∪ B)' = A' ∩ B' (' is complement)
(A ∩ B)' = A' ∪ B'
15
Intersection
• The intersection of event A and B is the event
that occurs when both A and B occur.
• The intersection of events A and B is denoted by
(A and B) or AB.
• The joint probability of A and B is the probability
of the intersection of A and B, which is denoted
by P(A and B) or P(AB).
16
Union
• The union event of A and B is the event
that occurs when either A or B or both
occur.
• At least one of the events occur.
• It is denoted “A or B” OR AB
17
Addition Rule
For any two events A and B
P(A  B) = P(A) + P(B) - P(A  B)
18
Complement Rule
• The complement of event A (denoted by
AC) is the event that occurs when event A
does not occur.
• The probability of the complement event is
calculated by
A and AC consist of all the
simple events in the
sample space. Therefore,
P(A) + P(AC) = 1
P(AC) = 1 - P(A)
19
MUTUALLY EXCLUSIVE EVENTS
• Two events A and B are said to be
mutually exclusive or disjoint, if A and B
have no common outcomes. That is,
A and B =  (empty set)
•The events A1,A2,… are pairwise mutually
exclusive (disjoint), if
Ai  Aj =  for all i  j.
20
EXAMPLE
• The number of spots turning up when a
six-sided dice is tossed is observed.
Consider the following events.
A: The number observed is at most 2.
B: The number observed is an even number.
C: The number 4 turns up.
21
VENN DIAGRAM
• A graphical representation of the sample
1
space.
A
AB
S
1
2
B
4
A
3
6
5
C
4
2
1
B
6
AB
A
22
4
AC = A and C are mutually exclusive
B
6
22
AXIOMS OF PROBABILTY
(KOLMOGOROV AXIOMS)
Given a sample space S, the probability
function is a function P that satisfies
1) For any event A, 0  P(A)  1.
2) P(S) = 1.
3) If A1, A2,… are pairwise disjoint, then


i 1
i 1
P( Ai )   P( Ai )
23
Probability
P
:
Probability
function
S
domain

[0,1]
range
24
THE CALCULUS OF
PROBABILITIES
•
If P is a probability function and A is any
set, then
a. P()=0
b. P(A)  1
c. P(AC)=1  P(A)
25
THE CALCULUS OF
PROBABILITIES
•
If P is a probability function and A and B
any sets, then
a. P(B  AC) = P(B)P(A  B)
b. If A  B, then P(A)  P(B)
c. P(A  B)  P(A)+P(B)  1 (Bonferroni Inequality)

d. P 



i 1

Ai  



 P  A  for any sets A , A ,
i
1
2
i 1
(Boole’s Inequality)
26
Principle of Inclusion-Exclusion
• A generalization of addition rule
n
n
i 1
i 1
P( Ai )   P( Ai )   P( Ai Aj )  ...  (1) n1 P( A1 A2 ...An )
i j
• Proof by induction
27
EQUALLY LIKELY OUTCOMES
• The same probability is assigned to each simple
event in the sample space, S.
• Suppose that S={s1,…,sN} is a finite sample
space. If all the outcomes are equally likely, then
P({si})=1/N for every outcome si.
28
ODDS
• The odds of an event A is defined by
P ( A)
P ( A)

C
P ( A ) 1  P ( A)
•It tells us how much more likely to see the
occurrence of event A.
•P(A)=3/4P(AC)=1/4 P(A)/P(AC) = 3.
That is, the odds is 3. It is 3 times more
likely that A occurs as it is that it does not.
29
ODDS RATIO
• OR is the ratio of two odds.
• Useful for comparing the odds under two
different conditions or for two different
groups, e.g. odds for males versus
females.
• If odds of event A is 4.2 for males and 2
for females, then odds ratio is 2.1. The
odds of observing event A is 2.1 times
higher for males compared to females.
30
CONDITIONAL PROBABILITY
• (Marginal) Probability: P(A): How likely is it
that an event A will occur when an
experiment is performed?
• Conditional Probability: P(A|B): How will
the probability of event A be affected by
the knowledge of the occurrence or
nonoccurrence of event B?
• If two events are independent, then
P(A|B)=P(A)
31
CONDITIONAL PROBABILITY
P(A  B)
P(A| B) 
if P(B)  0
P(B)
0  P(A | B)  1
P(A | B)  1  P(A C | B)
P(A | A)  1
P(A1  A 2 | B)  P(A1 | B)  P(A 2 | B)  P(A1  A 2 | B)
32
Example
•
•
•
•
Roll two dice
S=all possible pairs ={(1,1),(1,2),…,(6,6)}
Let A=first roll is 1; B=sum is 7; C=sum is 8
P(A|B)=?; P(A|C)=?
• Solution:
• P(A|B)=P(A and B)/P(B)
P(B)=P({1,6} or {2,5} or {3,4} or {4,3} or {5,2} or {6,1})
= 6/36=1/6
P(A|B)= P({1,6})/(1/6)=1/6 =P(A)
A and B are
independent
33
Example
• P(A|C)=P(A and C)/P(C)=P(Ø)/P(C)=0
A and C are disjoint
Out of curiosity:
P(C)=P({2,6} or {3,5} or {4,4} or {5,3} or {6,2})
= 5/36
CONDITIONAL PROBABILITY
P( AB)  P( A) P( B | A)  P( B) P( A | B)
P( A1 A2 ...An )  P( A1 )P( A2 | A1 )P( A3 | A1, A2 )...P( An | A1,..., An1 )
35
Example
• Suppose we pick 4 cards at random from
a deck of 52 cards containing 4 aces.
• A=event that we pick 4 aces
• Ai=event that ith pick is an ace (i=1,2,3,4)
 52
P( A)  1 /   1 / 270,725
4 
4 3 2 1
P( A)  P( A1 A2 A3 A4 )  P( A1 ) P( A2 | A1 )... 
 1 / 270,725
52 51 50 49
36
BAYES THEOREM
• Suppose you have P(B|A), but need
P(A|B).
P(A  B) P(B | A)P(A)
P(A | B) 

for P(B)  0
P(B)
P(B)
37
Example
• Let:
– D: Event that person has the disease;
– T: Event that medical test results positive
• Given:
– Previous research shows that 0.3 % of all Turkish
population carries this disease; i.e., P(D)= 0.3 % = 0.003
– Probability of observing a positive test result for someone
with the disease is 95%; i.e., P(T|D)=0.95
– Probability of observing a positive test result for someone
without the disease is 4%; i.e. P(T| D C )= 0.04
• Find: probability of a randomly chosen person having the
disease given that the test result is positive.
38
Example
• Solution: Need P(D|T). Use Bayes Thm.
P(D|T)=P(T|D)*P(D)/P(T)
P(T)=P(D and T)+P(D C and T)
= 0.95*0.003+0.04*0.997 = 0.04273
P(D|T) =0.95*0.003 / 0.04273 = 6.67 %
Test is not very reliable!
39
BAYES THEOREM
• Can be generalized to more than two events.
• If Ai is a partition of S, then,
P( B | Ai ) P( Ai )
P( B | Ai ) P( Ai )
P( Ai | B) 

P( B)
 P( B | A j ) P( A j )
j
• Can be rewritten in terms of odds
– Suppose A1,A2,… are competing hypotheses and B is evidence
or data relevant to choosing the correct hypothesis
P( Ai | B) P( B | Ai ) P( Ai )

P( A j | B) P( B | A j ) P( A j )
Posterior odds = likelihood ratio x prior odds
40
Independence
• A and B are independent iff
– P(A|B)=P(A) or P(B|A)=P(B)
– P(AB)=P(A)P(B)
• A1, A2, …, An are mutually independent iff
P( Ai )   P( Ai ) for every subset j of {1,2,…,n}
i j
i j
E.g. for n=3, A1, A2, A3 are mutually independent
iff P(A1A2A3)=P(A1)P(A2)P(A3) and
P(A1A2)=P(A1)P(A2) and P(A1A3)=P(A1)P(A3) and
P(A2A3)=P(A2)P(A3)
41
Independence
• If n=4, then the number of conditions for
independence is
 4  4  4

 4

3

 2
  11
     
• Find these conditions.
42
Sequences of events
• A sequence of events A1, A2, … is
increasing iff A1  A2  A3...
• A sequence of events A1, A2, … is
decreasing iff A1  A2  A3...

An   Ai
• If {An} is increasing, then lim
n 
i 1

An   Ai
• If {An} is decreasing, then lim
n 
i 1
43
Examples
• Let S=(0,1) and An=(1/n,1)
{An} is increasing. What is limit of An as n
goes to infinity?
• Let S=(0,1) and Bn=(0,1/n)
{Bn} is decreasing. What is limit of Bn as n
goes to infinity?
44
Problems
1. Show that two nonempty events cannot
be disjoint and independent at the same
time.
Hint: First, prove that if they are disjoint,
then they are not independent. Second,
prove that if they are independent, then
they are not disjoint.
Problems
2. If P(A)=1/3 and P(Bc)=1/4, can A and B
be disjoint? Explain.
Problems
3. Either prove the statement is true or disprove it:
If P(B|A)=P(B|AC), then A and B are independent.
Problems
4. An insurance company has three types of customers –
high risk, medium risk, and low risk. Twenty percent of
its customers are high risk, and 30% are medium risk.
Also, the probability that a customer has at least one
accident in the current year is 0.25 for high risk, 0.16
for medium risk, and 0.1 for low risk.
a) Find the probability that a customer chosen at random
will have at least one accident in the current year.
b) Find the probability that a customer is high risk, given
that the person has had at least one accident during
the current year.
Problems
5. Eleven poker chips are numbered consecutively
1 through 10, with two of them labeled with a 6
and placed in a jar. A chip is drawn at random.
i) Find the probability of drawing a 6.
ii) Find the odds of drawing a 6 from the jar.
iii) Find the odds of not drawing a 6.
49
Problems
6. If the odds in favor of winning a horse
race are 3:5, find the probability of winning
the race.
50
Problems
7. In a hypothetical clinical study, the following
results were obtained.
Treatment
Ibuprofen 400
mg
Placebo
Total
number
of
patients
treated
Number who
achieved at
least 50% pain
relief
Number who
did not achieve
at least 50%
pain relief
40
22
18
40
7
33
Find the odds ratio and interpret.
51