2. PROBABILITY
Several concepts of probability have evolved over the time:
1. The classical approach,
2. The relative frequency approach
3. The axiomatic approach
Stochastic Processes - Probability
2-1
The classical approach
The probability of an event is calculated a priori by counting the
number of ways N(A) that an event A can occur and forming the
ratio:
P( A) 
N ( A)
N
where N is the number of all possible outcomes.
IMPORTANT NOTE: All the outcomes are equally likely
Example 2-1: drawing a specific set of cards from a deck of
cards
Stochastic Processes - Probability
2-2
The Relative Frequency Approach
Experimental approach, the same experiment has to be repeated
n times. If the A occurs n(A) times, then the relative frequency of
the event A is:
n ( A)
n
And the probability of the event A:
P( A)  lim
n
Stochastic Processes - Probability
2-3
n ( A)
n
Axiomatic Approach
This is a mathematical approach, part of a so called measure
theory.
The axiomatic approach introduces a probability space as its
main component.
Probability space consists of:
1. Sample space, denoted as S,
2. Collection of events, denoted as F
3. Probability measure, denoted by P
Stochastic Processes - Probability
2-4
ELEMENTARY SET THEORY
A set is a collection of objects.
These objects are called elements of the set
The notation
The notation
a A
a A
denotes that a is an element of A.
denotes that a is not an element of A.
Set A is a subset of set B, denoted by
A  B , if all
elements in A are elements of B.
The empty or null set is a set that contains no elements.
The notation is:
 or 
All sets are considered to be subsets of some universal set
Stochastic Processes - Probability
2-5
S
Set Operations
1. Equality: Two set are equal
A  B if, and only if
A  B and B  A
or
A  B is equivalent to the requirement
A  B and B  A
2. Transitivity Property: If
UA
Stochastic Processes - Probability
U  B and B  A then
2-6
A and B contains all the
A plus all the elements of B
3. Union: The union of sets
elements of


A  B   :   A or   B
Union is commutative: A  B  B  A
Union is associative:
A  B  C  A  B  C 
The definition can be extended to any finite numbers of sets:
n
A  A A
i
1
2
i 1
Stochastic Processes - Probability
2-7
     An
A and B is a set
consisting of all elements common to both A and B
4. Intersection: The intersection of sets


A  B   :   A and   B
Intersection is commutative: A  B  B  A
Intersection is associative:
 A  B  C  A  B  C 
A  B  C    A  B   A  C 
Intersection is distributive over unions:
The definition can be extended to any finite numbers of sets:
n
A  A A
i
1
2
i 1
Stochastic Processes - Probability
2-8
     An
Sets A and B are said to be mutually exclusive or disjoint if
they have no common elements:
A  B  
5. Complementation:
A
is the complement of
contains all the elements of
S
A if it
but not elements of


A   :   S and   A
A A  S
A  A  
Stochastic Processes - Probability
2-9
A
Graphical Representation – Venn Diagrams
A B
A
Stochastic Processes - Probability
A
B
A B
2-10
B
S
A
A
A
A
A-B
B
A-B is often called complement of B relative to A
Stochastic Processes - Probability
2-11
De Morgan’s Laws:
A B  A  B
A B  A  B
A  B  C   A  B   A  C 
Stochastic Processes - Probability
2-12
Sample space and events
Random Experiments: In the study of probability, any process of
observation is referred to as an experiment. If the outcome
cannot be predicted the experiment is called random experiment.
The results of the observation are called outcomes of the
experiment.
Example 2-2: tossing a coin, drawing a card
Sample Space: The set of all possible outcomes of a random
experiment is called the sample space (or universal set S). An
element of S is called the sample point. Each outcome of a
random experiment corresponds to a sample point.
Note: Any particular experiment can often have many different
sample spaces depending on the observation of interest.
Stochastic Processes - Probability
2-13
Example 2-3: Find the sample space for the experiment of
tossing a coin
(a) Once
(b) Twice.
(a) There are two possible outcomes, head or tail. Thus:
S  {H , T }
(b)
S  {HH , HT , TH , TT }
Example 2-4: Find the sample space for the experiment of
tossing a coin repeatedly and counting the number of tosses
required until the first head appears.
S  {1,2,3,4,...}
Note that there are an infinite number of outcomes.
Stochastic Processes - Probability
2-14
Example 2-5: Find the sample space for the experiment of
measuring (in hours) the lifetime of a transistor.
All possible outcomes are all nonnegative real numbers:
S   : 0    
A sample space S can be
 Discrete, if it consists of a finite number of sample points, or
infinite, but countable, number of sample points
 Countable, if its elements can be placed in one-to-one
correspondence with the positive integers,
 Continuous, if the sample points constitute a continuum.
Stochastic Processes - Probability
2-15
Events
Any subset of the sample space S is called an event.
The set of permissible events is denoted as F.
A sample point of S is often referred to as an elementary event.
The sample space S is the subset of itself, therefore:
S  the certain event
  S  the impossible event
Stochastic Processes - Probability
2-16
A  the event that A did not occur
AB = the event that either A or B or
both occurred
AB = the event that both A or B
occurred
Example 2-6: In the experiment of tossing the coin once, we can
model the set F as:
F  H , T , H , T , 
PH ,T   1; P  0
PH   1/ 2; PT   1/ 2
Stochastic Processes - Probability
2-17
Example 2-7: In the experiment of tossing a coin repeatedly and
counting the number of tosses required until the first head
appears express, express the following events:
(a) Event A, that the number of tosses required until first
head appears is even,
(b) Event B, that the number of tosses required until first
head appears is odd,
(c) Event C, that the number of tosses until first head
appears is less than 5.
,
B  ,
C  ,
A
Stochastic Processes - Probability
, , ,
, , ,
, , ,
2-18



Axiomatic Definition of Probability:
A probability P(A) is assigned to every event A (element of F).
To accomplish this a set of function
P is used that maps events
in F into the interval [0,1]
(P : F  [0,1])
The probability must satisfy three Axioms of Probability:
Stochastic Processes - Probability
2-19
Axiom 1.
Axiom 2.
Axiom 3. If
P( A)  0 for every A  F
P( S )  1
Ai  F , 1  i   , is any countable,
mutually exclusive sequence (i.e.
Ai  A j   for i  j )
of events, then:
  
P  Ai    P Ai 
 i 1  i 1
For a finite sample space S:
P A  B   P A  PB  if A  B  
Stochastic Processes - Probability
2-20
Elementary Properties of Probability:
1.
P  0
2.
PA   1  P A
3.
P A  PB  if
4.
P A  1
5.
A B
P A  B  P A  PB  P A  B
Stochastic Processes - Probability
2-21
Conditional Probability
The conditional probability of an event A, assuming (under the
condition) that event B has occurred, is defined as:
P A  B 
P A | B  
,
PB 
PB   0
Similarly
P A  B 
P  B | A 
,
P  A
P  A  0
From the two previous equations we get:
P A  B  P A | BPB  PB | AP A
and the following Bayes’ rule
Stochastic Processes - Probability
PB | AP A
P A | B  
P B 
2-22
Example 2-8: In the fair die experiment, the outcomes are f1, f2,
…, f6, the six faces of the die.
Let
A = { f2}, the event “a two occurs” and
B = { f2, f4 ,f6}, the event “an even outcome occurs
Then we have:
1
P  A  ,
6
1
, P B   ,
2
P  A  B   P  A
1/ 6 1
P f 2 |" even" 

1/ 2 3
Stochastic Processes - Probability
2-23
Example 2-9: A box contains three white balls w1, w2, w3, and two
red balls r1, and r2. We remove at random and without
replacement two balls in succession. What is the probability that
the first removed ball is white and the second is red?
3
P  first ball is white 
5
2 1
Psec ond is red | first is white  
4 2
Psec ond ball is red and first is white  
Psec ond is red | first is whiteP  first is white
13 3


2 5 10
Stochastic Processes - Probability
2-24
Total Probability: Let A1,
n
A  S
i
A2,…, An, be a partition of S, that is
and Ai  A j   for i  j
i 1
Let B be arbitrary event. Then:
PB   PB | A1 P A1   PB | A2 P A2  
....  PB | An P An 
This is known as the Total Probability Theorem.
Stochastic Processes - Probability
2-25
S
A2
B
A2B
A1
A4
A4B=
A1B
A3B
A3
A5
A5B=A5
A4B A4B
Venn diagram illustration of the Total Probability Theorem
Stochastic Processes - Probability
2-26
Combining the total probability and the Bayes rule we get:
PB | Ai P Ai 
P Ai | B  
P B 
PB | Ai P Ai 
P Ai | B  
PB | A1 P A1   PB | A2 P A2   ....  PB | An P An B 
The P(Ai) are called a priori probabilities, and P(Ai|B) are
called a posteriori probabilities.
Stochastic Processes - Probability
2-27
Example 2-10: We have four boxes. Box #1 contains 2000
components, 5% defective, box #2 contains 500 components,
40% defective, and boxes #3 and #4 contain 1000 components
each, 10% defective in both boxes. We select one box at random
and remove one component.
(a) What is the probability that this component is defective?
(b) If, after the examining the component we find out that it is
defective, what is the probability that it came from box #2?
(a) From the total probability theorem we have:
4
P(component is defective)   P(defective component | box # i ) P(box # i )
i 1
 (0.05)(0.25)  (0.4)(0.25)  (0.1)(0.25)  (0.1)(0.25)  0.1625
(b) From the Bayes rule we have:
P(box #2 | defective component) 
P(defective component | box #2) P(box #2)
(0.4)(0.25)

 0.615
0.1625
Stochastic Processes - Probability
2-28
P(component is defective)
Independent Events:
Two events A and B are independent if, and only if
P( A  B)  P( A) P( B)
If A and B are independent, then:
P( A  B) P( A) P( B)
P( A | B) 

 P( A)
P( B)
P( B)
Three events A1,
A2,and A3 are independent if:
P( Ai  Aj )  P( Ai ) P( Aj ) for i  j
P( A1  A2  A3 )  P( A1 ) P( A2 ) P( A3 )
Stochastic Processes - Probability
2-29
Example 2-11:
P( A1 )  P( A2 )  P( A3 )  1 / 5
P( A1  A2 )  P( A1  A3 )  P( A2  A3 )  1 / 25
P( A1  A2  A3 )  1 / 25
I
Independent or not independent?
A1
A3
A2
Stochastic Processes - Probability
2-30