Chapter 1. Probability
Introduction
 Random Signal
Is a time waveform that can be characterized only in some probabilistic
manner
Examples:
1) The bit sequence in a computer bit stream
2) Output voltage of a wind-powered generator
3) The voltage from a solar detector
4) The fading in wireless communication channel
5) The noise in communication systems
Effect of Noise on Communication System
𝑛(𝑡)
Transmitter
𝑥(𝑡)
Channel
𝑟(𝑡)
Receiver
𝑥(𝑡) : Transmitted signal
𝑛(𝑡) : Noise signal (undesired random signal)
𝑟 𝑡 = 𝑥 𝑡 + 𝑛(𝑡) : Received noisy signal
 System performance is measured by the probability of bit error or
Bit Error Rate (BER)
Effect of noise on digital signal
0
+5V
1
-5V
Introduction to Set Theory
 a set is a collection of elements (e.g., A={1,3,5} )
 empty or null set contains no elements denoted by Φ
 a set A is a subset of another set B if every element in A is
also and element of B
𝐴⊂𝐵
𝐴 𝑖𝑠 𝑐𝑜𝑛𝑡𝑎𝑖𝑛𝑒𝑑 𝑖𝑛 𝐵
 two sets, A and B are called disjoint or mutually
exclusive if they have no common elements
 a finite set has elements that can be counted, otherwise it is
infinite set
 Universal set (Space): if all sets under consideration are
subset of a set S, then S is called the Space.
Introduction to Set Theory
 Example :







A={1,3,5,7}
B={1,2,3,4,……}
C={0.5 < 𝑐 ≤ 8.5}
D={0}
E= Φ
Sets A, B, D are tabular specified while C is rule specified
Set A is countable and finite
Set B is countable but infinite
Set C is uncountable and infinite
𝐴 ⊂ 𝐵 and 𝐴 ⊂ 𝐶
Sets A and D are disjoint (NO common elements)
3 ∈ 𝐴 while 4 ∉ 𝐴
Introduction to Set Theory
 Example : the space of rolling the die experiment is
S={1,2,3,4,5,6}
 Sets A={1,3,5} and B={2,4,6} are subsets of the space S
 There are 26 = 64 subsets of S
Set Operations
 Equality:
Two sets A and B are equal if : 𝐴 ⊆ 𝐵 𝑎𝑛𝑑 𝐵 ⊆ 𝐴
Hence, 𝐴 = 𝐵
 Difference:
The difference of two sets A and B is a set containing all elements of
A that are not presents in B
Example:
A = {0.6 < 𝑎 ≤ 1.6}
B = {1 ≤ 𝑏 ≤ 2.5}
Then,
C = A − B = 0.6 < 𝑐 < 1
D = B − A = {1.6 < 𝑑 ≤ 2.5}
∴
A−B≠B−A
Set Operations
 Union:
The union of two sets A and B is written as
𝐶 = 𝐴⋃𝐵
is the set of all elements of A or B or both
 Intersection:
The intersection of two sets A and B is written as
𝐷 = 𝐴⋂𝐵
it is the set of all elements common to both A and B
 For disjoint sets A and B:
𝐴⋂𝐵 = Φ
 In general, the union and intersection of N sets
𝐴𝑛 , 𝑛 = 1,2, … , 𝑁:
𝐶 = 𝐴1 ⋃𝐴2 ⋃… ⋃𝐴𝑁 = ⋃𝑁
𝑛=1 𝐴𝑛
𝐷 = 𝐴1 ⋂𝐴2 ⋂… ⋂𝐴𝑁 = ⋂𝑁
𝑛=1 𝐴𝑛
Set Operations
 Complement:
The complements of a set A , denoted 𝐴, is the set of all elements not in A
Φ=𝑆
𝑆=Φ
 Duality principle:
𝐴⋃𝐴 = 𝑆
𝐴⋂𝐴 = Φ
𝐴
Set Operations
 Venn Diagrams
Set Operations
 Example:
S = 1 ≤ 𝑖𝑛𝑡𝑒𝑔𝑒𝑟𝑠 ≤ 12
A = {1,3,5,12}
B = 2,6,7,8,9,10,11 , Find:
1) 𝐴⋃𝐵
2) 𝐴⋂𝐵
3) 𝐴
4) 𝐵
1) 𝐴⋃𝐵 = {1,2,3,5,6,7,8,9,10,11,12}
2) 𝐴⋂𝐵= Φ
3) 𝐴={2,4,6,7,8,9,10,11}
4) 𝐵={1,3,4,5,12}
Algebra of Sets
 Three Laws:
1) Commutative law
𝐴⋃𝐵 = 𝐵⋃𝐴
𝐴⋂𝐵 = 𝐵⋂𝐴
2) Distributive Law
𝐴⋂(𝐵⋃𝐶) = (𝐴⋂𝐵)⋃(𝐴⋂𝐶)
𝐴⋃(𝐵⋂𝐶) = (𝐴⋃𝐵)⋂(𝐴⋃𝐶)
3) Associative law
(𝐴⋃𝐵)⋃𝐶 = 𝐴⋃(𝐵⋃𝐶)
𝐴⋂𝐵 ⋂𝐶 = 𝐴⋂ 𝐵⋂𝐶
 De Morgan’s Law:
𝐴⋃𝐵 = 𝐴⋂𝐵
𝐴⋂𝐵 = 𝐴⋃𝐵
Example
 Show that:
𝐵⋃(𝐵⋂𝐴) = 𝐵⋃𝐴
𝐵⋃(𝐵⋂𝐴) = (𝐵⋃𝐵)⋂ 𝐵⋃𝐴
= 𝑆⋂ 𝐵⋃𝐴
= 𝐵⋃𝐴
Probability Introduced Through Sets
 The mathematical model of a random experiment consists of :
1) Sample Space (S): set of all possible outcomes
2) Event : subset of S
3) Probability: function of the event defined on S
 Example: rolling the die experiment
1) Sample Space S={1,2,3,4,5,6}
2) Define events A={1}, B={2}
3) The probability of event A, denoted by P(A)=1/6
The probability of event B, denoted by P(B)=1/6
Probability definitions and axioms
 Probability : is a function of the events defined on sample space S
 Notation: P(A) “the probability of event A”
 The assigned probabilities are chosen so as to satisfy three axioms:
1) Axiom 1: 𝑃 𝐴 ≥ 0
2) Axiom 2: 𝑃 𝑆 = 1
3) Axiom 3: Given N disjoint events 𝐴𝑛 , 𝑛 = 1,2, … , 𝑁, defined
on the sample space S, then:
𝑁
𝑃 ⋃𝑁
𝐴𝑛 ⋂𝐴𝑚 =Φ for all 𝑛 ≠ 𝑚
𝑛=1 𝐴𝑛 = 𝑛=1 𝑃(𝐴𝑛 ) = 1 ,
𝑆
𝐴3
𝐴1
…………..
𝐴2
𝐴𝑁
Example
 An experiment consists of observing the sum of the numbers showing up when two
dice are thrown. Define the following events:
A={𝑠𝑢𝑚 = 7}, B={8 < 𝑠𝑢𝑚 ≤ 11} , C={𝑠𝑢𝑚 > 10}
1) Write down the sample space
2) Find the probabilities: P(A), P(B), P(C), P(A⋂C)
(1,1)
(2,1)
(3,1)
(4,1)
(5,1)
(6,1)
(1,2)
(2,2)
(3,2)
(4,2)
(5,2)
(6,2)
(1,3)
(2,3)
(3,3)
(4,3)
(5,3)
(6,3)
(1,4)
(2,4)
(3,4)
(4,4)
(5,4)
(6,4)
(1,5)
(2,5)
(3,5)
(4,5)
(5,5)
(6,5)
A
(1,6)
(2,6)
(3,6) B
(4,6)
(5,6)
(6,6)
C
P(A)= 6/36
P(B)=9/36
P(C)= 3/36
P(A⋂C)=P(Φ) = 0
Joint Probability
 The joint probability of two events A and B :
𝑃 𝐴⋂𝐵 = 𝑃 𝐴 + 𝑃 𝐵 − 𝑃(𝐴⋃𝐵)
 Equivalently :
𝑃(𝐴⋃𝐵) = 𝑃 𝐴 + 𝑃 𝐵 − 𝑃 𝐴⋂𝐵
 For disjoint event A and B :
𝑃 𝐴⋂𝐵 = 𝑃 𝐴 + 𝑃 𝐵 − 𝑃 𝐴⋃𝐵 = 0
⇒
𝑃(𝐴⋃𝐵) = 𝑃 𝐴 + 𝑃 𝐵
Conditional Probability
 Given some event B with nonzero probability (𝑃(𝐵) ≠ 0), we define
the conditional probability of an event A , given event B, as follows :
𝑃 𝐴⋂𝐵
𝑃 𝐴|𝐵 =
𝑃(𝐵)
 For disjoint event A and B :
𝑃 𝐴⋂𝐵
0
𝑃 𝐴|𝐵 =
=
=0
𝑃(𝐵)
𝑃(𝐵)
Total Probability
 Suppose we are given N disjoint events 𝐵𝑛 , 𝑛 = 1,2, … , 𝑁
These events satisfies :
𝐵𝑛 ⋂𝐵𝑚 = Φ for all 𝑛 ≠ 𝑚
⋃𝑁
𝑛=1 𝐵𝑛 = 𝑆
Then, the total probability of event A :
𝑁
𝑃 𝐴 =
𝑁
𝑃(𝐴|𝐵𝑛 ) 𝑃(𝐵𝑛 ) =
𝑛=1
𝑃(𝐴 ∩ 𝐵𝑛 )
𝑛=1
𝑆
𝐵3
𝐵1
A
𝐵2
…………..
𝐵𝑁
Total Probability
 Prove that :
𝑁
𝑃 𝐴 =
𝑁
𝑃(𝐴|𝐵𝑛 ) 𝑃(𝐵𝑛 ) =
𝑛=1
Proof:
𝑃(𝐴 ∩ 𝐵𝑛 )
𝑛=1
𝐴=𝐴∩𝑆
𝑁
𝐴=𝐴∩
𝐵𝑛
𝑛=1
Axiom 3: Given N disjoint events
𝑁
By distributive law:
𝑃
𝑁
𝐴=
The events 𝐴 ∩ 𝐵𝑛 are disjoint so by applying Axiom 3
𝑃 𝐴 =𝑃
𝑁
𝐴 ∩ 𝐵𝑛
𝑛=1
=
𝑃(𝐴𝑛 )
𝑛=1
if 𝐴𝑛 ⋂𝐴𝑚 =Φ for all 𝑛 ≠ 𝑚
𝑛=1
𝑁
𝐴𝑛
𝑛=1
𝐴 ∩ 𝐵𝑛
𝑁
=
𝑃(𝐴 ∩ 𝐵𝑛 )
𝑛=1
Bayes Theorem
 Bayes Theorem :
𝑃(𝐴|𝐵𝑛 ) 𝑃(𝐵𝑛 )
𝑃 𝐵𝑛 𝐴 =
𝑃 𝐴
Where the total probability 𝑃 𝐴 :
𝑁
𝑃 𝐴 =
𝑃(𝐴|𝐵𝑛 ) 𝑃(𝐵𝑛 )
𝑛=1
 Homework: use the definition of conditional probability and
commutative law to prove Bayes theorem
𝑆
𝐵3
𝐵1
A
𝐵2
…………..
𝐵𝑁
Example: Elementary binary communication system
Transmitter
Channel
Receiver
𝑃 𝐴1 𝐵1 = 0.9
𝐵1 = {1}
𝐴1 = {1}
𝐵2 = {0}
𝐴2 = {0}
𝑃 𝐴2 𝐵2 = 0.9
 Given 𝑃 𝐵1 = 0.6 and 𝑃 𝐵2 = 0.4 , Calculate
1) The received symbol probabilities 𝑃 𝐴1 and 𝑃 𝐴2
2) The conditional probabilities 𝑃 𝐵1 |𝐴1 and 𝑃 𝐵2 |𝐴1
1)
𝑃 𝐴2
2)
𝑃 𝐴1 = 𝑃 𝐴1 𝐵1 ×𝑃 𝐵1 + 𝑃 𝐴1 𝐵2 ×𝑃 𝐵2
𝑃 𝐴1 = 0.9×0.6 + 0.1×0.4=0.58
= 𝑃 𝐴2 𝐵1 ×𝑃 𝐵1 + 𝑃 𝐴2 𝐵2 ×𝑃 𝐵2 = 0.1 × 0.6 + 0.9 × 0.4 = 0.42
From Total probability :
𝑃 𝐴1 |𝐵1 𝑃 𝐵1
𝑃 𝐴1
= (0.9 × 0.6)/0.58=0.931
𝑃 𝐴1 |𝐵2 𝑃 𝐵2
𝑃 𝐴1
= (0.1 × 0.4)/0.58=0.069
From Bayes theorem : 𝑃 𝐵1 |𝐴1 =
𝑃 𝐵2 |𝐴1 =
Statistically Independent Events
 Two events A and B are statistically independent if :
𝑃 𝐴|𝐵 = 𝑃(𝐴)
𝑃 𝐵|𝐴 = 𝑃(𝐵)
 Hence:
𝑃 𝐴⋂𝐵
𝑃 𝐴|𝐵 =
=𝑃 𝐴
𝑃(𝐵)
⇒ 𝑃 𝐴⋂𝐵 = 𝑃 𝐴 𝑃(𝐵)
 Statistically independent events A and B ⇒ 𝑃 𝐴⋂𝐵 = 𝑃 𝐴 𝑃(𝐵)
 Disjoint events A and B ⇒ 𝑃 𝐴⋂𝐵 = 0
Statistically Independent Events
 Example: In an experiment, one card is selected from an ordinary 52- card deck.
Define events :
A={select a king}
B={select a jack or queen}
C={select a heart}
Determine whether events A, B, and C are statistically independent by pairs
 Statistically Independent ⇒
4
52

𝑃 𝐴 =

𝑃 𝐴⋂𝐵 = 0

𝑃 𝐴⋂𝐶 =
=

𝑃 𝐵⋂𝐶
=
1
52
2
=
52
𝑃 𝐴⋂𝐵 = 𝑃 𝐴 𝑃(𝐵)
𝑃 𝐵 =
≠
8
52
𝑃 𝐶 =
4
8
32
× = 2
52
52 52
4
13
1
𝑃 𝐴 ×𝑃 𝐶 = × =
52
52 52
8
13
2
𝑃 𝐵 ×𝑃 𝐶 = × =
52
52 52
𝑃 𝐴 ×𝑃 𝐵 =
13
52
⇒ dependent
⇒ independent
⇒ independent
Statistically Independent Events
 Example: Consider drawing four cards from an ordinary 52- card
deck. Let events 𝐴1 , 𝐴2 , 𝐴3 , 𝐴4 define drawing an ace on the first,
second, third, and fourth cards, respectively.
a) Find the probability of drawing the four cards with replacement
b) Find the probability of drawing the four cards without replacement
a) Independent events: 𝑃 𝐴1 ⋂𝐴2 ⋂𝐴3 ⋂𝐴4 = 𝑃(𝐴1 ) × 𝑃(𝐴2 ) × 𝑃 𝐴3 × 𝑃 𝐴4
=
4
4
4
4
× × ×
52
52
52
52
b) Dependent events:
𝑃 𝐴1 ⋂𝐴2 ⋂𝐴3 ⋂𝐴4 = 𝑃(𝐴1 ) × 𝑃(𝐴2 |𝐴1 ) × 𝑃 𝐴3 |𝐴1 ⋂𝐴2 × 𝑃 𝐴4 |𝐴1 ⋂𝐴2 ⋂𝐴3
=
4
3
2
1
× × ×
52
51
50
49
Permutation and Combinations
 Permutations:
multiple trails without replacement
orderings of 𝐾 elements taken from 𝑁 elements without replacement
𝑁!
𝑁
𝑃𝐾 = 𝑁 × 𝑁 − 1 × 𝑁 − 2 × ⋯ 𝑁 − 𝐾 + 1 =
𝑁−𝐾 !
 Note that 3! = 3 × 2 × 1 = 6
0! = 1
 Example: how many permutations are there for four cards taken
from 52-card deck?
𝑃452 = 52 × 51 × 50 × 49 = 6497400
 Combinations:
order of elements in a sequence is not important
number of combinations of 𝐾 elements taken from 𝑁 elements without replacement
𝑁!
𝑁
=
𝑁−𝐾 ! ×𝐾!
𝐾
Bernoulli Trials
 It involves any experiment for which there are only two possible outcomes:
 Flipping a coin
 Hitting or missing a target
 Passing or failing an exam
 Receiving a 0 or a 1 in a computer bit stream
 Given an experiment, with two possible elementary events 𝐴, 𝐴, we shall repeat
this experiment 𝑁 times and determine the probability that event 𝐴 is observed exactly
𝐾 times out of 𝑁 trails.
 Let 𝑃 𝐴 = 𝑝
then
𝑃 𝐴 =1−𝑝
𝑁 𝐾
 Then, 𝑃 𝐴 𝑜𝑐𝑐𝑢𝑟𝑠 𝑒𝑥𝑎𝑐𝑡𝑙𝑦 𝐾 𝑡𝑖𝑚𝑒𝑠 =
𝑝 (1 − 𝑝)𝑁−𝐾
𝐾
𝑁!
𝑁
where
=
,
𝑁 taken 𝐾 at a time
𝑁−𝐾 ! ×𝐾!
𝐾
𝑁
𝑁
 Note that
=1
=1
0
𝑁
Bernoulli Trials
 Example : a submarine attempts to sink an aircraft carrier. It will be successful only
if two or more torpedoes hit the carrier. If the submarine fires three torpedoes and
the probability of a hit is 0.4 for each torpedo, what is the probability that the carrier
will be sunk?
 Three trials 𝑁 = 3
 𝐴 ={torpedo hits}
⇒ 𝑃 𝐴 = 𝑝 = 0.4
 𝑃 𝑡𝑜 𝑠𝑖𝑛𝑘 𝑎𝑖𝑟𝑐𝑟𝑎𝑓𝑡 𝑐𝑎𝑟𝑟𝑖𝑒𝑟 = 𝑃 𝑡𝑤𝑜 𝑜𝑟 𝑚𝑜𝑟𝑒 ℎ𝑖𝑡𝑠
𝑁 𝐾
 𝑃 𝐴 𝑜𝑐𝑐𝑢𝑟𝑠 𝑒𝑥𝑎𝑐𝑡𝑙𝑦 𝐾 𝑡𝑖𝑚𝑒𝑠 =
𝑝 (1 − 𝑝)𝑁−𝐾
𝐾
3
 𝑃 𝑒𝑥𝑎𝑐𝑡𝑙𝑦 𝐾 = 2 ℎ𝑖𝑡𝑠 =
0.42 (1 − 0.4)3−2 = 0.288
2
3
 𝑃 𝑒𝑥𝑎𝑐𝑡𝑙𝑦 𝐾 = 3 ℎ𝑖𝑡𝑠 =
0.43 (1 − 0.4)3−3 = 0.064
3
 𝑃 𝑡𝑜 𝑠𝑖𝑛𝑘 𝑎𝑖𝑟𝑐𝑟𝑎𝑓𝑡 𝑐𝑎𝑟𝑟𝑖𝑒𝑟 = 𝑃 𝑡𝑤𝑜 𝑜𝑟 𝑚𝑜𝑟𝑒 ℎ𝑖𝑡𝑠
= 𝑃 𝑒𝑥𝑎𝑐𝑡𝑙𝑦 𝐾 = 2 ℎ𝑖𝑡𝑠 + 𝑃 𝑒𝑥𝑎𝑐𝑡𝑙𝑦 𝐾 = 3 ℎ𝑖𝑡𝑠
= 0.288 + 0.064 = 0.352