Review of Probability
Theory
Review of Probability Theory






Experiments, Sample Spaces and Events
Axioms of Probability
Conditional Probability
Bayes’s Rule
Independence
Discrete & Continuous Random Variables
© Tallal Elshabrawy
2
Random Experiment
 It is an experiment whose outcome cannot be predicted
with certainty
 Examples:
Tossing a Coin
© Tallal Elshabrawy
Rolling a Die
3
Random Experiment in Communications
Transmission of Bits across a Communication Channel
v
Waveform
Generator
x
Channel
y
Waveform r
Detection
+A V.
vi
vi=1
vi=0
0
T
0
T
xi
-A V.
+
zi  ]-∞, ∞[
yi
0
yi>0
yi<0
ri=1
ri
ri=0
Why is this a random experiment?
 We do not know


© Tallal Elshabrawy
The amount of noise that will affect the transmitted bit
Whether the bit will be received in error or not
4
Random Experiment in Networks
Transferring a Packet across a Communication Network
Packet
Packet
Why is this a random experiment?
 We do not know


© Tallal Elshabrawy
Whether the packet will reach the destination or not
If the packet reaches the destination, how long would it take to get
there?
5
Sample Space
 The set of all possible outcomes
 Tossing a coin
Heads
Tails
 S = {H,T}
 Rolling a die
 S = {1,2,3,4,5,6}
 The AWGN in a Communication Channel
 S = ] -∞, ∞ [
xi
+
yi
zi
© Tallal Elshabrawy
6
Event
 An event is a subset of the sample space S
 Examples
 Let A be the event of observing one head in a coin
flipped two times
 A = {HT,TH}
 Let B be the event of observing two heads in a coin
flipped twice
 B = {HH}
© Tallal Elshabrawy
7
Axioms of Probability
Probability of an event is a measure of how often
an event might occur
no. of sample pts in A
P( A) 
no. of sample pts in S
Axioms of Probability
1. 0  P  A  1
2. P     0,P  S   1
3. P  A  B   P  A +P  B  -P  A, B 
© Tallal Elshabrawy
8
Example
 Let Event A characterize
that the outcome of rolling
the die once is smaller
than 3
 A = {1,2}
 P(A) = 2/6 = 1/3
 Let Event B characterize
that the outcome of rolling
the die once is an even
number
 B = {2,4,6}
 P(B) = 3/6 = 1/2
© Tallal Elshabrawy
S
A
B
1
2
5
6
4
3
P  A, B   1/ 6
P  A  B   1/3  1/ 2  1/ 6
9
Conditional Probability
 Probability of event B given A has occurred
P  A, B 
P  B A 
P  A
 Probability of event A given B has occurred
P  A, B 
P A B 
P B
© Tallal Elshabrawy
10
Example
 Two cards are drawn in succession without
replacement from an ordinary (52 cards) deck.
Find the probability that both cards are aces
 Let A be the event that the first card is an ace
 Let B be the event that the second card is an
ace
P  A, B  =P  A   P  B A 
4 3
1
P  A, B  =  
52 51 16 17
© Tallal Elshabrawy
11
Conditional Probability in Communications
+A V.
vi
vi=1
vi=0
0
T
0
T
xi
-A V.
yi
+
0
yi<0
Pr[error v=1]= Pr[r=0 v=1]
Pr[error v=1]= Pr[y<0 x=1]
Pr[error v=1]= Pr[x+z<0 x=1]
ri
ri=0
0.8
0.7
0.6
0.5
© Tallal Elshabrawy
yi>0
zi  ]-∞, ∞[
Conditioned on v=1, what
is the probability of making
an error?
ri=1
0.4
0.3
r=0
Decision
Zone
0.2
0.1
0
-5
-4
-3
-2
-1
0
1
2
3
4
5
12
Bayes’s Rule
P  A, B 
P  B A 
P  A
P  A, B 
P A B 
P B
P(B A)P( A)
P( A B) =
P(B )
© Tallal Elshabrawy
13
Theorem of Total Probability
Let B1, B2, …, Bn be a set of mutually exclusive
and exhaustive events.
P( A) = ∑i =1 P A Bi P (Bi )
n
© Tallal Elshabrawy
( )
14
Bayes’s Theorem
Let B1, B2, …, Bn be a set of mutually exclusive
and exhaustive events.
(
)
P(B A) =
∑ P(A B )P(B )
P A Bi P(Bi )
i
n
i =1
© Tallal Elshabrawy
i
i
15
Independent Events
 A and B are independent if
 P(B|A) = P(B)
 P(A|B) = P(A)
 P(A,B) = P(A)P(B)
© Tallal Elshabrawy
16
Example
 Let A be the event that the grades will be out on
Thursday P(A)
 Let B be the even that I will get A+ in Random
Signals and Noise  P(B)
 So What is the probability that I get A+ if the
grades are out on Thursday  P(B|A) = P(B)
© Tallal Elshabrawy
17
Random Variable
 Characterizes the experiment in terms of real numbers
 Example
 X is the variable for the number of heads for a coin tossed three
times
 X = 0,1,2,3
 Discrete Random Variables
 The random variable can only take a finite number of values
 Continuous Random Variables
 The random variable can take a continuum of values
© Tallal Elshabrawy
18
Bernoulli Discrete Random Variable
 Represents experiments that have two possible outcomes.
These experiments are called Bernoulli Trials
 Associates values {0, 1} with the two outcomes such that
 P[X = 0] = 1-p
 P[X = 1] = p
 Examples
 Coin tossing experiment maps a ‘Heads’ to X = 1 and a ‘Tails’ to
X = 0 (or vice versa) such that p=0.5 for a fair coin
 Digital communication system where X = 1 represents a bit
received in error and X = 0 corresponds to a bit received correctly.
In such system p represents the channel bit error probability
© Tallal Elshabrawy
19
Binomial Discrete Random Variable
 A random variable that represents the number of
occurrences of ‘1’ or ‘0’ in n Bernoulli trials
 The corresponding random variable X may take and values
from {0, 1, 2, …, n}
 The probability mass function PMF for having k ‘1’ in n
Bernoulli trials is
P[X = k] = nCk pk(1-p)n-k
 Examples
 In a digital communication system, the number of bits in error in a
packet depicts a Binomial discrete random variable
© Tallal Elshabrawy
20
Geometric Discrete Random Variable
 Geometric distribution describes the number of Bernoulli
trials in succession are conducted until some particular
outcome is observed (lets say ‘1’)
 The corresponding random variable X may take and values
from {1, 2, 3, …, ∞}
 The probability mass function PMF for having k Bernoulli
trials in succession until an outcome of ‘1’ is observed
P[X = k] = (1-p)k-1p
 Examples:
 In a communication network, the number of transmissions until a
packet is received correctly follows a Geometric distribution
© Tallal Elshabrawy
21