PROBABILITY AND STATISTICS
FOR ENGINEERING
Random Variables
Hossein Sameti
Department of Computer Engineering
Sharif University of Technology
Random Variables
• If A  X 1 ( B ) belongs to the associated field F,
• Probability of the event:
" X ( )  B "  P( X 1 ( B)).
Sharif University of Technology
2
Random Variable
• Definition
– If   | X ( )  x  is an event ( F ) for every x in R,
– Random Variable is a finite single valued function X (  ) that maps
the set of all experimental outcomes  into the set of real
numbers R
Sharif University of Technology
3
Random Variable
• The specification of a random variable can also imply a redefinition
of the sample space.
Example
• we could define as a random variable “the total number of heads.”
• Experiment: Fair coin tossed 3 times,
Sharif University of Technology
4
Notation
• Notation:
– RV will always be denoted with uppercase letters
– The realized values of the variable (its range) will be denoted by
the corresponding lowercase letter.
– Thus, the random variable X can take the value x.
Sharif University of Technology
5
Borel collection
• X :r.v, X 1 ( B )  F
• B represents semi-infinite intervals of the form
{  x  a}
• The Borel collection B of such subsets of R is the smallest σ-field of
subsets of R that includes all semi-infinite intervals of the above
form.
• if X is a r.v, then

| X ( )  x
is an event for every x.
  X
Sharif University of Technology
 x

6
Probability Distribution Function(PDF)
(Cumulative Distribution Function )
P | X ( )  x   FX ( x )  0.
• FX (x ) is said to the Probability Distribution Function associated
with the r.v X.
• The role of the subscript X is only to identify the actual r.v.
Sharif University of Technology
7
Properties of a PDF
• if g(x) is a distribution function, then
– g ( )  1, g ( )  0,
g ( x1 )  g ( x2 ),
– If x1  x2 , then

g
(
x
)  g ( x ), for all x.
–
Sharif University of Technology
8
Properties of a PDF - Proof
• The probability distribution function is:
– nonnegative
– monotonically nondecreasing.
• Proof - We have:
FX ( )  P | X ( )     P()  1
FX ( )  P | X ( )     P( )  0.
• If x1  x2 , then ( , x1 )  ( , x2 ).
• X ( )  x1 implies X ( )  x2 . So,

| X ( )  x1
  
| X ( )  x2 ,
FX ( x1 )  P X ( )  x1   P X ( )  x2   FX ( x2 )
Sharif University of Technology
9
Properties of a PDF - Proof
• Let
x  xn  xn 1    x2  x1 ,
• Consider the event Ak    | x  X ( )  xk .
• Since  x  X ( )  xk   X ( )  x   X ( )  xk ,
• We get P ( Ak )  P  x  X ( )  xk   FX ( xk )  FX ( x ).
(mutually exclusive property)
• But  Ak 1  Ak  Ak 1  , and hence

lim Ak   Ak  
k 
k 1
 lim P ( Ak )  0.
k 
Sharif University of Technology
10
Properties of a PDF - Proof
• Thus
lim P ( Ak )  lim FX ( xk )  FX ( x )  0.
k 
k 


• But lim xk  x , the right limit of x, and hence FX ( x )  FX ( x),
k 
• i.e., FX (x) is right-continuous, justifying all properties of a distribution
function.
Sharif University of Technology
11
Additional Properties
• If FX ( x0 )  0 for some x0 , then FX ( x )  0, x  x0 .
This follows, since FX ( x0 )  P X ( )  x0   0 implies X ( )  x0 
is the null set, and for any x  x0 ,  X ( )  x  will be a subset of the null
set.
•
P X ( )  x   1  FX ( x ).
since
 X ( )  x   X ( )  x   
• P x1  X ( )  x2   FX ( x2 )  FX ( x1 ), x2  x1.
The events  X ( )  x1 and {x1  X ( )  x2 } are mutually exclusive and
their union represents the event  X ( )  x2 .
Sharif University of Technology
12
Additional Properties
P  X ( )  x   FX ( x )  FX ( x  ).
• Let x1  x   ,   0, and x2  x.
• We have: lim P x    X ( )  x   FX ( x )  lim FX ( x   ),
 0
 0

or P X ( )  x   FX ( x )  FX ( x ).
• Thus the only discontinuities of a distribution functionFX (x) occur at
points x0 where P X ( )  x0   FX ( x0 )  FX ( x0 )  0.
Sharif University of Technology
13
Continuous-type & Discrete-type R.V
• X is said to be a continuous-type r.v if FX ( x  )  FX ( x)
P X ( )  x0   FX ( x0 )  FX ( x0 )  0.
PX  x  0.
• If FX (x) is constant except for a finite number of jump discontinuities
(piece-wise constant; step-type), then X is said to be a discrete-type
r.v.
• If xi is such a discontinuity point, then
pi  PX  xi   FX ( xi )  FX ( xi ).
Sharif University of Technology
14
Example
• X is a r.v such that X ( )  c,   . Find FX (x).
Solution
• For x  c,  X ( )  x    , so that FX ( x )  0,
• For x  c, X ( )  x  , so that FX ( x )  1.
FX (x )
1
• at a point of discontinuity we get
c
x
P X  c   FX ( c )  FX ( c  )  1  0  1.
Sharif University of Technology
15
Example
•
In tossing a coin,   H ,T . Suppose the r.v X is
such that X (T )  0, X ( H )  1. Find FX (x).
Solution
•
For
x  0,  X ( )  x    , so that FX ( x)  0.
0  x  1,
x  1,
 X ( )  x    T , so that
 X ( )  x    H , T   ,
FX ( x)  P T
  1  p,
so that FX ( x)  1.
FX (x )
•
at a point of discontinuity we get
P X  0   FX (0)  FX (0 )  q  0  q.
Sharif University of Technology
1
q
x
1
16
Example
•
A fair coin is tossed twice, and let the r.v X represent the number of heads.
Find FX (x).
Solution
X ( HH )  2, X ( HT )  1, X (TH )  1, X (TT )  0.
x  0,
X ( )  x    FX ( x)  0,
0  x  1, X ( )  x   TT   FX ( x)  P TT   P (T ) P (T ) 
1
,
4
1  x  2, X ( )  x   TT , HT , TH   FX ( x)  P TT , HT , TH  
x  2,
X ( )  x    FX ( x)  1.
3
,
4
FX (x )
1
3/ 4
1/ 4
1
Sharif University of Technology
x
2
17
Probability Density Function (p.d.f)
(Probability Mass Function - for discrete RVs)
dFX ( x)
f X ( x) 
dx
Since
dFX ( x )
FX ( x  x )  FX ( x )
 lim
 0,

x

0
dx
x
from the monotone-nondecreasing nature of FX (x ),
f X ( x)  0 for all x
Sharif University of Technology
18
Probability Density Function (p.d.f)
• X is a continuous type r.v: f X (x) will be a continuous function
f X (x) has the general form
• X is a discrete type r.v:
f X (x )
f X ( x)   pi ( x  xi ),
i
pi
xi
Fig. 3.5
x
x
•
i represent the jump-discontinuity points in f X (x)
• f X (x) represents a collection of positive discrete masses,
• It is known as the probability mass function (p.m.f ) in the discrete
case.
Sharif University of Technology
19
Probability Density Function (p.d.f)
• We have: FX ( x) 
• Since

x

f X (u )du.
FX ()  1, we can say



f X ( x) dx  1,
P x1  X ( )  x2   FX ( x2 )  FX ( x1 )  
x2
x1
f X ( x )dx.
• the area under f X (x) in the interval ( x1 , x2 ) represents the
probability.
FX (x )
f X (x )
1
(a)
x1 x2
x
x1 x2
x
(b)
Sharif University of Technology
20
Example
Example
• X is a RV with p.d.f:
Find a and P(1< x <2).
Solution
•
3
𝑎𝑥 2 𝑑𝑥
0
=1
• 𝑃 1<𝑥<2 =
1
⇒ 𝑎=9
21 2
𝑥 𝑑𝑥
1 9
=
7
27
Sharif University of Technology
21
Example
X is a RV with cdf:
a) Compute f(x) and check its properties.
b) Find P(−1 <x<2) using both the pdf and the cdf.
Solution
a)
𝑑𝐹(𝑥)
𝑒 −𝑥
𝑓 𝑥 =
=
𝑑𝑥
(1 + 𝑒 −𝑥 )2
Properties:
∞
𝑓 𝑥 ≥ 0,
𝑓 𝑥 𝑑𝑥 = 1
−∞
b)
1
1
𝑃 −1 < 𝑥 < 2 = 𝐹 2 − 𝐹(−1) =
−
= 0.612
1 + 𝑒 −2 1 + 𝑒 1
Sharif University of Technology
22
Continuous-type Random Variables
X can take any value in the real line within a bounded or
unbounded interval.
•
•
•
•
•
•
•
Normal (Gaussian)
Uniform
Exponential
Gamma
Beta
Chi-Square
Rayleigh
•
•
•
•
Nakagami – m distribution
Cauchy
Laplace
Student’s t-distribution with n
degrees of freedom
•
Others: Weibull, Erlang, Lognormal,…
Sharif University of Technology
23
Normal Distribution
1
f X ( x) 
2
FX ( x )  
x

e
2
 ( x   ) 2 / 2 2
1
2 2
Where, G ( x ) 

x

e
.
 ( y   ) 2 / 2 2
1
2
e y
2
/2
x
dy  G 
,
  
dy
• Notation: X ~ N (  ,  )
2
mean
variance
Sharif University of Technology
f X (x )

x
24
Normal Distribution
The normal distribution is symmetrical around mean.
p.d.f
PDF
Sharif University of Technology
25
Normal Table
• The Normal Distribution illustrated in the table.
• With mean of 0 and a standard deviation of 1.
• In examples and exercises where z is used, it is found using the
formula (x- μ) / σ
• or (value given - mean) / standard deviation
Sharif University of Technology
26
Using the Normal Table
• The shaded area, A,
gives the probability
that Z is greater than
the given value.
Example
•
Calculate probability of:
– 0 < z < 0.11
– 0 < z < 0.34
– 0 < z < 0.53
Sharif University of Technology
27
Using the Table in Reverse
• Suppose you want the value of z such that the probability of
exceeding z is 33% or 0.33
Example
• Do the above for:
– A = 0.4443
– A = 0.3778
Sharif University of Technology
28
Applications
• Approximate normal distributions occur in many situations.
• When a large number of small effects acting additively and
independently.
• Example: Measurement errors are often assumed to be normally
distributed
Sharif University of Technology
29
Example
• What is the maximum number of people who can occupy a lift?
• Knowing that that the total weight of 8 people chosen at random
follows a normal distribution with
• mean of 550kg and
• standard deviation of 150kg.
• So, X ~ N ( 550, 22500 )
• What’s the probability that the total weight of 8 people exceeds
600kg?
Sharif University of Technology
30
Uniform Distribution
• Notation:
X ~ U ( a, b), a  b,
 1

, a  x  b,
f X ( x)   b  a

 0, otherwise.
p.d.f
PDF
Application
• Sampling from arbitrary distributions.(random number generation)
• inverse transform sampling method, which uses the cumulative distribution
function (CDF) of the target random variable.
Sharif University of Technology
31
Exponential distribution
 e  x , x  0,
f X ( x)  
 0, otherwise.
• Notation: X ~  ( )
f X (x )
x
Here,
X  1/ 
Fig. 3.9
Sharif University of Technology
32
Exponential distribution
p.d.f
PDF
Sharif University of Technology
33
Exponential Distribution
• Assume the occurences of nonoverlapping intervals are
independent, and assume:
– q(t): the probability that in a time interval t no event has
occurred.
– x: the waiting time to the first arrival
– Then we have: P(x>t)=q(t)
– t1 and t2 : two consecutive nonoverlapping intervals,
Sharif University of Technology
34
Exponential Distribution
• Then we have: q(t1) q(t2) = q(t1+t2)
• The only nontrivial bounded solution to the above equation is:
𝑞 𝑡 = 𝑒 −𝜆𝑡
• Hence
𝐹𝑋 𝑡 = 𝑃 𝑋 ≤ 𝑡 = 1 − 𝑞 𝑡 = 1 − 𝑒 −𝜆𝑡
So the pdf is exponential.
If the occurrences of events over nonoverlapping intervals are
independent, the corresponding pdf has to be exponential.
Sharif University of Technology
35
Memoryless Property Of Exponential Distribution
• Let
s, t  0
{x  t  s}
. Consider the events
{x  s}
P{ X  t  s}
P{ X  t  s | X  s} 
P{ X  s}

e
(t  s )
s
e
(since {X  t  s}  {X  s})
 e t  P{ X  t}
Sharif University of Technology
36
Application
• describing the lengths of the inter-arrival times in a homogeneous
Poisson processes.
– the time it takes before your next telephone call
• situations where certain events occur with a constant probability per
unit distance:
– the distance between mutations on a DNA strand;
• Because of memoryless property:
– well-suited to model the constant hazard rate in reliability theory.
Sharif University of Technology
37
Example
• Let X denote the time between log-ons to a server in a large
corporate network.
• Assuming that X has an exponential distribution with
minutes,
• What is the probability that a log-on occurs within 30 seconds after
starting the server?
Solution
Sharif University of Technology
38
Gamma Distribution
• Notation: X ~ G ( ,  )
(  0,   0)
 x 1
x / 

e
, x0
f X ( x )   ( )  

0, otherwise

( ) 
f X (x)

 1  x
x
e dx

0
• For integer
  n,
(n)  (n  1)!
Sharif University of Technology
39
Gamma Distribution
p.d.f
PDF
𝒌 =∝ : shape parameter, 𝜽 = 𝟏 𝜷: scale parameter, (𝜷 is called the rate parameter)
Sharif University of Technology
40
Beta Distribution
• Notation: X ~  ( a, b)
( a  0, b  0)
 1
x a 1 (1  x) b 1 , 0  x  1

f X ( x )    ( a, b)

0,
otherwise

where,  ( a, b) 

1

0
u a 1 (1  u ) b 1 du
( a )(b)
 ( a  b)
• Beta distribution with a=b=1 is
the uniform distribution on (0,1).
Sharif University of Technology
f X ( x)
0
1
x
41
Beta Distribution
• B(i, j) with integer values of i and j is the distribution of the i-th order
statistic of a sample of i + j − 1 independent random variables
uniformly distributed between 0 and 1.
• Used to model events which are constrained to take place within an
interval defined by a minimum and maximum value.
• Used in PERT, CPM and other project management / control
systems to describe the time to completion of a task.
Sharif University of Technology
42
Beta Distribution
p.d.f
PDF
∝ is a and 𝜷 is b in the formula.
Sharif University of Technology
43
Chi-Square Distribution
• Notation:
p.d.f
Note that for n=2, exponential distribution is
obtained.
Applications
• chi-square tests for goodness of fit
(k in figures is n in the formula)
Sharif University of Technology
PDF
44
Rayleigh
• Notation:
PDF
• when a two-dimensional vector (e.g. wind
velocity) has elements that are
• normally distributed,
• uncorrelated,
• with equal variance
• The vector’s magnitude (e.g. wind speed) will
then have a Rayleigh distribution.
Sharif University of Technology
p.d.f
45
Nakagami–m Distribution
p.d.f
Applications
• Used to model attenuation of wireless
signals traversing multiple paths.
(µ in figures is m in the formula and ω in figures is Ω in
the formula)
PDF
Sharif University of Technology
46
Cauchy Distribution
• Notation:
p.d.f
• The ratio of two independent
standard normal random variables is
a standard Cauchy variable
• It has no mean, variance or higher
moments defined.
(𝑥0 in figures is 𝜇 in the formula and 𝛾 in
figures is ∝ in the formula)
PDF
Sharif University of Technology
47
Laplace Distribution
1 −|𝑥−𝜇|
𝑓 𝑥 =
𝑒 𝜆 − ∞ < 𝑥 < ∞,
2𝜆
For 𝜇 = 0:
1 −|𝑥|
𝑓 𝑥 =
𝑒 𝜆 − ∞ < 𝑥 < ∞,
2𝜆
𝜆>0
𝜆>0
p.d.f
• The difference between two iid
exponential random variables is
governed by a Laplace distribution
(𝑏 in figures is 𝜆 in the formula)
PDF
Sharif University of Technology
48
Student’s t-Distribution
𝑛 > 0, 𝑟𝑒𝑎𝑙
• arises in the problem of estimating the mean of a normally
distributed population when the sample size is small.
p.d.f
(𝑘 in figures is n in the formula)
PDF
Sharif University of Technology
49
Discrete-type Random Variables
X can take only a finite (or countably infinite) number of values
•
•
•
•
•
•
•
•
Bernoulli
Binomial
Poisson
Hypergeometric
Geometric
Negative Binomial
Discrete-Uniform
Polya’s distribution
Sharif University of Technology
50
Bernoulli Distribution
X takes the values (0,1) and,
P ( X  0)  q ,
P ( X  1)  p.
Sharif University of Technology
51
Binomial Distribution
• Notation: X ~ B (n, p ),
 n  k n k
P( X  k )  
,
k 
p q
 
k  0,1,2, , n.
The probability of k successes in n experiments with replacement (in ball
drawing)
p.d.f
PDF
Sharif University of Technology
52
Binomial Distribution
Example
• Assume 5% of a very large population to be green-eyed.
• You pick 100 people randomly.
• X : The number of green-eyed people
•
X ~ B ( 100, 0.05 )
Sharif University of Technology
53
Poisson Distribution
• Notation: X ~ P ( ) ,
P( X  k )  e
p.d.f

k
k!
, k  0,1,2,, .
PDF
Sharif University of Technology
54
Poisson Distribution
Law Of Rare Events
In a Binomial distribution, as
• n approaches ∞
• p approaches 0
• np remains fixed at λ > 0 or at least np approaches λ > 0
The Binomial(n, p) distribution approaches the Poisson distribution with
expected value λ.
Sharif University of Technology
55
Poisson Distribution - Applications
• The Poisson distribution applies to various phenomena of discrete
nature.
• The probability of the phenomenon should be constant in time or
space.
• Examples: The number of…
– spelling mistakes one makes while typing a single page.
– phone calls at a call center per minute.
– times a web server is accessed per minute.
– roadkill (animals killed) found per unit length of road.
– pine trees per unit area of mixed forest.
– stars in a given volume of space.
Sharif University of Technology
56
Hypergeometric Distribution
• The probability of k successes in n experiments without
replacement (ball drawing from a bin with m white balls and N-m black
balls, success being drawing a white ball)
P( X  k ) 
m
 
k 
 
 N m 


 n k 


,
N 
 
n 
 
max(0, m  n  N )  k  min( m, n )
Application
• The classical application of the hypergeometric distribution is sampling
without replacement.
Sharif University of Technology
57
Geometric Distribution
• Notation:
P ( X  k )  pq k , k  0,1,2, , ,
q  1  p.
the geometric distribution is memoryless.
Sharif University of Technology
58
Example
• From past experience it is known that 3% of accounts in a large
accounting population are in error.
a) What is the probability that 5 accounts are audited before an
account in error is found?
b) What is the probability that the first account in error occurs in the
first five accounts audited?
Solution
a)
b)
Sharif University of Technology
59
Negative Binomial Distribution
• Notation:
 k  1 r k  r
P( X  k )  
,
p q
 r 1
•
k  r , r  1,
.
The geometric distribution is a special case of the negative binomial
distribution
• x independent trials each with two possible outcomes:
success/failure.
• The probability of success, denoted by p, is the same on every trial.
• Continues until r successes are observed
Sharif University of Technology
60
Example
• A basketball player is a 70% free throw shooter.
• During the season, what is the probability that he makes his
third free throw on his fifth shot?
• What is the probability that he makes his first free throw on his
fifth shot?
Sharif University of Technology
61
Discrete-Uniform Distribution
1
P( X  k ) 
, k  1,2,  , N .
N
Example
• In experiment of throwing a die, let X be the random variable
denoting what number is thrown.
P(X = 1) = 1/6
P(X = 2) = 1/6 etc
In fact, P(X = x) = 1/6 for all x between 1 and 6.
Sharif University of Technology
62
Example: Polya’s distribution Distribution
• A box contains a white balls and b black balls.
• A ball is drawn at random, and it is replaced along with c balls of
the same color.
• If X represents the number of white balls drawn in n such draws,
find the probability mass function of X.
Sharif University of Technology
63
Solution
PW : probability of drawing k successive white balls
a
ac
a  2c
pW 
a  b a  b  c a  b  2c
a  (k  1)c
a  b  (k  1)c
PK :probability of drawing k white balls ,followed by
n – k black balls
b
bc
pk  p w
a  b  kc a  b  ( k  1)c
k 1
n  k 1
i 0
j 0
  a ab icic

b  ( n  k  1)c
a  b  (n  1)c
b  jc
.
a b( j  k )c
Sharif University of Technology
64
Example – cont’d
• Polya distribution: probability of getting k white balls in n draws.
P( X  k ) 
n
 
k 
 
pk 
k 1
n
 
k 
  i 0

a ic
a  b ic
n  k 1

j 0
b  jc
,
a b( j  k )c
k  0,1, 2,
, n.
• if draws are done with replacement, then c = 0 and the above
formula simplifies to the binomial distribution
P( X  k ) 
n
 
k 
 
pk qn k ,
k  0,1, 2,
Sharif University of Technology
,n
65
Example – cont’d
• if the draws are conducted without replacement, then c = – 1 and it
gives
P( X  k ) 
n
 
k 
 
P( X  k ) 
a (a  1)( a  2) (a  k  1)
b(b  1) (b  n  k  1)
(a  b)( a  b  1) (a  b  k  1) (a  b  k ) (a  b  n  1)
n!
a !( a  b  k )!
b !( a  b  n )!

k !( n  k )! ( a  k )!( a  b)! (b  n  k )!( a  b  k )!
a  b 

  
 k   n k 
  

 a b 


 n 


• which represents the hypergeometric distribution.
Sharif University of Technology
66
Example – cont’d
• Finally c = +1 gives (replacements are doubled)
P( X  k ) 
=
n
 
k 
 
( a  k  1)! ( a  b  1)! (b  n  k  1)! ( a  b  k  1)!
( a  1)! ( a  b  k  1)!
(b  1)! ( a  b  n  1)!
 a  k 1   b  n  k 1

 


 

k
n

k

 

 a  b  n 1 




n


.
• This is Polya’s +1 distribution.
Sharif University of Technology
67