Good Review Materials
http://www.imageprocessingbook.com/DIP2E/dip2e_downloads/review_material_downloads.htm
Random Variables
and
Random Vectors
• (Gonzales & Woods review materials)
• Chapt. 1: Linear Algebra Review
• Chapt. 2: Probability, Random Variables,
Random Vectors
Tim Marks, Cognitive Science Department
Tim Marks, Cognitive Science Department
Random variables
Discrete random variables
• Samples from a random variable are real numbers
• Discrete random variables have
a pmf (probability mass function), P
P(X = a) = P(a)
• Example: Coin flip
1
– A random variable is associated with a probability distribution
over these real values
– Two types of random variables
• Discrete
– Only finitely many possible values for the random variable:
X ∈ {a1, a2, …, an}
– (Could also have a countable infinity of possible values)
» e.g., the random variable could take any positive integer value
– Each possible value has a finite probability of occurring.
X = 0 if heads
X = 1 if tails
– What is the pmf of this
random variable?
• Continuous
– Infinitely many possible values for the random variable
– E.g., X ∈ {Real numbers}
P (a )
0.5
0
0
Tim Marks, Cognitive Science Department
Continuous random variables
• Discrete random variables have
a pmf (probability mass function), P
1
P(X = a) = P(a)
P (a )
• Example: Die roll
• Continuous random variables have a
pdf (probability density function), p
• Example: Uniform distribution
p(x)
p(1.3) = ?
0
0
1
2
3
4
5
6
a
p(2.4) = ?
What is the probability
that X = 1.3 exactly:
P(X = 1.3) = ?
0.5
0.5
Tim Marks, Cognitive Science Department
a
Tim Marks, Cognitive Science Department
Discrete random variables
X ∈ {1, 2, 3, 4, 5, 6}
– What is the
pmf of this
random variable?
1
1
2
x
Probability corresponds to area
under the pdf.
1.5
P(1 < X < 1.5) = ∫? p(x) dx = 0.25
Tim Marks, Cognitive Science Department
1
–1
Continuous random variables
Random variables
• What is the total area under any pdf ?
• How much change do you have on you?
∞
– Asking a person (chosen at random) that question can
be thought of as sampling from a random variable.
∫ p( x)dx = 1?
−∞
• Example continuous random variable:
Human heights
p(h)
p(h)
• Is the random variable
“Amount of change people carry”
discrete or continuous?
p(h)
h
h
h
Tim Marks, Cognitive Science Department
Tim Marks, Cognitive Science Department
Random variables: Mean & Variance
An important type of random variable
• These formulas can be used to find the mean and
variance of a random variable when its true
probability distribution is known.
Definition
Mean
µ
µ = E(X)
Variance
E ((X − µ) 2 )
Var(X)
Discrete r.v.
Continuous r.v.
µ = ∑ ai P (ai )
µ=
2
i
∫ x p(x) dx
−∞
i
∑ (a − µ)
∞
∞
P(ai )
∫ (x − µ)
2
p(x) dx
−∞
i
Tim Marks, Cognitive Science Department
Tim Marks, Cognitive Science Department
The Gaussian distribution
p(x) =
1
2π σ
e
Estimating the Mean & Variance
−
( x − µ )2
2σ
2
X ~ N(µ, σ 2)
Tim Marks, Cognitive Science Department
– After sampling from a random variable n times,
these formulas can be used to estimate the
mean and variance of the random variable.
• Samples x1, x2, x3, …, xn
Estimated mean:
m=
1 n
∑ xi
n i=1
Estimated variance:
σ2 =
1 n
∑ (x i − m)2
n i=1
σ2 =
1
∑ (x i − m)2 ← unbiased estimate
n −1 i=1
← maximum
likelihood estimate
n
Tim Marks, Cognitive Science Department
–2
Finding mean, variance in Matlab
Example continuous random variable
– Samples x = [x1 x 2 x 3 L x n ]
– Mean
>> m = (1/n)*sum(x)
• People’s heights (made up)
– Variance
1
σ = [x1 − m
n
2
Method 1:
Method 2:
 x1 − m 


x − m
x 2 − m L x n − m]  2
 M 


x n − m
>> v = (1/n)*(x-m)*(x-m)’
>> z = x-m
>> v = (1/n)*z*z’
– Gaussian
µ = 67 , σ2 = 20
• What if you went
to a planet where
heights Gaussian
µ = 75 , σ2 = 5
– How would they be
different from us?
Tim Marks, Cognitive Science Department
Tim Marks, Cognitive Science Department
Example continuous random variable
• Time people woke up this morning
– Gaussian
µ = 9 , σ2 = 1
Random vectors
– An n-dimensional random vector consists of n
random variables that are all associated with the same
events.
– Example 2-D random vector:
X 
V = 
Y 
where X is random variable of human heights
Y is random variable of wake-up times
– Sample n times from V.
v1 v 2 L v n
x1 x 2 L x n 


y1 y 2 L y n 
Tim Marks, Cognitive Science Department
Random Vectors
• What will the graph of V look like?
67
m = 
10
Tim Marks, Cognitive Science Department
x (heights)
Mean of a random vector
• Estimating the mean
of a random vector
Mean
– Mean of X is 67
– Mean of Y is 10
y
(wake-up
times)
Tim Marks, Cognitive Science Department
– n samples from V
• What is mean of V ?
Let’s collect some
samples and graph them:
m=
v1 v 2 L v n
x1 x 2 L x n 


y1 y 2 L y n 
n
n
1
1 x  m x 
v i = ∑ i  =  
∑
n i=1
n i=1 y i  m y 
– To estimate mean of V in Matlab
>> (1/n)*sum(v,2)
Tim Marks, Cognitive Science Department
–3
Random vector
– Example 2-D random vector:
X 
V = 
Y 
where X is random variable of human heights
Y is random variable of human weights
v1 v 2 L v n
x1 x 2 L x n 


y1 y 2 L y n 
• Sample n times from V
– What will graph
look like?
Covariance of two random variables
• Height and wake-up time are uncorrelated, but
height and weight are correlated.
• Covariance
Cov(X, Y) = 0 for X = height, Y = wake-up times
Cov(X, Y) > 0 for X = height, Y = weight
– Definition:
Cov(X,Y ) = E ((X − µx )(Y − µy ))
If Cov(X, Y) < 0 for two random variables X, Y , what would a
scatterplot of samples from X, Y look like?
Tim Marks, Cognitive Science Department
Tim Marks, Cognitive Science Department
Estimating covariance from samples
• Sample n times:
Cov(X,Y ) =
Cov(X,Y ) =
x1

y1
Estimating covariance in Matlab
– Samples
x2 L xn 

y2 L yn 
– Means
mx ← m_x
my ← m_y
x = [x1 x 2 x 3 L x n ]
y = [y1 y 2 y 3 L y n ]
1 n
∑ (x i − mx )(y i − my )
n i=1
← maximum
likelihood estimate
1 n
∑ (x i − mx )(y i − my )
n −1 i=1
← unbiased estimate
– Covariance
1
Cov(X,Y ) = [x1 − m x
n
x 2 − mx
 y1 − m y 


y 2 − my 

L x n − mx ]
 M 


y n − my 
• Cov(X, X) = ?Var(X)
Method 1: >> v = (1/n)*(x-m_x)*(y-m_y)’
• How are Cov(X, Y) and Cov(Y, X) related?
Method 2: >> w = x-m_x
>> z = y-m_y
>> v = (1/n)*w*z’
Cov(X, Y) = Cov(Y, X)
Tim Marks, Cognitive Science Department
Covariance matrix of a
D-dimensional random vector
(
X − µX 
= E
[X − µX
Y − µY 
Example covariance matrix
• People’s heights
(made up)
• In 2 dimensions
X 
V = 
Y 
T
Cov(V) = E (V − µ )(V − µ )
Tim Marks, Cognitive Science Department
X ~ N(67, 20)
)
  Var(X ) Cov(X,Y )
Y − µY ] = 

Var(Y) 
 Cov(X,Y )
• In D dimensions
T
Cov(V) = E ((V − µ )(V − µ ) )
• When is a covariance matrix symmetric?
A. always, B. sometimes, or C. never
Tim Marks, Cognitive Science Department
• Time people woke up
this morning
Y ~ N(9, 1)
• What is the covariance
matrix of
X 
V = 
Y 
?
20 0


 0 1
Tim Marks, Cognitive Science Department
–4
Estimating the covariance matrix from
samples (including Matlab code)
– Sample n times and find mean of samples
v1 v 2 L v n
x
V = 1
y1
x2 L xn 

y2 L yn 
1
Cov(V ) =
n
 x1 − m x

 y1 − m y
• 1-dimensional Gaussian is completely determined
by its mean, µ, and variance, σ 2 :
mx 
m= 
my 
– Find the covariance matrix
 x1 − mx

x 2 − mx L x n − mx   x 2 − mx

y 2 − my L y n − my   M

x n − mx
>> m = (1/n)*sum(v,2)
>> z = v - repmat(m,1,n)
>> v = (1/n)*z*z’
Tim Marks, Cognitive Science Department
Gaussian distribution in D dimensions
X ~ N(µ, σ 2)
y1 − m y 

y 2 − my 
M 

y n − my 
p(x) =
1
2π σ
e
−
(x− µ )2
2σ
2
• D-dimensional Gaussian is completely determined
by its mean, µ, and covariance matrix, Σ :
X ~ N(µ, Σ)
p(x) =
1
(2π )
D/2
Σ
1/ 2
e
− 12 (x−µ )T Σ −1 (x−µ )
–What happens when D = 1 in the Gaussian formula?
Tim Marks, Cognitive Science Department
–5