Covariance of two random
variables
•
•
Height and wake-up time are uncorrelated, but height and weight
are correlated.
Covariance
Cov(X, Y) = 0
for X = height, Y = wake-up times
Cov(X, Y) > 0
for X = height, Y = weight
• Definition:
Cov( X , Y ) % C XY % E !( X $ # x )(Y $ # y ) "
Question: If Cov(X, Y) < 0 for two random variables X, Y , what
would a scatterplot of samples from X, Y look like?
Question: If we add arbitrary constants to the random variables X, Y ,
how does the covariance change?
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch
13
Estimating covariance from samples
Again, we assume that we do not know the
underlying probability distributions. But consider
we sample n times and estimate:
1 n
Cov ( X , Y ) - , ( xi $ mx )( yi $ m y )
n i -1
Questions:
What is Cov(X, X) ?
How are Cov(X, Y) and Cov(Y, X) related?
+ x1
)y
* 1
x 2 ! xn (
y2 ! yn &'
“sample covariance”
Cov(X, X) = Var(X)
Cov(X, Y) = Cov(Y, X)
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch
14
Estimating covariance in Matlab
•
Sample mean:
x - [ x1 x2 x3 ! xn ]
y - [ y1 y2 y3 ! yn ]
mx
my
. m_x
. m_y
+ y1 $ m y (
)y $ m &
1
y&
2
Cov( X , Y ) - [ x1 $ mx x2 $ mx ! xn $ mx ] )
) " &
n
)
&
)* yn $ m y &'
Method 1: >> v = (1/n)*(x-m_x)*(y-m_y)’
•
Covariance
Method 2: >> w = x-m_x
>> z = y-m_y
>> v = (1/n)*w*z’
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch
15
Coefficient of Correlation
The coefficient of correlation is defined as:
0 xy %
E !( X $ # x )(Y $ # y ) "
/ x/ y
-
Cov ( X , Y )
/ x/ y
Properties:
• -1 ! Cxy ! 1
• if Cxy = 0 : X and Y uncorrelated
• if Cxy bigger/smaller zero : X and Y are positively/negatively
correlated
• Advantage: we can multiply X and Y with arbitrary factors and
Cxy stays the same.
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch
16
Correlation of two random
variables
Definition:
C orr ( X , Y ) - E! XY "
If X and Y have zero mean, this is the same as the covariance.
If in addition, X and Y have variance of one this is the same as
the coefficient of correlation.
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch
17
Correlation, Covariance, Corr.Coeff.
C orr ( X , Y ) - E! XY "
Correlation
Cov( X , Y ) - E !( X $ # x )(Y $ # y ) "
Covariance
C xy -
E !( X $ # x )(Y $ # y ) "
/ x/ y
-
Cov ( X , Y )
/ x/ y
Coefficient of Corr.
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch
18
Covariance Matrix
Covariance:
Cov( A, B ) - E !( A $ # A )( B $ # B ) "
Now consider random vector:
X - ( x1 , x2 , ! , xn )T
We can compute covariance
between two components, say
between x2 and x5:
Cov( x2 , x5 ) - c25 - E ( x2 $ # x2 )( x5 $ # x5 )
Doing this for all combinations
gives us the elements of the
covariance matrix:
!
6 c11
4
4 c12
7X % 4 "
4
4
4c
5 1n
c21 !
c22
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch
#
!
"
cn1 3
1
1
" 1
1
1
cnn 12
19
Properties of Covariance Matrix
6 c11
4
4 c12
7X % 4 "
4
4
4c
5 1n
c21 !
c22
#
!
cn1 3
1
1
" 1
1
1
cnn 12
!
Cov( xi , x j ) - cij - E ( xi $ # xi )( x j $ # x j )
"
It is symmetric, because cij - c ji .
2
Its diagonal elements are the individual variances: cii - / i - var( xi ) .
If it is diagonal, the xi are all
6 / 12
0 !
0 3
1
4
uncorrelated and we have:
2
1
4 0 /
7 X % 44 "
4
44
5 0
2
#
!
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch
1
" 1
1
1
/ n 2 12
20
Example covariance matrix
people’s heights:
X ~ N(67, 20)
time people woke up this
morning: Y ~ N(9, 1)
Question: what is the covariance matrix of V = (X Y)T ?
X and Y should be uncorrelated:
+X (
V -) &
*Y '
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch
+20 0(
) 0 1&
'
*
21
Estimating the covariance matrix from
samples (including Matlab code)
Sample n times and find mean of samples:
v1
+ x1
V -)
* y1
v2 ! vn
x2 ! x n (
y2 ! yn &'
+ mx (
m-) &
*m y '
Find the covariance matrix:
1
Cov (V ) n
+ x1 $ mx
)y $ m
y
* 1
x2 $ m x
y2 $ m y
+ x1 $ mx
! xn $ mx ( )) x2 $ mx
! yn $ m y &' ) "
)
)* xn $ mx
y1 $ m y (
y2 $ m y &&
" &
&
yn $ m y &'
>> m = (1/n)*sum(v,2)
>> z = v - repmat(m,1,n)
>> v = (1/n)*z*z’
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch
22
Gaussian distribution in D dimensions
Of course, the most important distribution can also be extended to higher
dimensions. Recall that a 1-dimensional Gaussian is completely
determined by its mean, #, and its variance, /82:
X ~ N!#:
/82)
1
p( x) 2" /
$
e
( x$# )2
2/ 2
A D-dimensional Gaussian (multivariate Gaussian) is completely
determined by its mean, #, and its covariance matrix, 79
X ~ N!#: 7)
p ( x) -
1
(2")
D/2
$ 12 ( x $µ ) T 7 $1 ( x $µ )
7
1/ 2
e
Question: what happens when D = 1 for the D-dimensional Gaussian?
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch
23
The Gaussian in D dimensions
Question: What does a set of equiprobable points look like for a
2-dim. Gaussian? What for a D-dim. Gaussian?
In 2D, it’s an ellipse. In D dimensions, it’s an ellipsoid.
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch
24
Equiprobable contours
of a Gaussian
If a Gaussian random vector has covariance matrix that is diagonal
(all of the variables are uncorrelated, then the axes of the ellipsoid
are parallel to the coordinate axes.
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch
25
Equiprobable contours
of a Gaussian
If a Gaussian random vector has covariance matrix that is not
diagonal (some of the variables are correlated), then the axes of the
ellipsoid are perpendicular to each other, but are not parallel to the
coordinate axes.
Jochen Triesch, UC San Diego, http://cogsci.ucsd.edu/~triesch
26