Those who ignore Statistics are condemned to reinvent it.
Brad Efron
Lecture 15: Expectation for
Multivariate Distributions
Probability Theory and
Applications
Fall 2008
Outline
•
•
•
•
•
Correlation
Expectations of Functions of R.V.
Covariance
Covariance and Independence
Algebra of Covariance
Correlation Intuition
• Covariance is a measure of how much RV
vary together.
Wife’s Age and Husband’s Age
Correlation .97
Example from http://cnx.org/content/m10950/latest/
Sometimes not so perfect
Arm Strength Versus Grip Strength
Pearson’s Correlation R=.63
Negative Correlation
Child Labor versus GDP
Extreme Correlation 1
Linear relation with positive slope
Extreme: Correlation -1
Linear relation with negative slope
Zero Correlation
Independent Random
Guess Covariance???
Positive, Negative, 0
•
•
•
•
•
•
Crime Rate, Housing Price
SAT Scores, GPA Freshman Year
Weight and SAT Score
Average Daily Temperature, Housing Price
GDP, Infant Mortality
Life Expectancy, Infant Mortality
Expectations of Functions of R.V.
 
  g ( x, y) f ( x, y)dydx
E ( g ( x, y )) 
 
 
E( X  Y ) 
  ( x  y) f ( x, y)dxdy
 





  x   f ( x, y )dy  dx   y   f ( x, y )dx  dy
  


 




 xf


x
( x)dx 
 yf
y
( y )dy  E ( X )  E (Y )

NOTE substitute appropriate summation for discrete
Variance and Covariance
Univariate becomes variance
var( X )  E ( X   X ) 2    ( x   X ) 2  f X ( x)
x
Multivariate becomes covariance
cov( X , Y )  E  ( X   X )( y  Y ) 
   ( x   X )( y  Y )  f ( x, y )
x, y
Note:
var( X )  cov( X , X )
NOTE substitute appropriate integral for continuous
Calculating Covariance
Can simplify
cov( X , Y )  E  ( X   X )(Y  Y ) 
   xy  y  X  xY   X Y  f ( x, y )
x, y
   xy  f ( x, y )     y  X   xY  f ( x, y )   X Y
x, y
x, y
 E ( XY )   X Y   X Y   X Y
 E ( XY )  E ( X ) E (Y )
Correlation of X and Y
Definition
 ( X ,Y ) 
cov( X , Y )
 XY
The correlation always falls in [ -1, 1]
It a measure of the linear relation between X
and Y
Extreme Cases
If X=Y then ρ=1.
If X=-Y then ρ=-1.
If X and Y independent, then ρ=0.
If X=-2Y then ρ=?.
Example
2 x  0, y  0, x  y  1
f ( x, y )  
o.w.
0
Joint is
Find correlation of X and Y
1 x
1 y

fY ( y ) 
2dx  2(1  y ) 0  y  1
f X ( x) 
 2dy  2(1  x)
0  x 1
0
0
1
1
E (Y )   2 y (1  y ) dy  1/ 3
E ( X )   2 x(1  x) dx  1/ 3
0
0
1
E (Y )   2 y (1  y ) dy  1/ 6
E ( X )   2 x 2 (1  x) dx  1/ 6
var(Y )  1/ 6  1/ 9  1/18
var( X )  1/ 6  1/ 9  1/18
1
2
2
0
2
0
Example
Joint is
2 x  0, y  0, x  y  1
f ( x, y )  
o.w.
0
Find correlation of X and Y
1 1 y
E( X ,Y )  

0 0
1
2 xydxdy   y (1  y ) 2 dy  1/12
0
cov( X , Y )  1/12  1/ 9  1/ 36
cov( X , Y )
1/ 36
 ( X ,Y ) 

 1/ 2
 XY
1/18 1/18
Properties of Covariance
a) var( X )  cov( X , X )
b) cov(aX+bY)=ab[cov(X,Y)]
E (aXbY )  abE ( XY )
E (aX )  aE ( X )
E (bY )  bE (Y )
cov(aX  bY )  E (aXbY )  E (aX ) E (bY )
 ab[ E ( X , Y )  E ( X ) E (Y )]  ab cov( X , Y )
Properties of Covariance
c) var(aX )  a 2 var( X )
d)  (aX  bY )  sign(a * b)  ( X , Y )
ab cov( X , Y )
 (aX  bY ) 
 ( X ,Y )
| a |  X | b | Y
ab

 ( X , Y )  sign(a * b)  ( X , Y )
| a || b |
Properties of Covariance
e) If X and Y are independent
cov( X , Y )  0
correl ( X , Y )  0
Proof:
E ( X , Y )    xyf x ( x) f y ( y )dxdy
  xf x ( x)dx  yfY ( y )dy  E ( X ) E (Y )
cov( X , Y )  E ( XY )  E ( X ) E (Y )  0
Find Covariance
X\Y
0
1
-1
0
.3
0
.4
0
1
0
.3
Are X and Y independent
X\Y
0
1
-1
0
.3
0
.4
0
1
0
.3
Note
Cov(X,Y)=0 does not imply independence of
X and Y
Independence of X and Y implies
cov(X,Y)=0
In this case Y=X2 so the variables are
definitely not independent but their
covariance is 0 because they have no
linear relation.
Algebra of
variance/covariance/correlation
Given: E ( X )  Y
var( X )   X2
E (Y )  Y
var(Y )   Y2
cov( X , Y )   XY
Calculate mean of Z=2X-3Y+5
variance of Z=2X-3Y+5
Long steps
E ( X )  2 X  3Y  5
Z  E ( Z )  2( X   X )  3(Y  Y )
[ Z  E ( Z )]2  4( X   X ) 2  3(Y  Y ) 2  12( X   X )(Y  Y )
E [ Z  E ( Z )]2   4 var( X )  8 var(Y )  12cov( X , Y )
Working Rules for linear
combinations
Z  2 X  3Y  5
Write formula
2 X  3Y
Discard Constants
2
2
4
X

9
Y
 12 XY
Square it
Replace squared R.V 4 var( X )  9 var(Y )  12cov( X , Y )
with var and crossterms
with cov
Example
Given var(X)=4
var(Y)=10
ρ(X,Y)=1/2
Find variance of X-5Y+6?
Given same facts as previous problem
Find covariance x-5y+6 and -4X+3Y+2
Working rule works also for more
than two variables
Find variance of W=2x-3Y+5Z+1