Chapter 1 Basic Probability Theory
Dep
An outcome
specific result
a
is
Def The sample space S
Def
An event A
is
is the
set
of all possible outcomes
subset of outcomes
a
Def The events Ai Az
in
S
partition of S if they
mutually disjoint and their union is S
Theorem
If
are
RBs Bz
is
a
a
partition of S
then for any
event
A
YILANBi
A
S
B
are
Bz
B
A AABst An Bat AnBz
g
Def A
P
function
called
which assigns a numerical value
to events
is
probabilitypunction if it satisfies the Axioms of
a
Probability
1 PLA
P s
2
3
P
is
O
L
If AsAz
a
PCAUB
function
are
disjoint
PffA
from the space
PlA PCB if A n B
are
disjoint
ÉPLA
of events to
the real line
Theorem
Forevents A and B
1 PLAY
I PCA
2 PCO
O
3 PCA El
4
If
p
then PLA s PCB
ACB
5 PLAUB
7 PLANB Z
If
likely
Dep
If
PCB
PCB
PAIT PCB
an experiment
symmetric
the
in
sense
outcomes
ai
an
that each outcome
which
are
is equally
IN
P ai
then
L
has N
the conditional probability of A given B is
0
PLANB
PCB
PCAIB
Dep
PLANB
PCA t PCB
61 PLAUB EP A
Theorem
probability
Largersets have larger
The events A and B
are independent
A and B have absolutely
outcome of one has
no
no
effect
influence
on
if PCANB PCA PCB
on
each other
the outcome
the
of
the other
Theorem
If A and B
then P AIB
are independent with PCA
PCA
PCBIA
PCB
O and PCB
o
Theorem
If
Ba Ba
then
Theorem
If
is
0
of S and P Bi
PCB
a
so
P BIA P A
P AIB
P
P BIA PCA PCBAc
B of
Def A collection
O for all i
É PLAIBIP Bi E P AMBI
PLA
PLA
a partition
sets is
a
pay
sigma
BIA PCA
PCB
field if it satisfies
the following 3 properties
HOEB
2
3
EB then A EB
If As Az EB then QiAi EB
If
A
Def The Borel Sigma field is the smallest sigma field on IR
It contains all open intervals
containing all open intervals
at
and closed intervals and their countable unions intersections
and
A
of
complements
be generated from a finite collection
by taking all unions intersections and
sigmafield can
events
complements
Chapter 2 Random Variables
Dep
A
random variable is
the sample
real valued
a
S
space
outcome a ruction from
to the real line IR
Represent random outcomes numerically
Think of
value is
number or
Dep
Dep
X
a random variable
and
unknown
t
is
discrete if
infinite number
elements
X
there is
is
a
with this
realization
a
as
a
specific
outcome
The set it
If
as a random object whose
of
a
discrete set
it has
It
a
finite
such that
or countably
PEX et
I then
discrete random variable The smallest set It
property is
I The support
is
the
positive probability
Jt 3Taitz
of
try
X
the
support
range
of outcomes which
of
receive
occurrence
support points
Def The probability mass function of a random variable is ITA PK
the probability thattherandom variable X equals a specific value x
When evaluated at the suppoint points
4Function that gives
the
IT
we write
IT IT es
probability that a discrete random variable
is exactly equal to some value
x
If X is
Theorem
g It
function
random
Def For
the
V g X for
G C IR then Y is also
expectation
random variable
X
of
series
X
a
is
X with
25
support
is
TT
ETH
While
some
variable
a discrete
if the
and
a random variable
non convergent
Etr
is
either infinite or notdefined
is conveyed
random
If
Ete
is non random since
itis
a
fixed feature of the distribution
Theorem
Proof
For any
Etat GH
Etatbe
IE
at
GEEK
at
bij
IT
FEET
a
a t
Def The
and b
constant a
b EEN
distribution function
CDF
is
a
discrete random variable with
FCT
Cumulative
sum of
É TE
the
PLA E x
the
3143
probability of the event
For
FL
support
probabilities less
points
than j
I
The CDF is constant between the support points
faction with jumps at each support point
Theorem
If FA
is
CDF
a
s
FG is
2
EM Feel
0
3
fits Fk
L
4
Feel is right continuous
At
non decreasing
right since
If Xu Fk
meaning
fix FG
Fk
points where
discontinuous
Def
step
and
FK has a step Feel is
to the left but continuous t the
FL
Fk
PIKE
is continuous
x
then X
is continuous
random variable
PH
EM PEX
x
The
probability that
equal
PUA
to
a
big Fete
X
FA
o
a continuous random variable is
be
specific value must
PLAS X
PAIN PH
exte
FK
L FK
zero
Def When FLA is differentiable its density
Theorem
A function fix
a
is
density function
PDF is
fee
if and only if
I FCA 20 HX
Ifield
2
Plaek b
L
Jab field
The probability that X is in the interval ta b is
the integral of the density over ta b This shows
that the area underneath the density functions are
probabilities
Def The support It of a continuous random
smallest set containing Reifel x
Def
If X is
continuously distributed
expectation is defined
Def
the
is the
with density feel
its
as
Exfelde
EFX
when
variable
integral is convergent
The
mean of
X
The
variance
of X is
is
M ETX
EE
var
X
ETH
ER
8220
X
is
degenerate
ie
X
if for some
is non random
r
52
C
0
Pk
c
L
dad
The
Theorem var
ETH
X
For any
random variable
is
X if get
EIGHT
g EEA
function of an
than the expectation of
EFX
then
a concave function
convex
is a convex
SEEK
g ETA
4 A
Vr
o
EEN
function then
If get
is
62bark
Theorem Var lat bl
Theorem
X
standard deviation of
expectation
the
is
less
transformation
E ETXT
Etlog KDE log EET
exp EEN SETUP XD
The moments of a
the powers of X
distributions are
Dep The Mth moment of
Def For
Mm
m
I
moment of
X
ETH ETHIm
moments
finite infinite
finite or infinite
values of
ELI
is M'm
the mth central
Odd moments
Even
X
the expected
or
undefined
is
Def The moment generating function
M
The
MGF is either finite
or
4 Non negative
Theorem
If MIA
finite for
is
O then
D MIO
L
in OYE
to
iii
5Mt
JE
Itm
in
The
the
curvature of
distribution
or
X
is
Jeep tx fade
Etexplex
t
MOF
to
to
Mct at
of X
t in
a
infinite
neighborhood
of
ELF
E x2
ETXm
for any
moment which
is finite
to encodes all moments
of
Chapter 4 Multivariate Distributions
Dep
A pair
of bivariate random variables is
numerical
outcomes
a function
pair
a
of
from the sample space
to IR
Dep
A pair
of bivariate random variables
variables
with
a
random
distribution
joint
The joint distributionfunction of X Y is
F x g
PIX ex YE y P HEA n 38 93
F
4
weakly increasing in each argument and
satisfies O E F x y
1
is
Satisfies the following
Plack sb C Y Ed
Flak
Y
d
F bd
b
FaC
c
Fla d
F la c
X
F 6c
C
A pair
a
Fb
F bd
a
Dep
are two
of random variables is discrete if there is
discrete set 94112 such that
Y ES
L
I
PIG
support of
TEsTE
XY
and
Rt's TY
Dep The
joint probability mass function
ITCHY
PEX
x
Def When F
xy
is
Y y
and differentiable its
continuous
density equals
The
b
Flay
Roy
probability
that the
region in 1122 equals
this
Dep
The
Jfk
Y ed
c
marginal
in
under the density
area
distribution
PAIN
of
X
PIX
In the
is
ex
YEA
Fay
lying
continuous case
Fx k
lying
I If
uu
The
marginal densities of
are
density f x y
the
In
dedy
pair X Y lies
random
the
g
region
Fx K
Dep
joint
O
fley
L Plas
is
II fix
practice we
g dy
treat
du du
X and Y
Fy
a
y
marginal
Iff
given
curldudu
a
joint
fay de
PDF the
same
a
over
Def The expectation of real valued g K Y
Efg X Y
ay
Ernay
for the discrete
case
for the
continuous case
gk
go y f
I Ight
X
y de dy
fay dxdy
discrete distribution
has a
distribution function
of Y given
Fy lyin PLYEgl X
For
any
e such
Pl X
that
4 Distribution function
Def
If Fuk lyk
PLAe x
given
o
X
X
on one variable
Ig
I Ix fee y dedy It
E TX
If
x
only depers
E Ight
Dep
and
II
If
949 TWD
o
Efg X Y
Note
is
x
for the
e
e
E
x
de
the conditional
K
x
is
x
o
subpopulation where
tix
is differentiable withrespect to y and
then the conditional density function of
is
FK
y
x
de
OFK GK
Oy
Y
Density
Def For
Y for the
of
X and Y the
continuous
given
X
X is
Fk
for any
subpopulation where
x
let A Hex
tix
conditional density or
Y
FS
y x
FK
such that
Fx
t
so
B 31 193 The probability that they
both occur is
PLANB
PLA e x Y Ey
Flag
The product of their probabilities is
PCA
and
PIX Ex PLYEy
PIB
Def The
random variables
if for all
k
n
are statistically independent
Fy
x
Ffs
probability is equal
to the product of
Def The discrete random variables X and
if for all
k
n
they
Y are
probabilities
statistically independent
y
TK g
Ip X and Y
s
y
FG g
The joint
X and Y
EG E
Tx
have
are statistically
x
Ty y
differentiable distribution function
independent if
for all k n y
a
x
Theorem
If X
and
Y are
independent and continuously
distributed
f
hey
Fuld
x
FXIX K ly
If
X and Y
FX
x
independent
are
the
conditional density
the marginal density i.e he does not
affect the shape of the density of Y
equals
Theorem
Bayes Theorem for Densities
Fxx
Fy k SK
Theorem
If X and Y
rly
FA x
are
fy
y
independent
g IR IR and hi IR IR
Eth Y Loo
such
EISK HH
Def
If
X and Y have
finite
then for any
that
Efg X
functions
200 and
EIGHTEEN Y
variances
X and Y
is
Cov X Y
ELK ELKIN EEN
EEN EEN EEN
the
covariance between
Def
If X and Y have
K and Y is
finite
the
variances
correlation
between
Cork Y
Corr X Y
varlet var Y
Theorem
X and Y are independent with finite variances
then ETH ETH ETH and hence Cov X Y 0
Theorem
If X and Y have
If
bark
var Ktv
Theorem
For any
variances
finite
var
Y
random variable
X
2 Cov X Y
and Y
EVEENELT
ETH
Cauchy Schwarz inequality
Dep
The
conditional expectation of
expected value
of the
and is written as
For discrete variables
ELY he
For
continuous
Y
ELY Xx
Ygiven Xx
conditional
distribution
ETUI text
me
the
is
fuk
lyk
Y within
Expected value on
FEI
Tx
the subpop
The
population
f
which
tee
e
Expectedvalue of
g
for
fun
Y within the infinitesimally small
for which
ye
dy
tax
t
Y
flat dy
Iffhs
dy
Remember
Et VI X x
is not random it is a feature of
the joint distribution
However Et VIX is a random variable a transformation
of X In other words EE YIN is a function of
the
random
variable
X
JEENA FAH de
ELENA
II
Iffy
Y
FK lyk
Exceldyde
flay dydx
E EY
The average
Theorem
If ELY
across
Loo
group averages is the grand average
then ELETTIN
EET
conditional variance of Y given X x is the
variance of the conditional distribution Fuk six
and is written as var Y l x or 8241 It equals
Def The
var
HK
ETH ELINT
ELVIRA
Theorem
vartl
LELY
Xx
KIK
EEYHtvarg.IE
variance
variance
Dep
A
multivariate random vector is
a
space
Dep
to
112m
written
The joint distribution
F
PA
x
Def When FG is
Xm
Xm Xm
the joint probability
vectors
mass
P Xx
IT e
is
ki ka
sample
function is
PCIE Xs
ex
Def For discrete random
function
X
as
from the
function
continuous
and
differentiable
its joint
density feel equals
F
Def The
expectation
of its
OMFG
X
ON
of
r
X EIR
OXm
is
the
vector or expectations
elements
Eth
EEK
EFX
Il
Def The mem
covariance
varTX
z
matrix of
Xt IR
is
ETH ETH X EEN
I
EE
i i
É
One
i
one
Z
Theorem
Any
mem covariance matrix
I Symmetric
2
Theorem
Positive
Z
satisfies
Z
For any
semi definite
If X EIR with
I
Mtt
ato
a
mass expectation
M and mem covariance
matrix I and A is gem then AX is a random
vector with mean AM and covariance matrix AZA
Dep The joint distribution function of X HE IRMARm
FG g
P XS x Y Ey
Dep The joint density function of X HE IRMARm
f
x
y
datmy F
Fx
density functions
Fy
Def The conditional densities of
Y
y
is
OMy
Jfk yldy
x
is
xy
DX
Def The marginal
of
X and Y are
fay de
y
Y given X
x
and X given
are
FK SK
Def
za z o
Iff
finely
III
The conditional expectation of Y given Xx
ELY X X Syfy yle dy
is
Chapter 7 Law of Large Numbers
Large sample asymptotic approximations
Def A
n
has
an
sequence
ou lying an
oo
a
limit
a
Example
for
0
tanks
to 0
and
1
Fix 820
2
Find As such that Ian
3
If
2 can be done for
converges
to 0
n
Let 8
If
an
a
as
there is some ng Coo
n
An
as
a
Converges
Ovitt
solving
written
od
to show an
found by setting
oo
Ian al ES
Mm
Proof
a
if for all 8
such that for all n ng
an
as n
12
n
0
Is
8
In
then Ian I
S
e
s
A ES
n
8 for ne ng
any 8 then an
Def
A
sequence
probability
for all
We call
8
c
of random variables Zn e IR converges in
to
c
as n soo
denoted
Play Zn
c
if
0
big PII Zn
c
hit Pt
4 s
Zn
L
E8
or
o
the plim of Zn
of random variables concentrating about a point
For any Sso the event 312n c 283 Occurs when
Zu is within 8 of C
Pt Zn c les is the probability
Sequence
Pt Zn CI es
approaches
I
as
n
Note The definition concerns the distribution
variable not the realizations
oo
of
the random
Note The definition uses the concept of a conventional
limit but applied to a sequence of probabilities not
directly to Zn
or
their
realizations
Theorem For
any random
Chebyshev
variable
X
and any 8 0
EtK EtX
py.fm zgyevarty
82
4 The probability that the randomvariable differs fromitsmean by
with finite mean me and finite variant
Take any X
PIK M1
We want to bond the probability
distributions and all 8
Theorem
For any
sequence
Vartzn
8
for all
of random variables Zn such that
Zn EEZn
then
O
P
O as
N soo
0
oh
is sufficient
Chebyshev's
for
Inequality
Theorem For
any random variable X and any 8
Markov
ETH1141412837
Et
pix
Theorem
WUN
gy
ti
If
EIA
8
independent and identically distributed
are
then
L OO
Samplemeant
sample
expectation
Proof
Since
mean
approach the
can
PC
converges
4 For any distribution
Xi
is
use
H
M
as
I I Xi
An
The
0
a
the
E
n
00
this equivalent to
P
ELK
fit P
o
in probability to the population
with a finite expectation thesample mean
population expectation
random variable
in probability as
with
Chebyshev Inequality
s
Itn Mlse
and
Varlet
where
nooo
ETH
we
Eiti
In
PL
IE MI
as n
Varki
NTI E
o
850
of a parameter is consistent if
00
There is
the
f
Eez
E
E
É
An estimator
Dep
Var In
size n sufficiently large such that
sample
a
I will
estimator
be arbitrarily close to the true
0
value
with high probability
Dep
A
function
Ht
CH E S
A
function
hat
is
continuous
that
hall
is continuous
if
at
te
if
te
o
78
o
E E
small changes in an
input result
the output
small changes in
r
hidthe 1
I
414
ha E
W
e's
Theorem
CMT
If Zn
then
Ig
e c as
h Zn
P II Zn ell E
E
n
P
soo
hic
I
and he is
as
n soo
as n soo
I
P Ilh Zn
h
C 11 EE
I
as
nooo
continuous
at
C
in
Proof Let
Eso
hf is continuous at C
that halls E
78 It 41 8
Since
Evaluated
at
Zn
x
we
Pliant hulled
Zak
Since
This implies
Theorem Let
then
C
nan
find
PtyZn ell
Pll Zn ell
P
se
I
se
hic
I
denote the set of distributions for which
ETH COO Then for any sample size n and any E so
I
For
P PTIta ETH
e
L
there is a probability distribution such that
the probability that the sample mean differs from the
any
n
population expectation by E is
Failure
one
of Uniform convergence while
ETA it does not converge
L is
for which ETA
For any sample
the sample mean
In
converges
to
uniformly across distributions
size there is
a
distribution such that
is not close to the population expectation
Theorem
I
Let
EB
Var X
N
s
distributions which satisfy
of
o
as
PIItn
EEN
O
e
A
sufficient condition is to found a moment
assume vark EB for some Boo
greater than one
this restriction the WUN
across the
set of
Define
00
for all
E o
Xi kilo
as
n
distributions
and
of
surely to
c
random variables
as
Pfling Za
It
of
4 In
computes
a
the
c
sic
O
ZuelR
Zn
converges
c
almos
in
L
probability of a limit rather than
probability
a
which
tot In Eiti Then
e
denoted
noo
for
oo
FI PIIFT EEP
sequence
holds uniformly
distributions
Let It denote the set of
82
Def A
Then for all E
00
00
Under
Theorem
B
for some
SEE
4
set
denote the
implies
Zu
P
C
the limit
Theorem
SUN
If
Xi
ETA
are
then
COO
and identically distributed and
independent
noo
as
In t.E.fi
a
s
Efx
Chapter 8 Central limit Theory
Def let Zn be
a
vectors
of random variables or
We
that
PEZ
sequence
with distribution Gn u
need
say
Zn converges in distribution to Z as n soo denoted
d
Z
Zn
if for all u at which Glu PEZ u
is continuous
Gn
A sequence of
the sequence
a
write the
equivalent
WUN
From the
Since
convergence in
convergence
To
In
to
obtain
Gulu
we
s
know
converges to
is degenerate
1
d c which is
Zn
as
PEZ 1
c
that
I
s
M
probability to a constant is the same
this
in
Glu
convergence
Zn
in distribution
a
converges in distribution
of distribution functions
the limit distribution G
we can
n soo
as
Zn
random variable
distribution auction
limit
When
Glu
u
means
non degenerate distribution
as
In dm
we need
to rescale
rn F
M
Itn
q
Zn
ETZn
Moments
of
m
which means
Vartzn
O
Zn
as
O
Etzi
82
ELZA
Kafir
E IZ
Yn 304
E
Ksdk
Efta
As the
of the
Kofi
82
O
10km
384
h
O
115Ky kit
lousy
1582
155
sample increases these moments converge
to
those
normal distribution
If X has
finite rth moment then EEZ
where Zn No 04
a
to EEZ
d
02
n soo
Elza
Zn
Ern Fyi
Theorem
Zn
Theorem
For random variable with
Z if Eteeplezn
Mn
t
Eterpltzn
Etelp EZ
a
finite
exp
MGF
E
converges
for every
te IR
If ti
Theorem
CLT
d
Tn ta gu
where
The simple
M ETA
ETA
and
i id
are
and
Loo then as
nooo
N 0,04
82
ETH M
of averaging induces normality
The distribution of the random variable rn ti p
approximately
process
the same as NCO 02 when
is
u is large
Tna Nlm
For
ZuelR
multivariate
Theorem
Zn
d
Z
ipp t Zn
d
t Z for
every
HEIR with
L
Let Xi
a vectorvalued
observations
Varix
EFX M
let Zn rn In M
Theorem
If
Xi ER
are
and
In
sample
averages
n z
i id
rn In pi
and
d
ETH
NCO E
Coo
then
as
nooo
Theorem
CMT
If
Zn
d
and h km
Z as noo
IR
has the set
of discontinuity points Dh such that Pl Ze Dn
then h Zn d h z as nooo
If
his
Convergence
d
then h zu
continuous
h Z
in distribution is preserved by continuous
transformations
Theorem
Slutsky
If
3
Delta
Method
d
Z and Cn
d
I Zn t Cn
2
Theorem
Zn
Zn Cn
d
d
In
If F E
oo
ther
Ze
d
differentiable in
n
n
as
Zte
E
o
se
Cto
if
5 and heal
is continuously
neighborhood of
a
other
oo
Tn
Where
H
u
ha
ha
Juhu
H
and H H
In particular if bunco
rn ha
d
ha
V
d
o
then as
noo
NCO H'VA
as
0