Lecture 1. Probability and Distributions
- This is based on Ch.1 of the textbook of this course: “Introduction to
Mathematical Statistics,” written by Robert V. Hogg, Joseph W. Mckean, and
Allen T. Craig. Don’t circulate it.
Lecture 1. Probability and Distributions
1 / 75
Probability and Distributions
Union of Sets
[
Ai = {x : x ∈ Ai for ∃i ∈ I},
i∈I
where I is an index set.
I = {1, 2, 3, ..., n}:
[
Ai =
n
[
Ai = A1 ∪ A2 ∪ · · · ∪ An
i=1
i∈I
I = {1, 2, 3, ...}:
[
i∈I
Ai =
∞
[
Ai = A1 ∪ A2 ∪ · · ·
i=1
Lecture 1. Probability and Distributions
2 / 75
Probability and Distributions
Intersection of Sets
\
Ai = {x : x ∈ Ai for ∀i ∈ I},
i∈I
where I is an index set.
I = {1, 2, 3, ..., n}:
\
Ai =
n
\
Ai = A1 ∩ A2 ∩ · · · ∩ An
i=1
i∈I
I = {1, 2, 3, ...}:
\
i∈I
Ai =
∞
\
Ai = A1 ∩ A2 ∩ · · ·
i=1
Lecture 1. Probability and Distributions
3 / 75
Probability and Distributions
De Morgan’s Law
Let Ai be a subset of the uinversal set U for every i ∈ I, where I is an index
set. Then,
[
\
\
[
( Ai )c =
Aci ,
( Ai )c =
Aci
i∈I
i∈I
i∈I
Lecture 1. Probability and Distributions
i∈I
4 / 75
Probability and Distributions
Definition 1.1
The sample space is the collection of all possible outcomes in a random
experiment.
• The sample space will be usually denoted by Ω.
Example 1.1
Tossing a coin once: Ω = {H, T }
Casting a dice: Ω = {1, 2, 3, 4, 5, 6}
Choosing a point randomly on the interval [0, 1]: Ω = [0, 1]
• A subset C of the sample space Ω is called an “event”.
Lecture 1. Probability and Distributions
5 / 75
Probability and Distributions
Definition 1.2 (Probability)
Let Ω be a sample space and let F be a the set of all subsets of Ω. Let P be a
real valued function defined on F. Then P is a probability (set) function if
1
P(C) ≥ 0 for all C ∈ F,
2
P(Ω) = 1,
3
C1 , C2S, ... ∈ F are P
disjoint (that is, Cm ∩ Cn = ∅ for all m 6= n ∈ {1, 2, ...})
∞
∞
⇒ P( n=1 Cn ) = n=1 P(Cn ).
• “disjoint” ⇔ “mutually exclusive”
Lecture 1. Probability and Distributions
6 / 75
Probability and Distributions
Theorem 1.1
If P is a probability function, then P(∅) = 0.
Proof.
Let
C1
= ∅
C2
= ∅
..
.
Then, C1 , C2 , ... ∈ F are disjoint (that
S∞ is, Cm ∩
PC∞n = ∅ for all
m 6= n ∈P
{1, 2, ...}). Therefore, P( n=1 Cn ) = n=1 P(Cn ), that is,
∞
P(∅) = n=1 P(∅). If P(∅) = c > 0, then it should hold that c = ∞, which is
a contradiction. Hence, P(∅) = 0.
Lecture 1. Probability and Distributions
7 / 75
Probability and Distributions
Theorem 1.2
C1 , C2S, ..., Cn ∈ FPare disjoint (that is, Ci ∩ Cj = ∅ for all i 6= j ∈ {1, 2, ..., n})
n
n
⇒ P( i=1 Ci ) = i=1 P(Ci ).
Proof.
Let
Cn+1
= ∅
Cn+2
= ∅
..
.
Then, C1 , C2 , ..., Cn , Cn+1 , Cn+2 , ... S
∈ F are disjoint
P∞ (that is, Ci ∩ Cj = ∅ for
∞
allSi 6= j ∈ {1, P
2, ...}). Therefore, P( i=1 Ci ) = i=1 P(Ci ), that is,
n
n
P( i=1 Ci ) = i=1 P(Ci ).
Lecture 1. Probability and Distributions
8 / 75
Probability and Distributions
Theorem 1.3
For each event C ∈ F, it holds that P(C) = 1 − P(C c ).
Proof.
Since C, C c ∈ F are disjoint (that is, C ∩ C c = ∅), it holds that
P(C ∪ C c ) = P(C) + P(C c ). However, P(C ∪ C c ) = P(Ω) = 1 so that
1 = P(C) + P(C c ).
Lecture 1. Probability and Distributions
9 / 75
Probability and Distributions
Theorem 1.4
C1 , C2 ∈ F and C1 ⊂ C2 ⇒ P(C1 ) = P(C2 ) − P(C2 \ C1 ) ≤ P(C2 ).
Proof.
Since C1 , C2 \ C1 ∈ F are disjoint (that is, C1 ∩ (C2 \ C1 ) = ∅), it holds that
P(C1 ∪ (C2 \ C1 )) = P(C1 ) + P(C2 \ C1 ). However, P(C1 ∪ (C2 \ C1 )) = P(C2 )
so that P(C2 ) = P(C1 ) + P(C2 \ C1 ). Therefore,
P(C1 ) = P(C2 ) − P(C2 \ C1 ) ≤ P(C2 ), where the inequality holds since
P(C2 \ C1 ) ≥ 0.
Lecture 1. Probability and Distributions
10 / 75
Probability and Distributions
Theorem 1.5
For each C ∈ F, 0 ≤ P(C) ≤ 1.
Proof.
The first inequality holds by the definition of the probability function. Since
C ⊂ Ω, we have P(C) ≤ P(Ω) = 1
Lecture 1. Probability and Distributions
11 / 75
Probability and Distributions
Theorem 1.6
C1 , C2 ∈ F ⇒ P(C1 ∪ C2 ) = P(C1 ) + P(C2 ) − P(C1 ∩ C2 ).
Proof.
Since C1 \ C2 , C1 ∩ C2 ∈ F are disjoint, it holds that
P((C1 \ C2 ) ∪ (C1 ∩ C2 )) = P(C1 \ C2 ) + P(C1 ∩ C2 ). However,
P((C1 \ C2 ) ∪ (C1 ∩ C2 )) = P(C1 ) so that P(C1 ) = P(C1 \ C2 ) + P(C1 ∩ C2 ).
Therefore, P(C1 \ C2 ) = P(C1 ) − P(C1 ∩ C2 ). Similarly, we have
P(C2 \ C1 ) = P(C2 ) − P(C1 ∩ C2 ). Since C1 \ C2 , C1 ∩ C2 , C2 \ C1 ∈ F are
disjoint, it holds that
P((C1 \ C2 ) ∪ (C1 ∩ C2 ) ∪ (C2 \ C1 )) = P(C1 \ C2 ) + P(C1 ∩ C2 ) + P(C2 \ C1 ).
However, P((C1 \ C2 ) ∪ (C1 ∩ C2 ) ∪ (C2 \ C1 )) = P(C1 ∪ C2 ) so that
P(C1 ∪ C2 ) = P(C1 \ C2 ) + P(C1 ∩ C2 ) + P(C2 \ C1 ) =
P(C1 ) + P(C2 ) − P(C1 ∩ C2 ).
Lecture 1. Probability and Distributions
12 / 75
Probability and Distributions
Theorem 1.7
Pn
Sn
P( i=1 Ci ) ≤ i=1 P(Ci ) for all C1 , C2 , ..., Cn ∈ F.
Proof.
Let
B1
= C1
B2
= C2 \ C1
B3
= C3 \ (C1 ∪ C2 )
..
.
= Cn \ (C1 ∪ C2 ∪ · · · ∪ Cn−1 )
Sn
Pn
Then, B1 , B2 , ..., Bn ∈ F are disjoint. Therefore,
Sn P( i=1SBni ) = i=1 P(Bi ).
Note that B
2, ..., n and
i =
i=1 Ci . Therefore,
Sin⊂ Ci for i =S1,
Pn that i=1 BP
n
n
we have P( i=1 Ci ) = P( i=1 Bi ) = i=1 P(Bi ) ≤ i=1 P(Ci ).
Bn
Lecture 1. Probability and Distributions
13 / 75
Probability and Distributions
• A sequence of events {Cn } is said to be an increasing sequence
S∞ if Cn ⊂ Cn+1
for allSn ∈ {1, 2, ...}, in which case we write limn→∞ Cn = n=1 Cn or
∞
Cn ↑ n=1 Cn .
• A sequence of events {Cn } is said to be a decreasing sequence
T∞ if Cn ⊃ Cn+1
for allTn ∈ {1, 2, ...}, in which case we write limn→∞ Cn = n=1 Cn or
∞
Cn ↓ n=1 Cn .
Lecture 1. Probability and Distributions
14 / 75
Probability and Distributions
Theorem 1.8
Let {Cn } be an increasing
sequence of events. Then
S∞
limn→∞ P(Cn ) = P( n=1 Cn ), that is, limn→∞ P(Cn ) = P(limn→∞ Cn ).
Let {Cn } be a decreasing
T∞ sequence of events. Then
limn→∞ P(Cn ) = P( n=1 Cn ), that is, limn→∞ P(Cn ) = P(limn→∞ Cn ).
Proof.
We first consider the case where {Cn } is an increasing sequence of events. Let
B1
= C1
B2
= C2 \ C1
B3
= C3 \ C2
..
.
Then,
3 , ... ∈ F are disjoint. Therefore,
S∞ B1 , B2 , BP
Pn
Sn
∞
P( n=1 Bn ) = n=1 P(Bn ) = limk→∞ k=1 P(Bk ) = limk→∞ P( k=1 Bk ).
Lecture 1. Probability and Distributions
15 / 75
Probability and Distributions
Sn
S∞
Since n=1 Bn = n=1
S∞ Cn and k=1 Bk = Cn , we have
limn→∞ P(Cn ) = P( n=1 Cn ).
Now consider the case where {Cn } is a decreasing sequence of events. In this
case, {Cnc } is an increasing
sequence of events. Therefore, we have
S∞
c
c
limSn→∞ P(Cnc ) = P(
C
T∞P(Cn ) = 1 − P(Cn ) and that
T∞n=1 nc). Note that
∞
c
P( n=1 Cn ) = P(( T
n=1 Cn ) ) = 1 − P( n=1 Cn ). Therefore, we have
∞
limn→∞ P(Cn ) = P( n=1 Cn ). S∞
Lecture 1. Probability and Distributions
16 / 75
Probability and Distributions
Theorem 1.9 (Boole’s Inequality)
Let {Cn } be an arbitrary sequence of events. Then
∞
[
P(
Cn ) ≤
n=1
∞
X
P(Cn ).
n=1
Proof.
Let
B1
= C1
B2
= C2 \ C1
B3
= C3 \ (C1 ∪ C2 )
..
.
Then,
2 , B3 , ... ∈ F are disjoint, Bn ⊂ Cn for n ∈ {1, 2, 3, ...}, and
S∞ B1 , BS
∞
B
=
Cn . Therefore,
n=1
n=1S
S∞ n
P∞we have
P∞
∞
P( n=1 Cn ) = P( n=1 Bn ) = n=1 P(Bn ) ≤ n=1 P(Cn ).
Lecture 1. Probability and Distributions
17 / 75
Probability and Distributions
Definition 1.3
For C1 , C2 ∈ F with P(C1 ) > 0, the conditional probability of C2 given C1 is
denoted by P(C2 |C1 ) and defined by
P(C2 |C1 ) =
P(C1 ∩ C2 )
.
P(C1 )
Lecture 1. Probability and Distributions
18 / 75
Probability and Distributions
Remark 1.1
For any C ∈ F with P(C) > 0, the followings hold.
1
P(C 0 |C) ≥ 0 for all C 0 ∈ F,
2
P(C|C) = 1,
3
C1 , C2S, ... ∈ F and CP
m ∩ Cn = ∅ for all m 6= n ∈ {1, 2, ...}
∞
∞
⇒ P( n=1 Cn |C) = n=1 P(Cn |C).
Proof.
P(
∞
[
Cn |C)
=
n=1
=
=
S∞
S∞
P(C ∩ ( n=1 Cn ))
P( n=1 (C ∩ Cn ))
=
P(C)
P(C)
P∞
∞
X P(C ∩ Cn )
n=1 P(C ∩ Cn )
=
P(C)
P(C)
n=1
∞
X
P(Cn |C)
n=1
Lecture 1. Probability and Distributions
19 / 75
Probability and Distributions
Remark 1.2
For C1 , C2 ∈ F with P(C1 ) > 0, P(C1 ∩ C2 ) = P(C1 )P(C2 |C1 ).
Example 1.2 (Tossing a fair coin twice independently)
In this case, Ω = {HH, HT, T H, T T }. Let the probability function P satisfy
P({HH}) = P({HT }) = P({T H}) = P({T T }) = 14 . Then, for
C1 = {HH, HT } and C2 = {HH}, we have
P(C2 |C1 ) =
P(C1 ∩ C2 )
P({HH})
1
=
= .
P(C1 )
P({HH, HT })
2
Lecture 1. Probability and Distributions
20 / 75
Probability and Distributions
Theorem 1.10 (Bayes’ theorem)
Sk
Let C1 , C2 , ..., Ck be mutually exclusive(disjoint) and exhaustive( i=1 Ci = Ω)
events such that P(Ci ) > 0 for i = 1, 2, ..., k. Then, for any event C with
P(C) > 0, it holds that
P(Cj )P(C|Cj )
for j = 1, ..., k.
P(Cj |C) = Pk
i=1 P(Ci )P(C|Ci )
Proof.
P(Cj |C)
=
=
=
P(Cj ∩ C)
P(Cj )P(C|Cj )
=
P(C)
P(C ∩ Ω)
P(Cj )P(C|Cj )
P(Cj )P(C|Cj )
= Sk
Sk
P(C ∩ ( i=1 Ci ))
P( i=1 (C ∩ Ci ))
P(Cj )P(C|Cj )
P(Cj )P(C|Cj )
= Pk
Pk
i=1 P(C ∩ Ci )
i=1 P(Ci )P(C|Ci )
Lecture 1. Probability and Distributions
21 / 75
Probability and Distributions
Definition 1.4
Let C1 and C2 be two events. We say that C1 and C2 are independent if
P(C1 ∩ C2 ) = P(C1 )P(C2 ).
Definition 1.5
Let C1 , ..., Cn be n events such that n ≥ 2. We say that C1 , ..., Cn are
(mutually)independent if P(Cd1 ∩ · · · ∩ Cdk ) = P(Cd1 ) · · · P(Cdk ) for any k
(2 ≤ k ≤ n) distinct integers d1 , ..., dk ∈ {1, ..., n}.
Definition 1.6
Let C1 , ..., Cn be n events such that n ≥ 2. We say that C1 , ..., Cn are pairwise
independent if P(Ci ∩ Cj ) = P(Ci )P(Cj ) for all i 6= j ∈ {1, ..., n}.
Remark 1.3
If C1 , ..., Cn (n ≥ 2) are (mutually) independent, then they are pairwise
independent. However the converse is not true.
Lecture 1. Probability and Distributions
22 / 75
Probability and Distributions
Example 1.3
Let Ω = {1, 2, 3, 4}. Then, under the probability function P satisfying
P({1}) = P({2}) = P({3}) = P({4}) = 14 , the events C1 = {1, 2}, C2 = {2, 3},
and C3 = {1, 3} are pairwise independent but not (mutually) independent.
∵
P(C1 ∩ C2 )
=
P(C2 ∩ C3 )
=
P(C1 ∩ C3 )
=
1
= P(C1 )P(C2 )
4
1
P({3}) = = P(C2 )P(C3 )
4
1
P({1}) = = P(C1 )P(C3 )
4
P({2}) =
but
P(C1 ∩ C2 ∩ C3 ) = P(∅) = 0 6= P(C1 )P(C2 )P(C3 )
Lecture 1. Probability and Distributions
23 / 75
Probability and Distributions
Remark 1.4
If C1 and C2 are independent events, then
C1 and C2c are independent
C1c and C2c are independent
Proof.
P(C1 )
=
P(C1 ∩ Ω) = P(C1 ∩ (C2 ∪ C2c ))
=
P((C1 ∩ C2 ) ∪ (C1 ∩ C2c )) = P(C1 ∩ C2 ) + P(C1 ∩ C2c )
=
P(C1 )P(C2 ) + P(C1 ∩ C2c )
Therefore, P(C1 ∩ C2c ) = P(C1 )(1 − P(C2 )) = P(C1 )P(C2c ).
Lecture 1. Probability and Distributions
24 / 75
Probability and Distributions
Definition 1.7
A random variable is a real valued function defined on the sample space.
Example 1.4 (Tossing a fair coin twice independently)
Ω = {HH, HT, T H, T T }. Let X denote the number of heads. Then, X is a
random variable:
X(HH)
=
2
X(HT )
=
1
X(T H)
=
1
X(T T )
=
0
Lecture 1. Probability and Distributions
25 / 75
Probability and Distributions
Remark 1.5
Let X be a random variable. We usually use the notation P(X ∈ B) to denote
P({ω ∈ Ω : X(ω) ∈ B}). That is,
P(X ∈ B) := P({ω ∈ Ω : X(ω) ∈ B}) for B ⊂ R
• {ω ∈ Ω : X(ω) ∈ B} = X −1 (B)
Remark 1.6
When B = (−∞, x] or B = (a, b] or B = [x, ∞) etc. with x, a, b ∈ R, we often
use P(X ≤ x) or P(a < X ≤ b) or P(X ≥ x) etc., respectively, instead of
P(X ∈ B).
Lecture 1. Probability and Distributions
26 / 75
Probability and Distributions
Example 1.5 (Tossing a fair coin twice independently)
Ω = {HH, HT, T H, T T }. Let X denote the number of heads. Then,
P(X ≥ 1)
= P(X ∈ [1, ∞)) = P({ω ∈ Ω : X(ω) ∈ [1, ∞)})
3
= P({HH, HT, T H}) =
4
Definition 1.8
Let X be a random variable. Then, P(X ∈ B) for B ⊂ R, is called the
distribution of the random variable X.
Lecture 1. Probability and Distributions
27 / 75
Probability and Distributions
Definition 1.9 (Cumulative Distribution Function)
Let X be a random variable. Then its cumulative distribution function(cdf) is
defined by
FX (x) = P(X ≤ x) := P({ω ∈ Ω : X(ω) ≤ x}), x ∈ R.
• The cdf of a random variable uniquely determines the distribution of it.
Lecture 1. Probability and Distributions
28 / 75
Probability and Distributions
Theorem 1.11
Let X be a random variable with cdf F (x). Then,
(a) For all a, b ∈ R such that a < b, F (a) ≤ F (b) (F is an increasing function)
(b) limx→−∞ F (x) = 0
(c) limx→∞ F (x) = 1
(d) limx↓x0 F (x) = F (x0 ) for every x0 ∈ R (that is, F is right continuous on
R)
Lecture 1. Probability and Distributions
29 / 75
Probability and Distributions
Proof.
(a) F (a) = P(X ≤ a) = P({ω ∈ Ω : X(ω) ≤ a}) ≤ P({ω ∈ Ω : X(ω) ≤ b}) =
P(X ≤ b) = F (b)
(b) Let
C1
=
{ω ∈ Ω : X(ω) ≤ −1}
C2
=
{ω ∈ Ω : X(ω) ≤ −2}
C3
=
..
.
{ω ∈ Ω : X(ω) ≤ −3}
Then, {Cn } is a decreasing sequence of events. Therefore,
T∞
limx→−∞ F (x) = limn→∞ F (−n) = limn→∞ P(Cn ) = P( n=1 Cn ) = P(∅) = 0.
Lecture 1. Probability and Distributions
30 / 75
Probability and Distributions
(c) Let
C1
=
{ω ∈ Ω : X(ω) ≤ 1}
C2
=
{ω ∈ Ω : X(ω) ≤ 2}
C3
=
..
.
{ω ∈ Ω : X(ω) ≤ 3}
Then, {Cn } is an increasing sequence of events. Therefore,
S∞
limx→∞ F (x) = limn→∞ F (n) = limn→∞ P(Cn ) = P( n=1 Cn ) = P(Ω) = 1.
(d) Let
C1
=
{ω ∈ Ω : X(ω) ≤ x0 + 1}
C2
=
{ω ∈ Ω : X(ω) ≤ x0 + 1/2}
C3
=
..
.
{ω ∈ Ω : X(ω) ≤ x0 + 1/3}
Then, {Cn } is a decreasing sequence of events. Therefore, T
∞
limx↓x0 F (x) = limn→∞ F (x0 + 1/n) = limn→∞ P(Cn ) = P( n=1 Cn ) =
P({ω ∈ Ω : X(ω) ≤ x0 }) = F (x0 ). Lecture 1. Probability and Distributions
31 / 75
Probability and Distributions
Theorem 1.12
Let X be a random variable with cdf F (x). Then, for a < b,
P(a < X ≤ b) = F (b) − F (a).
Proof.
F (b)
= P(X ≤ b) := P({ω ∈ Ω : X(ω) ≤ b})
= P({ω ∈ Ω : X(ω) ≤ a}) + P({ω ∈ Ω : a < X(ω) ≤ b})
= P(X ≤ a) + P(a < X ≤ b)
= F (a) + P(a < X ≤ b)
Lecture 1. Probability and Distributions
32 / 75
Probability and Distributions
Theorem 1.13
For any random variable X with cdf F (x),
P(X = x) = F (x) − F (x−) for x ∈ R.
Proof.
For any given x ∈ R, let
C1
= {ω ∈ Ω : X(ω) ≤ x − 1}
C2
= {ω ∈ Ω : X(ω) ≤ x − 1/2}
C3
= {ω ∈ Ω : X(ω) ≤ x − 1/3}
..
.
Lecture 1. Probability and Distributions
33 / 75
Probability and Distributions
Then,
F (x−)
=
=
=
lim F (x −
n→∞
1
)
n
lim P({ω ∈ Ω : X(ω) ≤ x −
n→∞
1
})
n
lim P(Cn )
n→∞
∞
[
= P(
Cn )
n=1
= P({ω ∈ Ω : X(ω) < x})
= P(X < x)
Therefore, we have
P(X = x) = P(X ≤ x) − P(X < x) = F (x) − F (x−)
Lecture 1. Probability and Distributions
34 / 75
Probability and Distributions
• The space or range of a random variable X is the set D = {X(ω) : ω ∈ Ω}.
Lecture 1. Probability and Distributions
35 / 75
Probability and Distributions
Definition 1.10 (Discrete Random Variable)
We say a random variable is a discrete random variable if its space is either
finite or countable.
Lecture 1. Probability and Distributions
36 / 75
Probability and Distributions
Definition 1.11 (Probability Mass Function)
Let X be a discrete random variable with the space D. The probability mass
function(pmf) of X is given by
pX (x) = P(X = x) for x ∈ R.
Remark 1.7
pX (x) = 0 for x ∈
/ D.
Lecture 1. Probability and Distributions
37 / 75
Probability and Distributions
Remark 1.8
(i) 0 ≤ pX (x) ≤ 1
P
(ii)
x∈R pX (x) = 1
• The pmf of a discrete
P random variable uniquely determines the distribution
of it: P(X ∈ B) = x∈B pX (x).
Lecture 1. Probability and Distributions
38 / 75
Probability and Distributions
• Let X be a discrete random variable with space D. The support of X is
defined by S = {x ∈ R : pX (x) > 0}.
Remark 1.9
S ⊂ D and pX (x) = 0 for x ∈
/ S.
Lecture 1. Probability and Distributions
39 / 75
Probability and Distributions
Remark 1.10
Let X be a discrete random variable with support S.
FX (x) is discontinuous at x if and only if x ∈ S.
If (a, b] ∩ S = ∅, then F is constant on [a, b].
Proof.
Note that FX is a right continuous function. Therefore,
FX (x) is discontinuous at x ⇔ FX (x) − FX (x−) > 0 ⇔ pX (x) > 0.
Let x ∈ [a, b] be given. Then, F (a) ≤ F (x) ≤ F (b). However,
F (b) − F (a)
= P(a < X ≤ b) = P(X ∈ (a, b])
X
pX (x) = 0.
=
x∈(a,b]
That is, F (a) = F (b) so that F (a) = F (x) = F (b)
Lecture 1. Probability and Distributions
40 / 75
Probability and Distributions
Example 1.6 (Tossing a fair coin twice independently)
Ω = {HH, HT, T H, T T }. Let X denote the number of heads. Then,
S = {0, 1, 2}
1
1
pX (0) = pX (2) = , pX (1) =
4
2

0 for x < 0,


 1
4 for 0 ≤ x < 1,
FX (x) =
3
for 1 ≤ x < 2,


 4
1 for x ≥ 2.
Lecture 1. Probability and Distributions
41 / 75
Probability and Distributions
• Let Y = g(X) be a transformation of a discrete random variable X. Then,
Y is also a discrete r.v. with the space DY = {y = g(x) : x ∈ DX }, and
pY (y) = P(Y = y) = P[g(X) = y]
= P[X ∈ g −1 ({y})]
X
=
pX (x),
x∈g −1 ({y})
where g −1 ({y}) := {x ∈ R : g(x) = y}.
Lecture 1. Probability and Distributions
42 / 75
Probability and Distributions
Example 1.7 (A sequence of independent flips of a fair coin)
In this case, the sample space is Ω = {w1 w2 ...... : wi = H or T for i = 1, 2, ...}.
Let X denote the number of flips needed to obtain the first head. Then
( x
1
for x ∈ DX = SX = {1, 2, ....},
2
pX (x) =
0 elsewhere.
Let Y := X 2 . Then
(
pY (y) =
X
x∈g −1 ({y})
pX (x) =
√y
√
pX ( y) = 21
for y ∈ DY = SY = {1, 4, 9, ....},
0 elsewhere.
Let Z := (X − 5)2 . Then DZ = SZ = {0, 1, 4, 9, ....} and
5
4 6
3 7
pZ (0) = 12 , pZ (1) = 12 + 12 , pZ (4) = 12 + 12 , pZ (9) =
2 8
9
10
11
1
+ 21 , pZ (16) = 12 + 12 , pZ (25) = 12
, pZ (36) = 12
, ....
2
Lecture 1. Probability and Distributions
43 / 75
Probability and Distributions
Definition 1.12
We say a random variable X is a continuous random variable if its cdf FX (x)
is a continuous function for all x ∈ R.
Theorem 1.14
Let X be a random variable. Then, X is a continuous random variable if and
only if P(X = x) = 0 for all x ∈ R.
Proof.
FX (x) is continuous at x for every x ∈ R ⇔ FX (x) − FX (x−) = 0 for every
x ∈ R ⇔ P(X = x) = 0 for every x ∈ R.
Lecture 1. Probability and Distributions
44 / 75
Probability and Distributions
• Most continuous random variables are absolutely continuous, that is,
Z x
FX (x) =
fX (t)dt for a nonnegative function fX (t).
−∞
The function fX (t) is called the probability density function(pdf) of X
d
dx FX (x) = fX (x) for x ∈ R at which fX is continuous
The support of X is defined by S = {x ∈ R : fX (x) > 0}
Rb
P(a < X ≤ b) = FX (b) − FX (a) = a fX (x)dx
P(a < X ≤ b) = P(a ≤ X ≤ b) = P(a ≤ X < b) = P(a < X < b)
Lecture 1. Probability and Distributions
45 / 75
Probability and Distributions
The pdf fX satisfies
(i) fRX (x) ≥ 0
∞
(ii) −∞ fX (x)dx = 1
• A continuous random variable means an absolutely continuous one, unless
otherwise stated.
• The pdf of a continuous random variable uniquely determines the
distribution of it:
Z
P(X ∈ B) =
fX (x)dx for B ⊂ R.
B
Lecture 1. Probability and Distributions
46 / 75
Probability and Distributions
• Distribution of a transformation of a continuous r.v.
Example 1.8 (cdf technique(1))
Let Y = X 2 where X be a r.v. whose cdf is

 0 x<0
x2 0 ≤ x < 1
FX (x) =

1 1 ≤ x.
Find the pdf of Y .
Solution.
For y < 0: FY (y) = P(Y ≤ y) = P(X 2 ≤ y) = 0.
√
√
For 0 ≤ y < 1: FY (y) = P(Y ≤ y) = P(X 2 ≤ y) = P(− y ≤ X ≤ y) =
√
√
P(X ≤ y) = FX ( y) = y.
√
√
For y ≥ 1: FY (y) = P(Y ≤ y) = P(X 2 ≤ y) = P(− y ≤ X ≤ y) = P(X ≤
√
√
y) = FX ( y) = 1.
Lecture 1. Probability and Distributions
47 / 75
Probability and Distributions
That is,

 0
y
FY (y) =

1
y<0
0≤y<1
1 ≤ y.
Therefore, by differentiation, we have
1
fY (y) =
0
0<y<1
elsewhere.
Lecture 1. Probability and Distributions
48 / 75
Probability and Distributions
Example 1.9 (cdf technique(2))
Let Y = X 2 where X be a r.v. whose pdf is
1
−1<x<1
2
fX (x) =
0 elsewhere.
Find the pdf of Y .
Solution.
For y < 0: FY (y) = P(Y ≤ y) = P(X 2 ≤ y) = 0.
For 0 ≤ y < 1:
R √y
√
√
√
FY (y) = P(Y ≤ y) = P(X 2 ≤ y) = P(− y ≤ X ≤ y) = −√y 12 dx = y.
√
√
For y ≥ 1: FY (y) = P(Y ≤ y) = P(X 2 ≤ y) = P(− y ≤ X ≤ y) =
R √y
R1 1
√ f (x) dx =
dx = 1.
− y X
−1 2
Lecture 1. Probability and Distributions
49 / 75
Probability and Distributions
That is,

 0 y<0
√
y 0≤y<1
FY (y) =

1 1 ≤ y.
Therefore, by differentiation, we have
1
√
0<y<1
2 y
fY (y) =
0 elsewhere.
Lecture 1. Probability and Distributions
50 / 75
Probability and Distributions
Theorem 1.15
Let X be a continuous r.v. with pdf fX (x) and support SX . Let Y = g(X),
where g(x) is a one-to-one differentiable function on the support of X, SX .
Let x = g −1 (y) so that dx/dy = d[g −1 (y)]/dy. Then the pdf of Y is given by
(
fX (g −1 (y)) dx
y ∈ SY
dy
fY (y) =
0 elsewhere,
where SY = {g(x) : x ∈ SX } is the support of Y .
Proof.
Since g is a one-to-one function on SX , it is either strictly increasing or
strictly decreasing on SX .
We first consider the case where g is a strictly increasing function on SX : For
y ∈ SY ,
FY (y) = P(Y ≤ y) = P[g(X) ≤ y] = P[X ≤ g −1 (y)] = FX (g −1 (y))
Therefore, we have
dg −1 (y)
dx
dx
= fX (g −1 (y))
= fX (g −1 (y))
dy1. Probability and Distributions
dy
dy
Lecture
fY (y) = fX (g −1 (y))
51 / 75
Probability and Distributions
Now we consider the case where g is a strictly decreasing function on SX : For
y ∈ SY ,
FY (y) = P(Y ≤ y) = P[g(X) ≤ y] = P[X ≥ g −1 (y)] = 1 − FX (g −1 (y))
Therefore, we have
fY (y) = −fX (g −1 (y))
dx
dg −1 (y)
dx
= −fX (g −1 (y))
= fX (g −1 (y))
dy
dy
dy
Remark 1.11
We refer to dx/dy = d[g −1 (y)]/dy as the Jacobian (denoted by J) of the
transformation.
Lecture 1. Probability and Distributions
52 / 75
Probability and Distributions
Remark 1.12
We can apply the theorem to solve the examples before it.
• Example(cdf technique(1)):
fX (x) =
2x 0 < x < 1
0 elsewhere
g(x) = x2 is a one-to-one function mapping SX = (0, 1) onto SY = (0, 1).
y = x2
⇔
x=
√
y
1
J = √ = |J|
2 y
Therefore, for y ∈ SY ,
fY (y) = fX (g −1 (y))
dx
1
√
√ 1
= fX ( y) √ = 2 y √ = 1
dy
2 y
2 y
Thus, we get the same result.
Lecture 1. Probability and Distributions
53 / 75
Probability and Distributions
• Example(cdf technique(2)):
g(x) = x2 maps SX = (−1, 1) onto SY = (0, 1), but it is not a one-to-one
function on SX = (−1, 1).
y = x2
⇔
√
x=± y
For y ∈ SY ,
√
fY (y) = fX ( y)
1 1
1
1
1
1 1
√
√ + fX (− y) − √ =
√ +
√ = √
2 y
2 y
22 y 22 y
2 y
Thus, we get the same result.
Lecture 1. Probability and Distributions
54 / 75
Probability and Distributions
Definition 1.13 (Expectation)
R∞
If X is a continuous r.v. with pdf f (x) and −∞ | x | f (x)dx < ∞, then
the expectation of X is
Z ∞
E(X) =
xf (x)dx.
−∞
P
If X is a discrete r.v. with pmf p(x) and x | x | p(x) < ∞, then the
expectation of X is
X
E(X) =
xp(x).
x
Remark 1.13
Sometimes the expectation E(X) is called the mathematical expectation of
X, the expected value of X, or the mean (value) of X, and usually
denoted by µ.
Lecture 1. Probability and Distributions
55 / 75
Probability and Distributions
Remark 1.14
The expectation E(X) is finite by the integrability(or absolute convergence)
condition without which the expectation may not exist or may not be finite(
that is, it can be ±∞). Thus, in this course, when we say that the expectation
E(X) of a r.v. X exists, we mean it exists and is finite.
Example 1.10
For a constant k, E(k) = k
Lecture 1. Probability and Distributions
56 / 75
Probability and Distributions
• Indicator function
1A (x) =
1
0
if x ∈ A
otherwise
Example 1.11
Let the r.v. X of the discrete type have the pmf given by the table
x
p(x)
1
2
3
4
4 1 3 2
10 10 10 10
Then E(X) = 23/10. If A = {1, 2}, then E(1A (X)) = 1/2.
Lecture 1. Probability and Distributions
57 / 75
Probability and Distributions
Example 1.12
Let X have the pdf
f (x) =
4x3 0 < x < 1
0 elsewhere.
Then,
Z ∞
E(X)
=
xf (x) dx
−∞
Z 1
4x4 dx
=
0
=
4/5
Lecture 1. Probability and Distributions
58 / 75
Probability and Distributions
Theorem 1.16
Let X be a r.v. and let Y = g(X) for some function g.
R∞
Suppose X is continuous with pdf fX (x). If −∞ | g(x) | fX (x)dx < ∞,
then the expectation of Y exists and is given by
Z ∞
E(Y ) =
g(x)fX (x)dx.
−∞
P
Suppose X is discrete with pmf pX (x). If x | g(x) | pX (x) < ∞, then
the expectation of Y exists and is given by
X
E(Y ) =
g(x)pX (x).
x
Proof.
We omit the proof.
Lecture 1. Probability and Distributions
59 / 75
Probability and Distributions
Theorem 1.17
Let g1 (X) and g2 (X) be functions of a random variable X. Suppose the
expectations of g1 (X) and g2 (X) exist. Then for any constants k1 and k2 , the
expectation of k1 g1 (X) + k2 g2 (X) exists and is given by
E[k1 g1 (X) + k2 g2 (X)] = k1 E[g1 (X)] + k2 E[g2 (X)].
Proof.
We prove the theorem for the continuous random variables case.
Z ∞
| k1 g1 (x) + k2 g2 (x) | fX (x)dx
−∞
Z ∞
(| k1 || g1 (x) | + | k2 || g2 (x) |)fX (x)dx
≤
−∞
Z ∞
Z ∞
= | k1 |
| g1 (x) | fX (x)dx+ | k2 |
| g2 (x) | fX (x)dx
−∞
−∞
< ∞
Lecture 1. Probability and Distributions
60 / 75
Probability and Distributions
Therefore, the expectation of k1 g1 (X) + k2 g2 (X) exists, and furthermore we
have
Z ∞
E[k1 g1 (X) + k2 g2 (X)] =
[k1 g1 (x) + k2 g2 (x)]fX (x)dx
−∞
Z ∞
Z ∞
= k1
g1 (x)fX (x)dx + k2
g2 (x)fX (x)dx
−∞
−∞
= k1 E[g1 (X)] + k2 E[g2 (X)]
Lecture 1. Probability and Distributions
61 / 75
Probability and Distributions
Example 1.13
Let X have the pdf
f (x) =
2(1 − x) 0 < x < 1
0 elsewhere.
Find E(X), E(X 2 ), and E(2X − 3X 2 ).
Solution.
Z ∞
E(X)
=
xf (x) dx
−∞
Z 1
2x(1 − x) dx
=
0
=
1/3
Lecture 1. Probability and Distributions
62 / 75
Probability and Distributions
E(X 2 )
Z ∞
=
x2 f (x) dx
−∞
Z 1
=
2x2 (1 − x) dx
0
=
E(2X − 3X 2 )
1/6
=
2E(X) − 3E(X 2 )
=
1/6
Lecture 1. Probability and Distributions
63 / 75
Probability and Distributions
Example 1.14
Let X have the pmf
x
x = 1, 2, 3
elsewhere.
6
p(x) =
0
Find E(X), E(X 3 ), and E(2X − 3X 3 ).
Solution.
E(X)
=
X
xp(x)
x
=
1×
=
7
3
1
2
3
+2× +3×
6
6
6
Lecture 1. Probability and Distributions
64 / 75
Probability and Distributions
E(X 3 )
=
X
x3 p(x)
x
=
13 ×
=
49
3
E(2X − 3X 3 )
2
3
1
+ 23 × + 33 ×
6
6
6
2E(X) − 3E(X 3 )
133
= −
3
=
Lecture 1. Probability and Distributions
65 / 75
Probability and Distributions
Definition 1.14 (Variance)
Let X be a r.v. with mean µ and such that E[(X − µ)2 ] exists. Then the
variance of X is defined to be E[(X − µ)2 ]. It is usually denoted by σ 2 or by
Var(X).
Remark 1.15
σ2
=
E[(X − µ)2 ]
=
E[X 2 − 2µX + µ2 ]
=
E[X 2 ] − 2µE[X] + µ2
=
E(X 2 ) − µ2
• We call σ (the positive square root of the variance) the standard
deviation of X.
Lecture 1. Probability and Distributions
66 / 75
Probability and Distributions
Remark 1.16
Let X be a r.v. and k be a constant. Then, P(X = k) = 1 if and only if µ = k
and σ 2 = 0.
Lecture 1. Probability and Distributions
67 / 75
Probability and Distributions
Example 1.15
Let X have the pdf
1
−1<x<1
elsewhere.
2 (x + 1)
f (x) =
0
Find µ and σ 2 .
Solution.
µ =
=
E(X)
Z ∞
xf (x) dx
−∞
Z 1
=
=
1
−1 2
x(x + 1)) dx
1/3
Lecture 1. Probability and Distributions
68 / 75
Probability and Distributions
E(X 2 )
Z ∞
=
x2 f (x) dx
−∞
Z 1
=
=
σ2
1 2
x (x + 1)) dx
−1 2
1/3
= E(X 2 ) − µ2
=
2/9
Lecture 1. Probability and Distributions
69 / 75
Probability and Distributions
Definition 1.15 (Moment Generating Function)
Let X be a r.v. such that for some h > 0, E[etX ] exists for −h < t < h. The
moment generation function (mgf ) of X is defined to be the function
M (t) = E[etX ] for −h < t < h.
Lecture 1. Probability and Distributions
70 / 75
Probability and Distributions
Theorem 1.18
Let X and Y be random variables with moment generating function MX and
MY , respectively, existing in an open intervals about 0. Then, X and Y have
the same distribution if and only if MX (t) = MY (t) for all t ∈ (−h, h) for
some h > 0.
Lecture 1. Probability and Distributions
71 / 75
Probability and Distributions
Theorem 1.19
Let X be a r.v. and let m be a positive integer. Suppose E[X m ] exists. If k is
a positive integer and k ≤ m, then E[X k ] exists.
Proof.
We prove it in the case where X is a continuous r.v.
Z ∞
Z ∞
|xk |f (x)dx =
|x|k f (x)dx
−∞
−∞
Z 1
Z −1
k
|x| f (x)dx +
=
−1
Z 1
≤
|x| f (x)dx +
−∞
Z −1
f (x)dx +
−1
Z ∞
≤
|x|m f (x)dx +
−∞
Z ∞
f (x)dx +
−∞
<
Z ∞
k
|x|k f (x)dx
1
Z ∞
|x|m f (x)dx
1
|x|m f (x)dx
−∞
∞
Lecture 1. Probability and Distributions
72 / 75
Probability and Distributions
Theorem 1.20 (Markov’s Inequality)
Let u(X) be a nonnegative function of the random variable X. If E[u(X)]
exists, then for every positive constant c,
P[u(X) ≥ c] ≤
E[u(X)]
c
Proof.
Let Y = u(X). Since Y is nonnegative, we have 1 = 1[0,c) (Y ) + 1[c,∞) (Y ).
Note that Y 1[c,∞) (Y ) ≥ c1[c,∞) (Y ).
E[u(X)]
= E[Y ] = E[Y {1[0,c) (Y ) + 1[c,∞) (Y )}]
=
E[Y 1[0,c) (Y )] + E[Y 1[c,∞) (Y )]
≥
E[Y 1[c,∞) (Y )] ≥ E[c1[c,∞) (Y )]
=
cE[1[c,∞) (Y )] = cP[Y ∈ [c, ∞)]
=
cP[Y ≥ c] = cP[u(X) ≥ c]
Lecture 1. Probability and Distributions
73 / 75
Probability and Distributions
Theorem 1.21 (Chebyshev’s Inequality)
Let X be a random variable with a finite positive variance σ 2 and mean µ.
Then for every k > 0,
1
P[| X − µ |≥ kσ] ≤ 2 ,
k
or, equivalently,
1
P[| X − µ |< kσ] ≥ 1 − 2 ,
k
Proof.
P[| X − µ |≥ kσ]
=
≤
P[(X − µ)2 ≥ k 2 σ 2 ]
E[(X − µ)2 ]
σ2
1
=
= 2,
2
2
2
2
k σ
k σ
k
where the inequality comes from Markov’s inequality with u(X) = (X − µ)2
and c = k 2 σ 2 .
Lecture 1. Probability and Distributions
74 / 75
Probability and Distributions
Theorem 1.22 (Jensen’s Inequality)
Let φ be convex on an open interval I and let X be a random variable whose
support is contained in I. Suppose the expectations of both X and φ(X)
exist. Then,
φ[E(X)] ≤ E[φ(X)]
If φ is strictly convex, then the inequality is strict unless X is a constant
random variable.
Lecture 1. Probability and Distributions
75 / 75