Inverse Transform,
Acceptance-Rejection, and
Transformation Methods
CS 5960/6960: Nonparametric Methods
Tom Fletcher
February 9, 2009
Review
I
What is the quantile function for a real-valued
random variable X ?
I
What is the empirical quantile function for data
{xi }?
I
What is a Q-Q plot?
Bernoulli Distribution
A Bernoulli trial is an experiment with two possible
outcomes, one with probability p, the other with
probability 1 − p.
An rv X representing a Bernoulli trial has two possible
values, X = 0 or X = 1. The pmf is given by
P(X = 0) = 1 − p,
P(X = 1) = p.
Bernoulli Distribution
E[X] = 0 · (1 − p) + 1 · p = p
E[X 2 ] = 02 · (1 − p) + 12 · p = p
Var(X) = E[X 2 ] − E[X]2 = p − p2 = p(1 − p)
Binomial Distribution
Consider n independent Bernoulli trials, X1 , . . . , Xn .
Then the total number of successes is given by the rv
P
X = ni=1 Xi .
X has a binomial distribution, which has pmf
n k
P(X = k) =
p (1 − p)n−k
k
Binomial Distribution
Because X is the sum of iid Bernoulli rv’s, we have:
"
E[X] = E
n
X
#
Xi =
i=1
Var(X) = Var
n
X
i=1
E[X1 ] = np.
i=1
!
Xi
n
X
=
n
X
i=1
Var(X1 ) = np(1 − p).
Geometric Distribution
Consider a sequence of independent Bernoulli trials with
success probability p. Let the rv X represent the number
of trials until the first success. Then X has a geometric
distribution, X ∼ Geom(p), with pmf
P(X = k) = p(1 − p)k−1 .
Geometric Distribution
E[X] =
∞
X
kp(1 − p)k−1 =
k=1
1
p
Var(X) = E[X 2 ] − E[X]2
∞
X
=
k2 p(1 − p)k−1 − (1/p)2
k=1
=
1−p
p2
Inverse Transform Method
Let X be an rv with cdf FX and let U ∼ Unif(0, 1).
Consider the rv Y = FX−1 (U):
FY (y) = P(Y ≤ y)
= P(FX−1 (U) ≤ y)
= P(U ≤ FX (y))
= FU (FX (y))
= FX (y)
So, Y has the same distribution as X .
Inverse Transform Method
This gives the following procedure for generating a
random number from the distribution FX :
1. Find a formula for the quantile function FX−1 .
2. Generate a uniform random number u.
3. Return the random number x = FX−1 (u).
Example: Exponential Distribution
X ∼ Exp(λ)
fX (x) = λe−λx
FX (x) = 1 − e−λx
FX−1 (p) = −
log(1 − p)
λ
Discrete Case
If X is a discrete rv whose pmf is non-zero at points
· · · < xi−1 < xi < xi+1 < · · · , then the quantile
function is given by
FX−1 (p) = xi ,
where FX (xi−1 ) < p ≤ FX (xi )
Acceptance-Rejection Method
Consider rv’s X and Y with densities fX and fY , such that
there exists a constant c > 0 with
fX (t)
≤c
fY (t)
for all t with fX (t) > 0.
The rv Y should be “easy” to generate and c should be
made as small as possible.
Acceptance-Rejection Method
Given this setup, we can generate random numbers from
the density fX as follows:
1. Generate a random number y from the density fY .
2. Generate a random number u from Unif(0, 1).
3. If u < fX (y)/(cfY (y)), return y. Otherwise, go back
to step 1.
What’s the Probability of Acceptance?
fX (Y)
fX (Y)
P(accept|Y) = P U <
=
cfY (Y)
cfY (Y)
Z
P(accept) =
P(accept|y) fY (y)dy
ZR
fX (y)
fY (y)dy
RZcfY (y)
1
fX (y)dy
=
c R
1
=
c
=
Expected Number of Iterations Needed
Since the probability of acceptance (success) is 1/c, the
number of iterations needed will follow a geometric
distribution with p = 1/c. So, on average it will take c
iterations to generate a number.
Proof Acceptance-Rejection Works
We want to show that we really get random numbers
from the density fX .
Using Bayes’ Rule:
P(accept|Y = y) fY (y)
P(accept)
fX (y)/(cfY (y)) fY (y)
=
1/c
= fX (y)
fY|accept (y|accept) =
Transformation Method
Let X be a real-valued rv and Y = g(X), where g is a
monotonic increasing function. Then as we showed
earlier, Y has pdf
fY (y) = fX
−1 dg
(y) ,
g−1 (y) dy
and cdf
FY (y) = FX g−1 (y) .
Example: Chi-square Distribution