Analysis of Monte Carlo
Integration
Fall 2013
By Yaohang Li, Ph.D.
• Last Class
Review
– Numerical Integration
– Monte Carlo Integration
• Crude Monte Carlo
• Hit-or-Miss Monte Carlo
• Comparison
– General Principle of Monte Carlo
• This Class
– Analysis of Monte Carlo Integration
– Variance Reduction Methods
– Assignment #2
• Next Class
– Random Numbers
Curse of Dimensionality
• Curse of Dimensionality
– If one needs n points to achieve certain accuracy for an 1-D
integral, to achieve the same accuracy one needs ns points for
s-dimensional integral
• Errors in high-dimensional functions
–
–
–
–
Rectangle Rule
Trapezoidal Rule
Simpson’s Rule
Convergence Rate
• O(n-/s)
High-Dimensional Monte Carlo
Integration
• Consider the following integral
1
1
0
0
 
   ... f ( x )dx
• Definition of expectation of a function on random
variable  that is uniformly distributed

1
1
0
0
 
E ( f ( ))   ... f ( x )dx  
• Standard error
– Independent of Dimension
– /n1/2
Convergence Rate
• Monte Carlo Integration
– Convergence Rate
– O(cN-1/2)
• Analysis
– Very slow convergence
– To get 1 digit of accuracy usually requires 100 times more
computations
Confidence Interval
• Estimator for the standard error
1 n
s 
( fi  f )2

n  1 i 1
2
• Confidence Interval
– 66%
[ f  s, f  s ]
– 95%
[ f  2 s, f  2 s ]
– 99%
[ f  3s, f  3s]
Variance Reduction Methods
• Variance Reduction Techniques
– Employs an alternative estimator
• Unbiased
• More deterministic
• Yields a smaller variance
• Methods
–
–
–
–
Stratified Sampling
Importance Sampling
Control Variates
Antithetic Variates
Stratified Sampling
• Idea
– Break the range of integration into several pieces
– Apply crude Monte Carlo sampling to each piece separately
• Analysis of Stratified Sampling
– Estimator
– Variance
• Conclusion
– If the stratification is well carried out, the variance of stratified
sampling will be smaller than crude Monte Carlo
Importance Sampling
• Idea
– Concentrate the distribution of the sample points in the parts of
the interval that are of most importance instead of spreading
them out evenly
• Importance Sampling
1
1
1
f ( x)
f ( x)
   f ( x)dx  
g ( x)dx  
dG( x)
g ( x)
g ( x)
0
0
0
– where g and G satisfying
x
G( x)   g ( y )dy
0
1
G(1)   g ( y )dy  1
0
– G(x) is a distribution function
Importance Sampling
• Variance
1
 2 f / g   ( f ( x) / g ( x)   ) 2 dG( x)
0
• How to select a good sampling function?
– How about g=cf?
– g must be simple enough for us to know its integral
theoretically.
Control Variates
• Control Variates
1
1
1
1
    ( x)dx   [ f ( x)   ( x)]dx t   f ( i ), t '    (i )
n i
n i
0
0
– (x) is the control variate with known integral
– Estimator
• t-t’+’ is the unbiased estimator
• ’ is the first integral
– Variance
• var(t-t’+’)=var(t)+var(t’)-2cov(t,t’)
• if 2cov(t,t’)<var(t’), then the variance is smaller than crude
Monte Carlo
– t and t’ should have strong positive correlation
Antithetic Variates
• Main idea
– Select a second estimate that has a strong negative correlation
with the original estimator
– t’’ has the same expectation of t
• Estimator
– [t+t’’]/2 is an unbiased estimator of 
– var([t+t’’]/2)=var(t)/4+var(t’’)/4+cov(t,t’’)/2
• Commonly used antithetic variate
– (t+t’’)/2=f()/2+ f(1-)/2
– If f is a monotone function, f() and f(1-) are negatively
correlated
Summary
• Analysis of Monte Carlo Integration
– Curse of Dimensionality
– Error Analysis of Monte Carlo Integration
• Variance Reduction Methods
–
–
–
–
Stratified Sampling
Importance Sampling
Control Variates
Antithetic Variates
What I want you to do?
• Review Slides
• Work on your Assignment 1 if you haven’t finished
• Work on your Assignment 2