A General Approach to
Sensitivity Analysis
Darryl Fenwick, Streamsim Technologies
Céline Scheidt, Stanford University
Role of sensitivity analysis
• Model calibration
•
(such as in history matching)
• Model identification
•
(which models best represent a physical phenomenon)
• Model reduction
•
(which parameters can be removed from the model)
• Model quality
•
May 9, 2012
(is a model valid)
SCRF 25th Annual Meeting
2
SA in reservoir modeling
• Sensitivity analysis in reservoir modeling has
been dominated by the use of experimental
design (ED) and response surface methods
(RSM)
• Why ED and RSM?
• Efficiency and strong mathematical basis
• Implemented in commercial software
May 9, 2012
SCRF 25th Annual Meeting
3
Sensitivity analysis & ED
Typically applied to:
• Continuous, “deterministic” parameters
• Single response (FOPT @ 10 years)
Regression
Sensitivity of parameters on CumOil
4
3000
2800
FOPR
3
2600
2400
2
2200
2000
1
1
0.5
1
0.5
0
May 9, 2012
PORO
0
0
-0.5
-0.5
-1
-1
PERMX
SCRF 25th Annual Meeting
0.2
0.4
0.6
0.8
1
1.2
1.4
4
ED & RSM: limitations
Limitations:
• ED ignores prior PDF
• Single response
• Smooth response
• Discrete parameters
• Fault interpretations
• Facies proportion cubes
• Stochastic “noise” in response
ED & RSM is one technique within
global SA approaches
May 9, 2012
SCRF 25th Annual Meeting
2800
FOPR
• Spatial uncertainty
3000
2600
2400
2200
2000
1
0.5
1
0.5
0
0
-0.5
PORO
-0.5
-1
-1
PERMX
5
A general SA approach
Spear and Hornberger – study of growth of nuisance alga
• Divided output of model into two classes, behavior B, and behavior B’
• Analyzed how model parameters influenced the classification
May 9, 2012
SCRF 25th Annual Meeting
6
Proposed generalized SA
1. Define model and the set of uncertain input parameters.
2. Assign prior PDFs to the input parameters.
3. Generate an ensemble of models through sampling of prior
PDFs.
4. Evaluate the models, creating output for the responses of
interest.
5. Classify the model ensemble based upon the responses of
interest.
6. Analyze the sample distributions of the input parameters
within each class.
7. Asses the influence of each input parameter based upon
distributions
May 9, 2012
SCRF 25th Annual Meeting
7
Example: f(x,y,z) = x + y
• 50 samples created of x,y,z ϵ U[0,1]
• Classify 50 models into 3 clusters
using value of response
• Compare the distributions of x, y, z in three
clusters with initial sample
May 9, 2012
SCRF 25th Annual Meeting
8
Example: f(x,y,z) = x + y
• Cumulative distribution functions
• Initial 50 models (black)
• 3 clusters (red, blue, green)
1
0.8
cdf
0.6
0.4
0.2
0
0
x
May 9, 2012
y
SCRF 25th Annual Meeting
0.2
0.4
0.6
0.8
1
z
z
9
Basic idea
• Influential parameter will distinguish the
models into the separate classes (clusters)
• Evident when comparing the distributions
• Non-influential parameters will have no
impact on the classification
• The distributions will be similar between classes
May 9, 2012
SCRF 25th Annual Meeting
10
Advantages
•
•
•
•
Account for any type of parameter distribution
Classification is not limited to a single response
Responses can be stochastic in nature
Model responses are used only for classification
• Proxy models can be employed
• Accuracy of the response itself is inconsequential.
• What is important is that the responses correctly
classify the models
May 9, 2012
SCRF 25th Annual Meeting
11
Classification in metric space
 Classify models by clustering in metric space
• Automatic clustering using iterative k-medoids
Metric Space
Next step: Analysis
Clustering
algorithm
May 9, 2012
Next step: Analysis
SCRF 25th Annual Meeting
12
Techniques for analysis
• Kolmogorov-Smirnov test – for continuous
distributions
Dn = maximum vertical distance
May 9, 2012
SCRF 25th Annual Meeting
13
Techniques for analysis
• Cramer-von Mises test (similar to K-S test)
• Chi-Squared Test – for categorical or binned
data
May 9, 2012
SCRF 25th Annual Meeting
14
General method – L1-Norm
• Statistical tests require a large number of samples
• We can also calculate a “distance” between sample
distribution F(x) and distribution in cluster F(x|C)
May 9, 2012
SCRF 25th Annual Meeting
15
Normalization of L1-norm
• L1 distance normalized using a resampling
procedure to estimate the statistical
significance of distance
• Attempts to resolve problem of small sample sizes
1
Cluster1
Cluster2
Cluster3
z
0.8
cdf
0.6
y
0.4
0.2
x
0
0
May 9, 2012
0.2
0.4
0.6
x
0.8
1
0
SCRF 25th Annual Meeting
0.5
1
1.5
2
2.5
16
Parameter interactions
• Response surfaces model them using “interaction”
terms
• In the general SA approach, we are interested in
how the distribution of x is influenced by y in C
 F(x|y,C)
• (More of a “dependency” instead of an interaction)
May 9, 2012
SCRF 25th Annual Meeting
17
Parameter interactions
• Measures of parameter dependency
• Correlation coefficient (ρxy)
• L1-norm:
Cluster3
1
0.8
0.6
cdf
• Bin y values (min, med, max)
• Construct F(x|y,C)
Y
• Compare to F(x|C)
F(x|y)
0.4
F(x|y,C)
0.2
X
0
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
x|y
May 9, 2012
SCRF 25th Annual Meeting
18
Application - WCA field
•
•
•
•
Offshore turbidite
20 producers
8 injectors
78 x 59 x 116
~ 100,000 active grid blocks
• 3-1/2 years production
• Uncertainty
•
Depositional scenario
Scheidt, C. and J.K. Caers, “Uncertainty Quantification in Reservoir Performance Using
Distances and Kernel Methods – Application to a West-Africa Deepwater Turbidite
May 9, 2012
SCRF 25th Annual Meeting
Reservoir”, SPEJ 2009
19
One realization – TI1
Upper Section
May 9, 2012
Lower Section
SCRF 25th Annual Meeting
20
Depositional scenarios
TI 1
May 9, 2012
TI 3
SCRF 25th Annual Meeting
21
Depositional scenarios
TI 8
May 9, 2012
TI 9
SCRF 25th Annual Meeting
22
Depositional scenarios
TI 10
May 9, 2012
TI 13
SCRF 25th Annual Meeting
23
Applications
• First application – 9 continuous parameters in the flow
simulation, specifically:
•
•
•
•
•
Swc for the levee and channel sands (2 parameters)
Sorw for the levee and channel sands (2)
Maximum water and oil rel perm values (2)
Water and oil Corey exponents (2)
Kv/Kh ratio (1)
• Single response: final cum. oil production
• Goal: compare general SA with traditional methods
• 60 runs created using Latin hypercube sampling
May 9, 2012
SCRF 25th Annual Meeting
24
Traditional SA methods
RSM
Correlation Coefficient
Sensitivity of parameters - Pearson
Sensitivity of parameters on CumOil
watExp
krwMax
oilExp
watExp
krwMax
SWCR sand
kroMax
kroMax
SWCR sand
SWCR levee
SWCR levee
oilExp
SOWCR sand
SOWCR levee
SOWCR levee
KvKh
KvKh
SOWCR sand
0
May 9, 2012
10
20
30
40
SCRF 25th Annual Meeting
0
0.2
0.4
0.6
0.8
25
Kolmogorov-Smirnov test
May 9, 2012
Cluster 1
Cluster 2
Cluster 3
KvKh
0.88 (0)
0.85 (0)
0.99 (0)
SOWCR levee
0.95 (0)
0.99 (0)
0.97 (0)
SOWCR sand
0.93 (0)
0.98 (0)
0.89 (0)
SWCR levee
0.38 (0)
0.57 (0)
0.58 (0)
SWCR sand
0.82 (0)
0.35 (0)
0.14 (0)
kroMax
0.97 (0)
0.69 (0)
0.90 (0)
krwMax
0.34 (0)
0.01 (1)
0.001 (1)
oilExp
0.97 (0)
0.90 (0)
0.99 (0)
watExp
0.98 (0)
0.63 (0)
0.08 (0)
SCRF 25th Annual Meeting
26
Cramer – von Mises test
May 9, 2012
Cluster 1
Cluster 2
Cluster 3
KvKh
0.42 (0)
0.12 (0)
0.05 (0)
SOWCR levee
0.02 (0)
0.007 (0)
0.03 (0)
SOWCR sand
0.12(0)
0.05 (0)
0.16 (0)
SWCR levee
0.61(0)
0.37 (0)
0.58 (0)
SWCR sand
0.36 (0)
0.84 (0)
0.90 (0)
kroMax
0.12 (0)
0.64 (0)
0.25(0)
krwMax
0.71 (0)
0.99 (1)
1 (1)
oilExp
0.05(0)
0.12 (0)
0.02 (0)
watExp
0.21 (0)
0.71 (0)
0.98 (1)
SCRF 25th Annual Meeting
27
General SA
Cluster1
Cluster2
Cluster3
watExp
oilExp
krwMax
kroMax
SWCR sand
SWCR levee
SOWCR sand
SOWCR levee
KvKh
0
0.5
1
1.5
2
2.5
Normalized L1-Norm Distance
May 9, 2012
SCRF 25th Annual Meeting
28
Summary of comparison
•
Results of tornado plot, RSM, correlation coefficient, and general
SA all in general agreement
•
Influential parameters
•
•
•
RSM: watExp, oilExp, krwMax, kroMax, SWCR_sand
CC: krwMax, watExp, SWCR_sand
General SA: krwMax, watExp, SWCR_sand
•
K-S test and C-vM test concur with general SA, but are more
conservative
•
RSM indicates that Corey oil exponent (oilExp) is very influential
parameter
•
May 9, 2012
Correlation coefficient and general SA does not
SCRF 25th Annual Meeting
29
Parameter interactions
• Comparison using RSM, correlation
coefficient, and normalized L1-norm distance
May 9, 2012
SCRF 25th Annual Meeting
30
Correlation coefficient
Normalized
L1-norm
Contribution of each cluster (L1)
krwMax|watExp
krwMax|watExp
watExp|krwMax
SWCR sand|krwMax
watExp|kroMax
KvKh|SWCR sand
SWCR sand|KvKh
SWCR sand|krwMax
kroMax|watExp
oilExp|SWCR levee
SOWCR levee|watExp
oilExp|SOWCR levee
SWCR levee|SOWCR levee
SOWCR sand|watExp
kroMax|KvKh
0
0.5
0
1
0.5
1
1.5
2
krwMax|watExp
krwMax|oilExp
KvKh|watExp
oilExp|watExp
SOWCR sand|oilExp
SWCR sand|krwMax
KvKh|krwMax
SWCR levee|krwMax
KvKh|oilExp
SOWCR sand|kroMax
SOWCR levee|SOWCR sand
SWCR sand|kroMax
May 9, 2012
SCRF 25th Annual Meeting
0
2
4
6
Experimental
design
8
10
31
A more sophisticated application
• Same 9 continuous parameters as before +
training image
• TI’s represent uncertainty of depositional
scenario
• Distance - S of difference over all TS of:
• Oil production rate for 20 producers
• Water injection for 8 injectors.
• 60 runs created using Latin hypercube
sampling
May 9, 2012
SCRF 25th Annual Meeting
32
A more sophisticated application
Challenges for traditional SA methods
1. Discrete parameter  TI
• For 6 training images, would require building 6
response surfaces
2. Multiple responses (oil & water rates)
3. Stochastic response
• Seed for geostatistical algorithm changes for
each run
May 9, 2012
SCRF 25th Annual Meeting
33
Results – focus on TI
Chi-Squared test
1
0.8
cdf
0.6
0.4
0.2
0
1
2
3
4
5
TI
CDF for TI in 3 clusters
• Black – initial sample
• Red = Cluster 3
• Green = Cluster 2
• Blue = Cluster 1
May 9, 2012
6
h
chi2stat
Cluster 1
0
4.25
Cluster 2
1
11.9
Cluster 3
1
24.3
Cluster 2 and Cluster 3 show
that TI is influential
SCRF 25th Annual Meeting
34
Normalized L1-norm distance
Cluster1
Cluster2
Cluster3
watExp
oilExp
krwMax
kroMax
TI
SWCR sand
SWCR levee
SOWCR sand
SOWCR levee
KvKh
0
0.5
1
1.5
krwMax, SOWCR_sand, and TI are influential
 Strong variation due to spatial uncertainty
May 9, 2012
SCRF 25th Annual Meeting
35
Conclusion of SA for WCA
• Training image has (slight) influence on
classification of models
• Parameter dependencies difficult to analyze
•
Parameters chosen such that many dependencies are
possible
• Analysis of results is very difficult for complex
applications
•
•
May 9, 2012
Different SA techniques can give different results
Requires interpretation by modeling team
SCRF 25th Annual Meeting
36
Conclusion
• General SA approach has been developed
•
•
•
Idea: Use model classification and parameter
distributions as basis for SA
Addresses some limitations in traditional approach to SA
in reservoir modeling
RSM & ED are still very powerful  general SA is a
complimentary approach
• New approach – ideas/concepts are new and
still developing
May 9, 2012
SCRF 25th Annual Meeting
37