Data-driven Kriging models
based on FANOVA decomposition
O. Roustant, Ecole des Mines de St-Etienne,
www.emse.fr/~roustant
joint work with
T. Muehlenstädt1, L. Carraro2 and S. Kuhnt1
1
University of Dortmund - 2 Telecom St-Etienne
15th February2011
1
1
Cliques of FANOVA graph
and block additive decomposition
f(x) = cos(x1+x2+x3)
+sin(x4+x5+x6)
+tan(x3+x4)
f(x) = f1,2,3(x1,x2,x3)
+f4,5,6(x4,x5,x6)
+f3,4(x3,x4)
Z(x)
= Z1,2,3(x1,x2,x3)
Cliques:
+ Z4,5,6
(x4,x5,x{3,4}
{1,2,3},
{4,5,6},
6)
+ Z3,4(x3,x4)
k(h) = k1,2,3(h1,h2,h3)
+ k4,5,6(h4,h5,h6)
2
+ k3,4(h3, h4)
This talk presents – with many pictures - the
main ideas of the corresponding paper: we refer
to it for details.
3
Introduction
Computer experiments
• A keyword associated to the analysis of timeconsuming computer codes
x1
x2
f
xd
5
y
Metamodeling and Kriging
• Metamodeling: construct a cheap-to-evaluate
model of the simulator (itself modeling the reality)
• Kriging: basically an interpolation method based on
Gaussian processes
6
Kriging model (definition)
Y(x) = b0 + b1g1(x) + … + bkgk(x) + Z(x)
linear trend (deterministic)
+ centered stationary Gaussian process (stochastic)
Some conditional
simulations
7
Kriging model (prediction)
Conditional mean
and 95% conf. int.
Some conditional
simulations
8
Kriging model (kernel)
• Kriging model is a kernel-based method
K(x,x’) = cov(Z(x), Z(x’))
 FLEXIBLE (see after…)
• When Z is stationary, K(x,x’) depends on h=x-x’
 we denote k(h) = K(x,x’)
9
Kriging model (kernel)
• “Making new kernels from old” (Rasmussen and Williams, 2006)
 K1 + K2
 cK, with c>0
 K1K2
…
10
Kriging model (common choice)
• Tensor-product structure
k(h) = k1(h1)k2(h2)…kd(hd)
with hi=xi – xi’, and ki Gaussian, Matern 5/2…
11
The main idea on an example
• Ishigami, defined on D = [-π,π]3, with A=7, B=0.1:
f(x) = sin(x1) + Asin2(x2) + B(x3)4sin(x1)
• This is a block additive decomposition
f(x) = f2(x2) + f1,3(x1,x3)
Z(x) = Z2(x2) + Z1,3(x1,x3)  k(h) = k2(h2) + k1,3(h1,h3)
12
The main idea on an example
• Comparison of the two Kriging models
– Training set: 100 points from a maximin Latin hypercube
– Test set: 1000 additional points from a unif. distribution
13
The schema to be generalized
f
k = k2 + k1,3
14
Outline
Introduction
[How to choose a Kriging model for the
Ishigami function]
1. From FANOVA graphs to block additive kernels
[Generalizes 1.]
2. Estimation methodologies
[With a new sensitivity index]
3. Applications
4. Some comments
15
From FANOVA graphs
to block additive kernels
FANOVA decomposition
(Efron and Stein, 1981)
• Assume that X1, …, Xd are independent random
variables. Let f be a function defined on D1x…xDd
and dν=dν1…dvd an integration measure. Then:
f(X) = μ0 + Σμi(Xi) + Σμi,j(Xi,Xj) + Σμi,j,k(Xi,Xj,Xk) + …
where all terms are centered and orthogonal to the
others. They are given by:
μ0
:= E(f(X)),
μi(Xi) := E(f(X)|Xi) – μ0
μi,j(Xi,Xj) := E(f(X)|Xi,Xj17
) - μi(Xi) - μj(Xj) – μ0
and so on…
FANOVA decomposition
Example. Ishigami function, with uniform measure
on D = [-π,π ]3. With a=π, b=2π5/5, we have:
f(x) = sin(x1) + Asin2(x2) + B(x3)4sin(x1)
= aA + sin(x1)(1+bB) + A(sin2(x2)-a) + B[(x3)4-b]sin(x1)
μ0
μ1(x1)
μ2(x2)
μ1,3(x1,x3)
(main effect)
(main effect)
(2nd order interaction)
18
FANOVA decomposition
Example (following)
• Some terms can vanish, due to averaging, as μ3 ,
or μ1 if B = -1/b, but this depends on the integration
measure, and only happens when there exist terms
of higher order
• On the other hand, and for the same reason, we
always have:
μ2,1 = μ2,3 = 0 and, under mild conditions: μ1,3 ≠ 0
19
FANOVA decomposition
• The name “FANOVA” becomes from the relation
on variances implied by orthogonality:
var(f(X)) = Σvar(μi(Xi)) + Σvar(μi,j(Xi,j)) + …
which measures the importance of each term.
• var(μJ(XJ))/var(f(X)) is often called a Sobol indice
20
FANOVA graph
Vertices: variables
Edges: if there is at least one
interaction (at any order)
Width: prop. to variances
Here (example above):
μ1,2 = μ2,3 = 0
The graph does not depend
on the integration measure,
(under mild conditions)
21
FANOVA graph and cliques
• A complete subgraph: all edges exist
• A clique: maximal complete subgraph
Cliques:
{1,3} and {2}
22
Cliques of FANOVA graph
and block additive decomposition
f(x) = cos(x1+x2+x3)
+sin(x4+x5+x6)
+tan(x3+x4)
f(x) = f1,2,3(x1,x2,x3)
+f4,5,6(x4,x5,x6)
+f3,4(x3,x4)
Cliques:
{1,2,3}, {4,5,6}, {3,4}
23
Why cliques?
f(x) = cos(x1+x2+x3)
+sin(x4+x5+x6)
+tan(x3+x4)
{1,2,3,4,5,6}
f(x) = f1,…,6(x1,…,x6) !!!
24
Incomplete subgraphs
 rough model forms
Why cliques?
f(x) = cos(x1+x2+x3)
+sin(x4+x5+x6)
+tan(x3+x4)
{1,2},{2,3},{1,3},{3,4}{4,
5,6}
f(x) = f1,2(x1,x2)
+f2,3(x2,x3)+f1,3(x1,x3)
+f4,5,6(x4,x5,x6)
+f3,4(x3,x4)
25
Non maximality
 wrong model forms
Cliques of FANOVA graph
and Kriging models
Cliques:
{1,2,3}, {4,5,6}, {3,4}
f(x) = f1,2,3(x1,x2,x3)
+f4,5,6(x4,x5,x6)
+f3,4(x3,x4)
Z(x) = Z1,2,3(x1,x2,x3)
+ Z4,5,6(x4,x5,x6)
+ Z3,4(x3,x4)
k(h) = k1,2,3(h1,h2,h3)
+ k4,5,6(h4,h5,h6)
26
+ k3,4(h3, h4)
Estimation methodologies
Graph estimation
• Challenge: estimate all interactions (at any order)
involving two variables
• Two issues:
1. The computer code is time-consuming
2. Huge number of combinations for the usual Sobol
indices
28
Graph estimation
• Solutions
1. Replace the computer code by a metamodel, for
instance a Kriging model with a standard kernel
2. Fix x3, …, xd, and consider the 2nd order interaction of
the 2-dimensional function:
(x1, x2)  f(x) = f1(x-2) + f2(x-1) + f12(x1,x2; x-{1,2})
Denote D12(x3,…,xd) the unnormalized Sobol indice, and
define:
D12 = E(D12(x3,…,xd))
Then D12 > 0 iif (1,2) is an edge of the FANOVA graph
29
Graph estimation
• Comments:
– The new sensitivity index is computed by averaging 2nd
order Sobol indices, and thus numerically tractable
– In practice “D12 > 0” is replaced by “D12 > δ”
 Different thresholds give different FANOVA graphs
30
Kriging model estimation
• Assume that there are L cliques of size d1,…,dL.
The total number of parameters to be estimated is:
ntrend + (d1+1) + … + (dL + 1)
(trend, “ranges” and variance parameters)
• MLE is used, 3 numerical procedures tested
31
Kriging model estimation
• Isotropic kernels are useful for high dimensions
• Example: suppose that C1={1,2,3}, C2={4,5,6},
C3={3,4}, and x7, …, x16 have a smaller influence
1st solution: C4={x7}, …, C13={x16}
N = ntrend + 4 + 4 + 3 + 10*2 = ntrend + 31
2nd solution: C4 = {x7, …, x16}, with an isotropic kernel
N = ntrend + 4 + 4 + 3 + 2 = ntrend + 13
32
Applications
a 6D analytical case
• f(x) = cos(- 0.8 - 1.1x1 + 1.1x2 + x3)
+ sin(- 0.5 + 0.9x4 + x5 – 1.1x6)
+ (0.5 + 0.35x3 - 0.6x4)2
• Domain: [-1,1]6
• Integration measure: uniform
• Training set: 100 points from a maximin LHD
• Test set: 1000 points drawn from a unif. dist.
34
a 6D analytical case
Estimated graph
Usual Sensitivity Analysis
(from R package sensitivity)
35
a 6D analytical case
36
a 16D analytical case
• Consider the same function, but assume that it is
in a 16D space (with 10 more inactive variables)
• Including all the inactive variables in one clique
is improving the prediction
37
A 6D case study
• Piston slap data set (Fang et al., 2006)
– Unwanted noise of engine, simulated using a finite
elements method
• Training set: 100 points
• Test set: 12 points
• Leave-one-out is also considered
38
A 6D case study
39
A 6D case study
Leave-one-out RMSE: 0.0864 (standard Kriging), 0.0371 (modified Kriging)
40
Some comments
Some comments
• Main strengths
– Adapting the kernel to the data in a flexible manner
– A substantial improvement may be expected in prediction
• Depending on the function complexity
• Some drawbacks
– Dependence on the first initial metamodel
– Sometimes a large nb of parameters to be estimated
• May decrease the prediction power
42
THANKS A LOT FOR
ATTENDING!
43