Approximate Bayesian Inference in Spatial GLMMs with
Skew Gaussian Latent Variables
Fatemeh Hosseini 1, Jo Eidsvik2 and Mohsen Mohammadzadeh1
1. Department of Statistics, Tarbiat Modares University, Tehran, Iran
2. Department of Mathematical Science, Norwegian University of Science and Technology, Norway
1
Spatial Generalized linear mixed models (Spatial GLMMs) ......................................................
• Likelihood: π(y|x), exponential family. y = (y1, . . . , yk ).
• Prior for latent spatial variable: π(x|η). x = (x1, . . . , xn), n ≥ k.
• Prior for model parameters: π(η).
GOALS:
• Present model with (closed) skew Normal π(x|η).
• Perform approximate Bayesian inference: π̂(x|y, η), π̂(η|y), π̂(xi|y).
(Closed) skew Normal generalization of Rue and Martino (2007), Eidsvik
et al. (2009), Rue et al. (2009), who use Normal π(x|η).
2
Examples ........................................................................
Registration sites for precipitation.
72
66
250 km
70
65.5
NORWEGIAN SEA
65
68
66
North
Trondelag
64.5
Latitude
Latitude
NORWAY
50 km
64
FINLAND
64
SWEDEN
62
63.5
South
Trondelag
60
63
58
More and
Romsdal
62.5
5
6
7
8
9
10
11
12
13
54
14
0
5
10
15
20
Longitude
Longitude
(a)
Figure 1:
DENMARK
56
SWEDEN
NORWAY
62
(b)
(a) Precipitation data. (b) Trout data.
y: Counts/Proportions
x: Latent variables/Logit field
η: Cov/Mean/Skewness
GOALS:: π̂(xi|y), π̂(η|y)
3
25
30
35
Examples ........................................................................
Figure 2: North Sea well log and synthetic seismic data (Karimi, Omre and Mohammadzadeh, 2009)
y: Seismic data, Gaussian/linear
x: Latent variables/elastic properties
η: Cov/Mean/Skewness
GOALS:: π̂(xi|y), π̂(η|y)
4
Skew Normal and Closed Skew Normal ........................................................................
• Azzalini (1985) introduced the skew normal (SN) distribution.
• SN i) is larger class than Normal. ii) SN includes Normal with an extra
skewness parameter
• Dominguez-Molina et al. (2003) introduced the closed skew normal
(CSN) distribution.
• CSN i) is larger class than SN. ii) CSN includes SN with extra skewness
parameters. iii) CSN is conjugate with Normal likelihood.
5
Skew Normal ........................................................................
• x ∼ SN (µ, Σ, λ)
1
−
f (x|λ, µ, Σ) = 2φn(x; µ, Σ)Φ(λ Σ 2 (x − µ)),
′
- λ ∈ Rn is a skewness parameter.
• If λ = 0 ⇒ SN reduces to Normal: x ∼ Nn(µ, Σ).
6
(1)
Skew Normal ........................................................................
7
7
7
6
6
6
5
5
5
4
4
4
3
3
3
2
2
2
1
1
1
−4
2
4
6
µ = [2; 4], Σ = [5 0.7; 0.7 1], λ = [5 0]
8
−2
0
2
µ = [2; 4], Σ = [5 0.7; 0.7 1], λ = [−5 0]
Figure 3: Contour plots of SN distribution.
7
−2
0
2
4
6
8
µ = [2; 4], Σ = [5 0.7; 0.7 1], λ = [0 0]
Closed Skew Normal ........................................................................
• x ∼ CSNn,q (µ, Σ, D, ν, ∆)
′
−1
fn,q (x|µ, Σ, D, ν, ∆) = Φq (0; ν, ∆ + DΣD )
φn(x; µ, Σ) Φq (D(x − µ); ν, ∆).
• If ν = 0 , ∆ = I and q = 1, CSN reduces to SN:
CSNn,1(µ, Σ, D, 0, 1) = SN (µ, Σ, λ), D = λ′Σ−1/2.
• If D = 0, CSN reduces to Normal, x ∼ N (µ, Σ).
• Expectation of CSN is E(X) = µ + ΣD′ψ,
Φ∗q (r, ν, ∆ + DΣD ′)
ψ = ψ(0, Σ, D, ν, ∆) =
|r=0 , Φ∗q (r; ν, Ω) = [∇Φq (r; ν, Ω)]′
′
Φq (0, ν, ∆ + DΣD )
8
(2)
Skew and Closed Skew Normal Distributions ............................................
7
7
0.18
0.12
6
6
0.16
0.1
0.14
5
5
0.12
4
x2
x2
0.08
4
0.1
0.06
0.08
3
3
0.06
0.04
2
2
0.04
0.02
1
0.02
2
4
6
1
8
x1
2
4
6
8
x1
µ=[2,4], Σ=[5 0.7;0.7 1], D=[5 0;0 5], ν=[0 0], ∆=I
µ=[2,4], Σ=[5 0.7;0.7 1], D=[5 0], ν=0, ∆=1
2
Figure 4: Contour plots of CSN distribution.
9
Closed Skew Normal ............................................
• The CSN distribution is closed under:
- Linear transformations: Let x be CSN, then Ax + b is
CSNk,q (Aµ + b, AΣA′, DΣA′(AΣA′ )−1, ν, ∆ + DΣD ′ − DΣA′(AΣA′ )−1AΣD ′ ),
- Marginalization: Let x = (x′1, x′2)′, x1 size k, x2 size n − k, then
x1 ∼ CSNk,q (µ1, Σ11, D ∗, ν, ∆∗),
- Conditioning: x2|x1
∗
CSNn−k,q (µ2 + Σ21Σ−1
11 (x1 − µ1 ), Σ22·1, D2 , ν − D (x1 − µ1 ), ∆).
where
.
∗
′
−1
D ∗ = D1 + D2Σ21Σ−1
11 , ∆ = ∆ + D2 Σ22·1 D2 , Σ22·1 = Σ22 − Σ21 Σ11 Σ12 .
µ1
Σ11 Σ12
, D = (D1, D2)
µ=
, Σ=
Σ21 Σ22
µ2
10
Closed Skew Normal ............................................
A Normal / conditioning formulation of CSN:
x
x0
∼ Nn+q
′
µ
Σ
ΣD
,
.
′
−ν
DΣ ∆ + DΣD
Then, (x|x0 > 0) ∼ CSNn,q (µ, Σ, D, ν, ∆).
Sample CSN by rejection sampling.
11
Spatial GLMM with SN prior ........................................................................
Spatial GLMM defined at three levels:
• π(yi|xi) = exp{yixi − b(xi) + c(yi)}, i = 1, . . . , k. Conditionally independent.
• x = (xobs, xpred), xobs = Ax at k observation sites, xpred at (n − k)
prediction sites. π(x|η) = SN (Hβ, Σθ , λ), λ = λ01.
1
2
exp − (x − Hβ)′Σθ −1(x − Hβ)
π(x|η) =
2
(2π)n/2|Σθ |1/2
1
−
· Φ(λ′Σθ 2 (x − Hβ))
|i−j|
• Σθ (i, j) = σ 2 exp(− ϕ ), θ = (σ, ϕ).
• Model parameters η = (β, θ, λ0). Proper prior density π(η).
12
Spatial GLMM with SN prior ...........................................
• Joint:
π(y, x, η) = π(y|x)π(x|η)π(η)
!
k
X
2
1
=
(x − Hβ)′ Σθ −1(x − Hβ)
[y
x
−
b(x
)
+
c(y
)]
−
exp
i
i
i
i
n/2
1/2
2
(2π) |Σθ |
i=1
′
×Φ(λ
− 12
Σθ (x
− Hβ))π(η),
• Full conditional π(x|y, η) not generally available.
• If Gaussian likelihood b(xi) = b0x2i , then π(x|y, η) is CSN.
13
(3)
Approximation Bayesian Inference .....................................
• Linearize full conditional for x:
π(x|y, η) ≈ CSNn,1(µx|y,η , Σx|y,η , Dx|y,η , ν x|y,η , 1),
⊲ µx|y,η = Zβ + Σθ A′R−1(z(y, xobs) − AZβ),
zi(yi, xi) = [yi − b′(xi) + xib′′(xi)]/b′′(xi).
R = AΣθ A′ + P and Pii = 1/b′′(xi), i = 1, · · · , k.
⊲ Σx|y,η = Σθ − Σθ A′R−1AΣθ .
1
−
⊲ Dx|y,η = λ′Σθ 2 ,
1
−
⊲ ν x|y,η = λ′Σθ 2 (Zβ − µx|y,η ).
14
Approximate Bayesian Inference .....................................
• Iterative algorithm for fitting a CSN approximation:
1. Choose starting value x(0), set m = 0.
2. Calculate π̂(x|y, η) = CSNn,1(x; µ̂x|y,η (x(m)), Σ̂x|y,η (x(m)), Dx|y,η, ν̂ x|y,η (x(m)), 1).
3. Calculate
′
x(m+1) = Ê(x|y, η) = µ̂x|y,η (x(m)) + Σ̂x|y,η (x(m))Dx|y,η
ψ̂,
ψ̂ = ψ(0, Σ̂x|y,η (x(m)), Dx|y,η , ν̂x|y,η (x(m)), 1)
4. Set m = m + 1.
5. Convergence is obtained after a few iterations.
π̂(x|y, η) = CSNn,1 (x; µ̂x|y,η (x(m+1)), Σ̂x|y,η (x(m+1)), Dx|y,η , ν̂ x|y,η (x(m+1)), 1).
15
Approximate Parametric Inference ...........................................
π(y|x)π(x|η)π(η) π̂(η|y) ∝
x=Ê(x|y ,η ) .
π̂(x|y, η)
Similar to the Laplace approximation of Tierney and Kadane (1986).
−6
−6
x 10
3
2.5
2
η2
1.5
0
1.5
−1
−1
1
−2
0.5
−3
−4
−6
2
1
η2
0
2.5
2
2
1
x 10
3
−4
−2
η1
0
2
1
−2
0.5
−3
−4
−6
−4
−2
η1
Figure 5: Numerical evaluation of π̂(η|y)
16
0
2
Approximate Spatial Prediction ........................................................................
• Predict the latent variables xpred.
• π̂(x|y, η) is CSN → marginal π̂(xj |y, η) is also CSN.
•
π̂(xj |y) =
X
π̂(xj |y, η ℓ) · π̂(η ℓ|y), j = k + 1, · · · , n.
ℓ
• Mixture of CSN distributions.
17
Approximate Spatial Prediction ........................................................................
• Improved approximation for predictive marginals.
•
π(y|x)π(x|η) π̃(xj |y, η) ∝
x=E(x−j |xj ,y ,η ) ,
π̃(x−j |xj , y, η)
• π̃(x−j |xj , y, η) is CSN, approximated and evaluated for fixed xj .
• π̃(xj |y) =
P
ℓ π̃(xj |y, η ℓ) · π̂(η ℓ|y), j = k + 1, · · · , n.
18
(4)
Example 1 : Precipitation in Middle Norway ........................................................................
Registration sites for precipitation.
66
65.5
NORWEGIAN SEA
65
North
Trondelag
Latitude
64.5
50 km
64
63.5
South
Trondelag
63
More and
Romsdal
62.5
SWEDEN
NORWAY
62
5
6
7
8
9
10
11
12
13
14
Longitude
Figure 6: Map of observation sites
• The data are number of rainy days and the number of days in operation
for i = 1, · · · , 92 registration sites.
xi
e
• Binomial data, π(yi|xi) = Bin( 1+exi , 14).
19
Example 1 : Precipitation in Middle Norway ........................................................................
Marginal density of standard devition
8
Skew−Normal
π(σ|y)
6
σSN<σN
Normal
4
2
0
0.2
0.25
0.3
0.35
0.4
σ
0.45
0.5
0.55
0.6
0.65
Marginal density of range
0.04
π(φ|y)
0.03
Normal
SN
φ
Skew−Normal
N
>φ
0.02
0.01
0
0
10
20
30
40
φ
50
60
70
80
90
Marginal density of skewness
0.2
π(λ|y)
0.15
λ>0
Skew−Normal
0.1
0.05
0
−4
−2
0
2
λ
4
6
8
10
Figure 7: Posterior marginals for parameters
Solid is approximate Bayesian inference.
Dashed is independent proposal MCMC π̂(η|y) · π̂(x|y, η).
Acceptance rates 70 − 80%.
20
Example 1 : Precipitation in Middle Norway ........................................................................
η̂ = (0.26, 51.2, 3.2), Location=(63.30,10.28)
Marginal density
2
Direct
Improved
MCMC
1.5
1
0.5
0
−1
−0.5
0
0.5
1
Latent variables
1.5
2
2.5
3
η̂ = (0.35, 35), Location=(63.30,10.28)
Marginal density
2
Direct
Improved
MCMC
1.5
1
0.5
0
−1
−0.5
0
0.5
1
Latent variables
1.5
2
2.5
3
Figure 8: Predictions at unobserved site. Top is SN, below is Normal.
Three approximations:
π̂(xj |y, η̂), π̃(xj |y, η̂), π mcmc(xj |y, η̂)
21
Example 2: Norwegian Lakes Acidification
72
250 km
70
68
66
Latitude
NORWAY
FINLAND
64
SWEDEN
62
60
58
DENMARK
56
54
0
5
10
15
20
25
30
35
Longitude
Figure 9: Map of observation sites.
• Data are high/low fish quality in 542 lakes.
xi
e
• Bernoulli data, π(yi|xi) = Bin( 1+exi , 1).
• Explanatory variable is acid capacity index (pH).
22
Example 2: Norwegian Lakes Acidification ........................................................................
Marginal density of standard deviation
Marginal density of standard deviation
2
1
0
1
1.5
2
2.5
σ
Marginal density of range
0.01
0.005
0
50
100
150
200
250
300
φ
Marginal density of β
350
1
0
3
π(φ|y)
π(φ|y)
π(σ|y)
π(σ|y)
2
1
0
100
400
200
250
300
φ
Marginal density of β
400
−6
−5
β
−4
0.5
0
−7
−3
−6
1
−5
β
1
Marginal density of skewness
0.2
π(λ0|y)
350
1
π(β1|y)
1
150
1
0.5
0.1
0
−5
3
0.005
1
π(β |y)
2
2.5
σ
Marginal density of range
0.01
1
0
−7
1.5
0
λ
5
10
0
Figure 10: Posterior marginals for model parameters.
Solid is approximate Bayesian inference.
Dashed is MCMC (Acceptance rates about 80 percent)
23
−4
−3
Example 2: Norwegian Lakes Acidification ........................................................................
η̂ = (1.65, 205, −4.55, 3), location j = (24.2, 69.9)
location j = (24.2, 69.9)
0.8
0.6
1
π(xj|y)
π(xj|η,y)
1.5
0.4
0.5
0.2
4
4.5
5
5.5
6
6.5
xj
η̂ = (1.68, 201, −4.58), location j = (24.2, 69.9)
0
7
5
6
x
j
location j = (24.2, 69.9)
7
8
7
8
π(x |y)
0.8
0.6
j
1
j
π(x |η,y)
1.5
4
0.5
0
0.4
0.2
4
4.5
5
5.5
xj
6
6.5
0
7
4
5
6
xj
1.5
Skew−Normal
Normal
0.4
Normal
j
Skew−Normal
π(x |y)
π(xj|η,y)
0.6
1
0.5
0
0.2
0
4
4.5
5
5.5
x
6
6.5
7
4
5
6
7
8
xj
j
Figure 11: Prediction at unobserved site. Left is π̂(xi |y, η̂). Right is π̂(xi |y).
24
Example 3: Seismic inversion
Figure 12: North Sea well log and synthetic seismic data (Karimi, Omre and Mohammadzadeh, 2009)
• Data are seismic reflection amplitudes for different traveltimes and several incidence angles.
• π(y|x) = N ormal(Gx, T ).
• In this case π(x|y, η) is CSN without approximations.
25
Example 3: Seismic inversion
(a)
(b)
Figure 13: CSN priors. From Karimi, Omre and Mohammadzadeh (2009)
Fitted tri-variate CSN prior from well logs.
Same for all depth, but correlated.
26
Example 3: Seismic inversion
(a)
(b)
Figure 14: Inversion result. From Karimi, Omre and Mohammadzadeh (2009)
CSN result (left) and Normal result (right).
27
Conclusion
• Skew Normal prior for latent variables in Spatial GLMMs.
• Closed skew Normal for approximate Bayesian inference → approximations comparable with MCMC results. Direct inference (seconds),
improved inference (minutes), MCMC (hours).
• Closed skew Normal results for parameter estimates differ from Normal: Larger ’correlation’, less ’standard deviation’. Skewness is hard to
estimate from our data.
• Closed skew Normal results for prediction differ from Normal only at
some sites, where the skewness ’kicks in’. More skewness ’directions’
could change this (larger q).
• Computational / numerical challenges: Φq (x), E(x|y, η), π̂(xi|y, η),
when n or k are very large.
28