Introduction
Approximate Bayes
GLMs
Conclusions
References
Module 4: Bayesian Methods
Lecture 7: Generalized Linear Modeling
Jon Wakefield
Departments of Statistics and Biostatistics
University of Washington
Introduction
GLMs
Approximate Bayes
Outline
Introduction and Motivating Examples
Generalized Linear Models
Definition
Bayes Linear Model
Bayes Logistic Regression
Generalized Linear Mixed Models
Approximate Bayes Inference
The Approximation
Case-Control Example
Conclusions
Conclusions
References
Introduction
Approximate Bayes
GLMs
Conclusions
References
Introduction
• In this lecture we will discuss Bayesian modeling in the context of
Generalized Linear Models (GLMs).
• This discussion will include the addition of random effects, to give
Generalized Linear Mixed Models (GLMMs).
• Estimation via a quick and R based technique (INLA) will be
demonstrated.
• An approximation technique that is useful in the context of Genome Wide
Association Studies (GWAS) (in which the number of tests is large) will
also be introduced.
Introduction
Approximate Bayes
GLMs
Conclusions
References
Motivating Example I: Logistic Regression
• We consider case control data for the disease Leber Hereditary Optic
Neuropathy (LHON) disease with genotype data for marker rs6767450:
Cases
Controls
Total
CC
6
10
16
CT
8
66
74
TT
75
163
238
Total
89
239
328
• Let x = 0, 1, 2 represent the number of T alleles, and p(x) the probability
of being a case, given x alleles.
• Within a likelihood framework one may fit the multiplicative odds model:
p(x)
= exp(α) × exp(θx).
1 − p(x)
• Interpretation:
• exp(α) is of little use given the case-control sampling.
• exp(θ) is the log odds ration describing the multiplicative change in risk
given an additional T allele.
Introduction
GLMs
Approximate Bayes
Conclusions
References
R code for Logistic Regression Estimation via Likelihood
x <− c ( 0 , 1 , 2 )
y <− c ( 6 , 8 , 7 5 )
z <− c ( 1 0 , 6 6 , 1 6 3 )
l o g i t m o d <− glm ( c b i n d ( y , z ) ˜ x , f a m i l y =” b i n o m i a l ” )
t h e t a h a t <− l o g i t m o d $ c o e f f [ 2 ]
# Log o d d s r a t i o
thetahat
x
0.4787428
e x p ( t h e t a h a t ) # Odds r a t i o
x
# Odds o f d i s e a s e a r e a s s o c i a t e d w i t h an i n c r e a s e
1.614044
# o f 61% f o r e a c h e x t r a T
V <− v c o v ( l o g i t m o d ) [ 2 , 2 ]
# s t a n d a r d e r r o r ˆ2
# Asymptotic c o n f i d e n c e i n t e r v a l f o r odds r a t i o
> e x p ( t h e t a h a t −1.96∗ s q r t (V ) )
x
0.987916
> e x p ( t h e t a h a t +1.96∗ s q r t (V ) )
x
2.637004
# So 95% i n t e r v a l ( j u s t ) c o n t a i n s 1 .
Introduction
GLMs
Approximate Bayes
Conclusions
R code for Logistic Regression Hypothesis Testing via a LRT
# Now l e t ’ s l o o k a t a l i k e l i h o o d r a t i o t e s t
> logitmod
Call :
glm ( f o r m u l a = c b i n d ( y , z ) ˜ x , f a m i l y = ” b i n o m i a l ” )
Coefficients :
( Intercept )
x
−1.8077
0.4787
D e g r e e s o f Freedom : 2 T o t a l ( i . e . N u l l ) ;
1 Residual
Null Deviance :
15.01
R e s i d u a l D e v i a n c e : 1 0 . 9 9 AIC : 2 7 . 7 9
> p c h i s q ( 1 5 . 0 1 − 1 0 . 9 9 , 1 , l o w e r . t a i l =F )
[ 1 ] 0.04496371
# So j u s t s i g n i f i c a n t a t t h e 5% l e v e l .
• So for these data both estimation and testing point towards borderline
significance.
• Under Hardy Weinberg Equilibrium (HWE) we would estimate (in the
controls) the minor allele (C) frequency (MAF) as
10 + 66/2
= 0.18,
239
which helps to explain the lack of power.
References
Introduction
Approximate Bayes
GLMs
Conclusions
References
Motivating Example II: FTO Data Revisited
Linear Model Example
• y = weight
• xg = fto heterozygote ∈ {0, 1}
• xa = age in weeks ∈ {1, 2, 3, 4, 5}
We examine the fit of the model
E[Y |xg , xa ] = β0 + βg xg + βa xa + βint xg xa .
> f t o <− r e a d . t a b l e ( ” h t t p : / /www . s t a t . w a s h i n g t o n . edu /˜ h o f f / SISG / f t o d a t a . t x t ” ,
h e a d e r=TRUE)
> l i n y <− f t o $ y
> l i n x g <− f t o $ x g
> l i n x a <− f t o $ x a
> l i n x i n t <− f t o $ x g x a
> f t o d f <− l i s t ( l i n y =l i n y , l i n x g=l i n x g , l i n x a=l i n x a , l i n x i n t = l i n x i n t )
> o l s . f i t <− lm ( l i n y ˜ l i n x g+l i n x a+l i n x i n t , d a t a=f t o d f )
> summary ( o l s . f i t )
Coefficients :
E s t i m a t e Std . E r r o r t v a l u e Pr ( >| t | )
( I n t e r c e p t ) −0.06822
1.42230
−0.048
0.9623
linxg
2.94485
2.01143
1.464
0.1625
linxa
2.84421
0.42884
6 . 6 3 2 5 . 7 6 e−06 ∗∗∗
linxint
1.72948
0.60647
2.852
0.0115 ∗
Introduction
GLMs
Approximate Bayes
Conclusions
References
Motivating Example III: Salamander Data
• McCullagh and Nelder (1989) describe data on the success of matings
between male and female salamanders of two population types
(roughbutts, RB, and whitesides, WS).
• The experimental design is complex but involves three experiments having
multiple pairings, with each salamander being involved in multiple
matings, so that the data are not independent.
• One way of modeling the dependence is through crossed random effects.
• The first lab experiment was in the summer of 1986, and the second and
third were in the fall of the same year.
Introduction
Approximate Bayes
GLMs
Conclusions
References
Motivating Example III: Salamander Data
• There are 20 females and 20 males, with half of each gender being RB and
half being WS.
• In each experiment each female is paired with 3 males (to give 30 matings
in total), and each male is paired with 3 females (to give 30 matings in
total).
• The data are:
Experiment 1 Experiment 2 Experiment 3
Y
%
Y
%
Y
%
RR
22
73 18
60 20
67
RW
20
67 14
47 16
53
WR
7
23
7
23
5
17
WW 21
70 20
67 19
63
The Observed number (Y ) and percentage (%) of successes out of 30
matings in three experiments.
Introduction
Approximate Bayes
GLMs
Conclusions
Motivating Example III: Salamander Data
• Mixed effects models consist of fixed effects and random effects.
• Fixed effects are population characteristics, while random effects are
unit-specific quantities that are assigned a distribution.
• Mixed effects models allow dependencies in responses to be modeled.
• Example: The simplest case is the one-way ANOVA model with n
observations in each of m groups:
Yij
=
�ij
∼iid
bi
∼iid
β0 + bi + �ij
N(0, σ�2 )
N(0, σb2 )
i = 1, ..., m; j = 1, ..., n.
• In this model, β0 is the fixed effect and b1 , ..., bm are random effects.
References
Introduction
Approximate Bayes
GLMs
Conclusions
References
Crossed versus Nested Designs
• Suppose we have two factors, A and B with a and b levels respectively. If
each level of A is crossed with each level of B we have a factorial design.
• In a nested situation j = 1 in level 1 of factor A has no meaningful
connection with j = 1 in level 2 of factor A.
Treatment
Treatment
Subject 1
2
3
4
Subject 1
2
3
4
1
× × × ×
1
×
2
× × × ×
2
×
3
× × × ×
3
×
4
× × × ×
4
×
5
× × × ×
5
×
6
× × × ×
6
×
7
× × × ×
7
×
8
× × × ×
8
×
Left: Crossed design. Right: Nested design.
The × symbols show where observations are measured.
Introduction
GLMs
Approximate Bayes
Conclusions
Motivating Example III: Salamander Data
• Let Yijk denote the response (failure/success) for female i and male j in
experiment k.
• There are 360 binary responses in total.
• For illustration we fit “model C” that was previously considered by Karim
and Zeger (1992) and Breslow and Clayton (1993):
m
m
logit Pr(Yijk = 1|β, bikf , bjk
) = xijk β k + bikf + bjk
where xijk is a 1 × 4 vector representing the intercept and indicators for
female WS, male WS, and male and female both WS, and β k is the
corresponding fixed effect (so that this model allows the fixed effects to
vary by experiment).
• The model contains six random effects:
2
bikf ∼iid N(0, σfk
),
2
bikm ∼iid N(0, σmk
),
k = 1, 2, 3
one for each of males and females, and in each experiment.
References
Introduction
Approximate Bayes
GLMs
Conclusions
References
Generalized Linear Models
• Generalized Linear Models (GLMs) provide a very useful extension to the
linear model class.
• GLMs have three elements:
1. The responses follow an exponential family.
2. The mean model is linear in the covariates on some scale.
3. A link function relates the mean of the data to the covariates.
• In a GLM the response yi are independently distributed and follow an
exponential family so that the distribution is of the form
p(yi |θi , α) = exp({yi θi − b(θi )}/α + c(yi , α)),
where θi and α are scalars.
• Examples: Normal, Poisson, binomial.
• It is straightforward to show that
E[Yi |θi , α] = µi = b � (θi ),
var(Yi | θi , α) = b �� (θi )α,
for i = 1, ..., n, with cov(Yi , Yj | θi , θj , α) = 0 for i �= j.
Introduction
GLMs
Approximate Bayes
Conclusions
References
Generalized Linear Models
• The link function g (·) provides the connection between µ = E[Y | θ, α]
and the linear predictor xβ, via
g (µ) = xβ,
where x is a vector of explanatory variables and β is a vector of regression
parameters.
• For normal data, the usual link is the identity
g (µ) = µ.
For binary data, a common link is the logistic
„
«
µ
g (µ) = log
.
1−µ
For Poisson data, a common link is the log
g (µ) = log (µ) .
Introduction
Approximate Bayes
GLMs
Conclusions
References
Bayesian Modeling with GLMs
• For a generic GLM, with regression parameters β and a scale parameter α,
the posterior is
p(β, α|y) ∝ p(y|β, α) × p(β, α).
• Two problems immediately arise:
• How to specify a prior distribution p(β, α)?
• How to perform the computations required to summarize the posterior
distribution (including the calculation of Bayes factors).
Introduction
Approximate Bayes
GLMs
Conclusions
References
Bayesian Computation
Various approaches are available:
• Conjugate analysis — the prior combines with likelihood in such a way as
to provide analytic tractability (at least for some parameters).
• Analytical Approximations — asymptotic arguments used (e.g. Laplace).
• Numerical integration.
• Direct (Monte Carlo) sampling from the posterior, as we have already seen.
• Markov chain Monte Carlo — very complex models can be implemented,
for example within the free software WinBUGS.
• Integrated nested Laplace approximation (INLA). Cleverly combines
analytical approximations and numerical integration.
Introduction
Approximate Bayes
GLMs
Conclusions
References
Integrated Nested Laplace Approximation (INLA)
• To download INLA:
i n s t a l l . p a c k a g e s ( c ( ” f i e l d s ” , ” numDeriv ” , ” pixmap ” ,
” mvtnorm ” , ”R . u t i l s ” ) )
s o u r c e ( ” h t t p : / /www . math . n t n u . no / i n l a / givemeINLA . R” )
l i b r a r y ( INLA )
i n l a . u p g r a d e ( t e s t i n g=TRUE)
• The homepage of the software is here:
http://www.r-inla.org/home
• The fitting of many common models is described here:
http://www.r-inla.org/models/likelihoods
• There are also lots of examples.
• INLA can fit GLMs, GLMMs and many other classes.
Introduction
GLMs
Approximate Bayes
Conclusions
References
INLA for the Linear Model
• We first fit a linear model to the FTO data with the default prior settings.
l i n y <− f t o $ y
l i n x g <− f t o $ X [ , 2 ]
l i n x a <− f t o $ X [ , 3 ]
l i n x i n t <− f t o $ X [ , 4 ]
f t o d f <− l i s t ( l i n y =l i n y , l i n x g=l i n x g , l i n x a=l i n x a , l i n x i n t = l i n x i n t )
l i n . mod <− i n l a ( l i n y ˜ l i n x g+l i n x a+l i n x i n t , d a t a=f t o d f ,
f a m i l y =” g a u s s i a n ” )
> summary ( l i n . mod )
Fixed e f f e c t s :
mean
sd 0.025 quant
0.5 quant 0.975 quant
( I n t e r c e p t ) −0.063972 1 . 3 7 3 9 2 3 2 −2.7804936 −0.06359089
2.653028
linxg
2 . 9 2 7 3 6 2 1 . 9 4 3 5 6 9 0 −0.9107281 2 . 9 2 9 1 8 2 4 3
6.767026
linxa
2.842509 0.4139664
2.0233339
2.84255770
3.661029
linxint
1.732381 0.5849059
0.5754914
1.73223551
2.889490
Model h y p e r p a r a m e t e r s :
mean
sd
0.025 quant 0.5 quant
P r e c i s i o n f o r t h e Gaus o b s e r s 0 . 3 0 5 1 0 . 1 0 0 4 0 . 1 4 8 9
0.2924
0.975 quant
P r e c i s i o n f o r t h e Gaus o b s e r v a t i o n s 0 . 5 3 8 1
>
>
>
>
>
>
• The posterior means and standard deviations are in very close agreement
with the OLS fits presented earlier.
Introduction
Approximate Bayes
GLMs
Conclusions
References
R Code for Marginal Distributions
b e t a x g g r i d <− l i n . m o d $ m a r g i n a l s . f i x e d $ l i n x g [ , 1 ]
b e t a x g m a r g <− l i n . m o d $ m a r g i n a l s . f i x e d $ l i n x g [ , 2 ]
b e t a x a g r i d <− l i n . m o d $ m a r g i n a l s . f i x e d $ l i n x a [ , 1 ]
b e t a x a m a r g <− l i n . m o d $ m a r g i n a l s . f i x e d $ l i n x a [ , 2 ]
b e t a x i n t g r i d <− l i n . m o d $ m a r g i n a l s . f i x e d $ l i n x i n t [ , 1 ]
b e t a x i n t m a r g <− l i n . m o d $ m a r g i n a l s . f i x e d $ l i n x i n t [ , 2 ]
p r e c g r i d <− l i n . m o d $ m a r g i n a l s . h y p e r p a r $
‘ P r e c i s i o n f o r the Gaussian observations ‘ [ , 1 ]
p r e c m a r g <− l i n . m o d $ m a r g i n a l s . h y p e r p a r $
‘ P r e c i s i o n f o r the Gaussian observations ‘ [ , 2 ]
p a r ( mfrow=c ( 2 , 2 ) )
p l o t ( b e t a x g m a r g ˜ b e t a x g g r i d , t y p e =” l ” , x l a b=e x p r e s s i o n ( b e t a [ g ] ) ,
y l a b =” M a r g i n a l D e n s i t y ” , c e x . l a b =1.5)
a b l i n e ( v =0 , c o l =” r e d ” )
p l o t ( b e t a x a m a r g ˜ b e t a x a g r i d , t y p e =” l ” , x l a b=e x p r e s s i o n ( b e t a [ a ] ) ,
y l a b =” M a r g i n a l D e n s i t y ” , c e x . l a b =1.5)
a b l i n e ( v =0 , c o l =” r e d ” )
p l o t ( b e t a x i n t m a r g ˜ b e t a x i n t g r i d , t y p e =” l ” , x l a b=e x p r e s s i o n ( b e t a [ i n t ] ) ,
y l a b =” M a r g i n a l D e n s i t y ” , c e x . l a b =1.5)
a b l i n e ( v =0 , c o l =” r e d ” )
p l o t ( p r e c m a r g ˜ p r e c g r i d , t y p e =” l ” , x l a b=e x p r e s s i o n ( s i g m a ˆ{ −2}) ,
y l a b =” M a r g i n a l D e n s i t y ” , c e x . l a b =1.5)
Introduction
Approximate Bayes
GLMs
Conclusions
References
−5
0
5
10
0.0 0.2 0.4 0.6 0.8 1.0
Marginal Density
0.00 0.05 0.10 0.15 0.20
Marginal Density
FTO Posterior Marginal Distributions
15
1
2
3
βg
4
5
3
2
0
1
Marginal Density
0.6
0.4
0.2
0.0
Marginal Density
4
βa
−2
−1
0
1
2
βint
3
4
5
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
σ−2
Marginal distributions of the three regression coefficients
corresponding to xg, xa and the interaction, and the precision.
Introduction
GLMs
Approximate Bayes
Conclusions
References
FTO Extended Analysis
• In order to carry out model checking we rerun the analysis, but now switch
on a flag to obtain fitted values.
l i n . mod <− i n l a ( l i n y ˜ l i n x g+l i n x a+l i n x i n t , d a t a=f t o d f ,
f a m i l y =” g a u s s i a n ” , c o n t r o l . p r e d i c t o r= l i s t ( compute=TRUE ) )
f i t t e d <− l i n . mod$summary . f i t t e d . v a l u e s [ , 1 ]
sigmamed <− 1/ s q r t ( . 2 9 2 4 )
• With the fitted values we can examine the fit of the model. In particular:
• Normality of the errors (sample size is relatively small).
• Errors have constant variance (and are uncorrelated).
• Linear model is adequate.
Introduction
GLMs
Approximate Bayes
Conclusions
Assessing the Model
r e s i d u a l s <− ( l i n y −f i t t e d ) / sigmamed
p a r ( mfrow=c ( 2 , 2 ) )
qqnorm ( r e s i d u a l s , main =””)
t i t l e (”( a )”)
a b l i n e ( 0 , 1 , l t y =2 , c o l =” r e d ” )
p l o t ( r e s i d u a l s ˜ l i n x a , y l a b =” R e s i d u a l s ” , x l a b =”Age ” )
t i t l e (”( b )”)
a b l i n e ( h=0 , l t y =2 , c o l =” r e d ” )
p l o t ( r e s i d u a l s ˜ f i t t e d , y l a b =” R e s i d u a l s ” , x l a b =” F i t t e d ” )
t i t l e (”( c )”)
a b l i n e ( h=0 , l t y =2 , c o l =” r e d ” )
p l o t ( f i t t e d ˜ l i n y , x l a b =”O b s e r v e d ” , y l a b =” F i t t e d ” )
t i t l e (”( d )”)
a b l i n e ( 0 , 1 , l t y =2 , c o l =” r e d ” )
References
Introduction
Approximate Bayes
GLMs
Conclusions
References
FTO Diagnostic Plots
(a)
(b)
●
●
1
● ●●
●
●
●
0
●●
●
●
●
●
●
●
●
−2
●
●
●
−1
0
1
2
3
4
(c)
(d)
1
●
0
●
●
●
●
●
●
●
●
●
●
●●
●
10
15
●
●
●
●●
5
−2
●
●
●
●
● ●
10
●
5
●
●
Fitted
●
●●
20
●
5
25
●
●
●
−1
2
Age
●
●
●
1
Theoretical Quantiles
15
−2
Residuals
●
●
−1
−1
●
●
●
●
●
● ●
●
●
●
Residuals
0
●
●
●
−2
Sample Quantiles
1
●
●
●● ●
●
●
20
25
5
10
Fitted
15
20
25
Observed
Plots to assess model adequacy: (a) Normal QQ plot, (b) residuals
versus age, (c) residuals versus fitted, (d) fitted versus observed.
Introduction
Approximate Bayes
GLMs
Conclusions
References
Bayes Logistic Regression
• The likelihood is
Y (x)|p(x) ∼ Binomial(N(x), p(x)),
• Logistic link:
log
• The prior is
„
p(x)
1 − p(x)
«
x = 0, 1, 2.
= α + θx
p(α, θ) = p(α) × p(θ)
with α ∼ N(µα , σα ) and θ ∼ N(µθ , σθ ).
• The first analysis uses the default priors in INLA (which are relatively flat).
• In the second analysis we specify α ∼ N(0, 1/0.1) and θ ∼ N(0, W ) where
W is such that the 97.5% point of the prior is log(1.5), i.e. we believe the
odds ratio lies between 2/3 and 3/2 with probability 0.95.
Introduction
GLMs
Approximate Bayes
Conclusions
References
R Code for Logistic Regression Example
x <− c ( 0 , 1 , 2 )
y <− c ( 6 , 8 , 7 5 )
z <− c ( 1 0 , 6 6 , 1 6 3 )
c c . d a t <− l i s t ( x=x , y=y , z=z )
c c . d a t <− a s . d a t a . f r a m e ( r b i n d ( y , z , x ) )
c c . mod <− i n l a ( y ˜ x , f a m i l y =” b i n o m i a l ” , d a t a=c c . dat , N t r i a l s=y+z )
summary ( c c . mod )
mean
sd
0.025 quant
0.5 quant 0.975 quant
( I n t e r c e p t ) −1.8069697 0 . 4 5 5 3 8 7 8 −2.749926947 −1.7902517 −0.9577791
x
0.4799929 0.2504718
0.008872592
0.4726171
0.9933651
# Now w i t h i n f o r m a t i v e p r i o r s
> Upper975 <− l o g ( 1 . 5 )
> W <− ( l o g ( Upper975 ) / 1 . 9 6 ) ˆ 2
> c c . mod2 <− i n l a ( y ˜ x , f a m i l y =” b i n o m i a l ” , d a t a=c c . dat , N t r i a l s=y+z ,
c o n t r o l . f i x e d= l i s t ( mean . i n t e r c e p t=c ( 0 ) , p r e c . i n t e r c e p t=c ( . 1 ) ,
mean=c ( 0 ) , p r e c=c ( 1 /W) ) )
> summary ( c c . mod2 )
mean
sd
0.025 quant
0.5 quant 0.975 quant
( I n t e r c e p t ) −1.5979411 0 . 3 8 7 4 1 5 8 −2.38646519 −1.5883333 −0.8630265
x
0 . 3 6 0 2 6 3 8 0 . 2 1 2 3 4 1 1 −0.04609342 0 . 3 5 6 4 4 7 3
0.7886091
>
>
>
>
>
>
>
The quantiles for θ can be translated to odds ratios by exponentiating.
Introduction
GLMs
Approximate Bayes
Conclusions
R Code For Extracting the Marginal Densities
• We obtain plots of the marginal distributions of α and θ, from the model
with informative priors.
• Alternatively, can use plot.inla.
a l p h a g r i d <− c c . m o d 2 $ m a r g i n a l s . f i x e d $ ‘ ( I n t e r c e p t ) ‘ [ , 1 ]
a l p h a m a r g <− c c . m o d 2 $ m a r g i n a l s . f i x e d $ ‘ ( I n t e r c e p t ) ‘ [ , 2 ]
t h e t a g r i d <− c c . m o d 2 $ m a r g i n a l s . f i x e d $ x [ , 1 ]
t h e t a m a r g <− c c . m o d 2 $ m a r g i n a l s . f i x e d $ x [ , 2 ]
p a r ( mfrow=c ( 1 , 2 ) )
p l o t ( a l p h a m a r g ˜ a l p h a g r i d , t y p e =” l ” , x l a b=e x p r e s s i o n ( a l p h a ) ,
y l a b =” M a r g i n a l D e n s i t y ” , c e x . l a b =1.5)
p l o t ( t h e t a m a r g ˜ t h e t a g r i d , t y p e =” l ” , x l a b=e x p r e s s i o n ( t h e t a ) ,
y l a b =” M a r g i n a l D e n s i t y ” , c e x . l a b =1.5)
a b l i n e ( v =0 , c o l =” r e d ” )
References
Introduction
Approximate Bayes
GLMs
Conclusions
References
0.0
0.0
0.2
0.5
1.0
Marginal Density
0.6
0.4
Marginal Density
0.8
1.5
1.0
Logistic Marginal Plots
−4
−3
−2
−1
0
−1.0
α
0.0
0.5
1.0
1.5
θ
Posterior marginals for the intercept α and the log odds ratio θ.
Introduction
Approximate Bayes
GLMs
Conclusions
A Simple ANOVA Example
• We begin with simulated data from the simple one-way ANOVA model
example:
Yij
=
�ij
∼iid
bi
∼iid
β0 + bi + �ij
N(0, σ�2 )
N(0, σb2 )
i = 1, ..., 10; j = 1, ..., 5, with β0 = 0.5, σ�2 = 0.22 and σb2 = 0.32 .
• Simulation:
s i g m a . b <− 0 . 3
s i g m a . e <− 0 . 2
m <− 10
n i <− 5
b e t a 0 <− 0 . 5
b <− rnorm (m, mean=0 , s d=s i g m a . b )
e <− rnorm (m∗ n i , mean=0 , s d=s i g m a . e )
Yvec <− b e t a 0 + r e p ( b , e a c h=n i ) + e
s i m d a t a <− d a t a . f r a m e ( y=Yvec , i n d=r e p ( 1 :m, e a c h=n i ) )
References
Introduction
GLMs
Approximate Bayes
Conclusions
References
A Simple ANOVA Example
> r e s u l t <− i n l a ( y ˜ f ( i n d , model=” i i d ” ) , d a t a = s i m d a t a )
> summary ( r e s u l t )
Fixed e f f e c t s :
mean
sd 0.025 quant
0.5 quant
0.975 quant
( I n t e r c e p t ) 0.3780418 0.09710096 0.1828018
0.3780368
0.5734883
Model h y p e r p a r a m e t e r s :
mean
sd
0.025 quant 0.5 quant 0.975 quant
P r e c i s i o n f o r Gaus o b s 2 2 . 8 1 6 4 . 9 4 3
14.439
22.389
33.755
Precision for ind
13.983 6.857
4.784
12.632
31.164
> s i g m a . e s t <− 1/ s q r t ( summary ( h y p e r ) $ h y p e r p a r [ , 4 ] )
> sigma . e s t
Gaus o b s
ind
# E x t r a c t the p o s t e r i o r medians
0.2119460
0.2795877
# o f t h e p r e c i s i o n and i n v e r t
Introduction
GLMs
Approximate Bayes
Conclusions
References
The Salamander Data: INLA Analysis
• We assume that for each of the random effects the residual odds lies
between 0.1 and 10 with probability 0.9, so that Ga(1,0.622) priors are
−2
−2
used for each of the six precisions, σfk
, σmk
for k = 1, 2, 3. See Fong
et al. (2010) for more details.
• We present results for the fixed effects and variance components, with the
model fitted using both restricted ML (REML) and INLA.
• We see the reduction in standard errors in the REML analysis.
• There is some attenuation of the Bayesian results here due to the INLA
approximation strategy. Again, see Fong et al. (2010) for more details.
Variable
Intercept
WSF
WSM
WSF × WSM
σf
σm
Experiment 1
REML
INLA
1.34±0.62
1.48±0.72
-2.94±0.88
-3.26±1.01
-0.42±0.63
-0.50±0.73
3.18±0.94
3.52±1.03
1.25�
1.29±0.46
0.27�
0.78±0.29
Experiment 2
REML
INLA
0.57±0.67
0.56±0.71
-2.46±0.93
-2.51±1.02
-0.77±0.72
-0.75±0.75
3.71±0.96
3.74±1.03
1.35�
1.38±0.50
0.96�
1.00±0.36
Experiment 3
REML
INLA
1.02±0.65
1.07±0.73
-3.23±0.83
-3.39±0.92
-0.82±0.86
-0.85±0.94
3.82±0.99
4.03±1.05
0.59�
0.80±0.28
1.36�
1.46±0.48
REML and INLA summaries for Salamander data. For the entries
marked with a � standard errors were unavailable.
Introduction
GLMs
Approximate Bayes
Conclusions
References
INLA Code for Salamander Data: Data Setup
## C r o s s e d Random E f f e c t s − S a l a m a n d e r
l o a d ( ” s a l a m . RData ” )
## o r g a n i z e d a t a i n t o a form s u i t a b l e f o r l o g i s t i c r e g r e s s i o n
d a t 0 <− d a t a . f r a m e ( ” y”=c ( s a l a m $ y ) ,
” f W”=a s . i n t e g e r ( s a l a m $ x [ , ”W/R”]==1 | s a l a m $ x [ , ”W/W”]==1) ,
”m W”=a s . i n t e g e r ( s a l a m $ x [ , ” R/W”]==1 | s a l a m $ x [ , ”W/W”]==1) ,
”W W”=a s . i n t e g e r ( s a l a m $ x [ , ”W/W”]==1 ) )
## add s a l a m a n d e r i d
i d <− t ( a p p l y ( s a l a m $ z , 1 , f u n c t i o n ( x ) {
tmp = w h i c h ( x==1)
tmp [ 2 ] = tmp [ 2 ] − 20
tmp
}) )
## i d s a r e s u i t a b l e f o r model A and C , b u t n o t B
i d . modA <− r b i n d ( i d , i d +40 , i d +20)
c o l n a m e s ( i d . modA) <− c ( ” f . modA” , ”m. modA” )
d a t 0 <− c b i n d ( dat0 , i d . modA , g r o u p =1)
d a t 0 $ e x p e r i m e n t <− a s . f a c t o r ( r e p ( 1 : 3 , e a c h =120))
d a t 0 $ g r o u p <− a s . f a c t o r ( d a t 0 $ g r o u p )
#
s a l a m a n d e r <− d a t 0
s a l a m a n d e r . e1 <− s u b s e t ( dat0 , d a t 0 $ e x p e r i m e n t ==1)
s a l a m a n d e r . e2 <− s u b s e t ( dat0 , d a t 0 $ e x p e r i m e n t ==2)
s a l a m a n d e r . e3 <− s u b s e t ( dat0 , d a t 0 $ e x p e r i m e n t ==3)
Introduction
GLMs
Approximate Bayes
Conclusions
INLA Code for Salamander Data: Model Fitting
• Below is the code for fitting to the data in experiment 1, along with the
output.
# s a l a m a n d e r . e1
> s a l a m a n d e r . e1 . i n l a . f i t <− i n l a ( y ˜ f .W + m.W + W.W +
f ( f . modA , model=” i i d ” , param=c ( 1 , . 6 2 2 ) ) +
f (m. modA , model=” i i d ” , param=c ( 1 , . 6 2 2 ) ) ,
f a m i l y =” b i n o m i a l ” , d a t a=s a l a m a n d e r . e1 ,
N t r i a l s=r e p ( 1 , nrow ( s a l a m a n d e r . e1 ) ) )
> s a l a m a n d e r . e1 . h y p e r p a r <− i n l a . h y p e r p a r ( s a l a m a n d e r . e1 . i n l a . f i t )
> summary ( s a l a m a n d e r . e1 . i n l a . f i t )
Fixed e f f e c t s :
mean
sd 0.025 quant
0.5 quant 0.975 quant
( Intercept )
1.4840357 0.7192119
0.1465009
1.4500754
3.007276
f .W
−3.2684715 1 . 0 0 9 0 1 5 7 −5.4481933 −3.2015287
−1.453743
m.W
−0.4971849 0 . 7 3 0 1 0 8 7 −1.9703752 −0.4870752
0.918794
W.W
3.5260067 1.0303279
1.6058935
3.4858297
5.672296
Random e f f e c t s :
Model h y p e r p a r a m e t e r s :
mean
sd
0.025 quant 0.5 quant 0.975 quant
P r e c i s i o n f o r f . modA 0 . 8 3 7 7 0 . 6 2 7 1 0 . 1 8 3 0
0.6505
2.6119
P r e c i s i o n f o r m. modA 2 . 3 4 8 2 1 . 7 2 7 0 0 . 4 6 9 1
1.8919
6.9513
References
Introduction
Approximate Bayes
GLMs
Conclusions
References
Approximate Bayes Inference
• Particularly in the context of a large number of experiments, a quick and
accurate model is desirable.
• We describe such a model in the context of a GWAS.
• We first recap the normal-normal Bayes model.
• Subsequently, we describe the approximation and provide an example.
Introduction
Approximate Bayes
GLMs
Conclusions
Recall: The Normal Posterior Distribution
For the model
• Prior: θ ∼ normal(µ0 , τ02 ) and
• Likelihood: Y1 , ..., Yn |θ ∼ normal(θ, σ 2 ).
Posterior
θ|y1 , ..., yn } ∼ normal(µ, τ 2 )
where
var(θ|y1 , ..., yn ) = τ 2
=
[1/τ02 + n/σ 2 ]−1
Precision = 1/τ 2
=
1/τ02 + n/σ 2
and
E[θ|y1 , ..., yn ] = µ
=
=
µ0 /τ02 + ȳ n/σ 2
1/τ02 + n/σ 2
„
«
„
«
1/τ02
n/σ 2
µ0
+ ȳ
1/τ02 + n/σ 2
1/τ02 + n/σ 2
References
Introduction
Approximate Bayes
GLMs
Conclusions
References
A Normal-Normal Approximate Bayes Model
• Consider again the logistic regression model
logit pi = α + xi θ
with interest focusing on θ.
• We require priors for α, θ, and some numerical/analytical technique for
estimation/Bayes factor calculation.
• Wakefield (2007, 2009) considered replacing the likelihood by the
approximation
where
b ∝ p(θ|θ)p(θ)
b
p(θ|θ)
b ∼ N(θ, V ) – the asymptotic distribution of the MLE,
• θ|θ
• θ ∼ N(0, W ) – the prior on the log RR. Can choose W so that 95% of
relative risks lie in some range, e.g. [2/3,1.5].
Introduction
GLMs
Approximate Bayes
Conclusions
Posterior Distribution
• Under the alternative the posterior distribution for the log odds ratio θ is
where
b rV )
θ|θb ∼ N(r θ,
W
.
V +W
• Hence, we have shrinkage to the prior mean of 0.
b and a 95% credible
• The posterior median for the odds ratio is exp(r θ)
interval is
√
exp(r θb ± 1.96 rV ).
r=
• Note that as W → ∞ the non-Bayesian point and interval estimates are
recovered.
References
Introduction
Approximate Bayes
GLMs
Conclusions
References
A Normal-Normal Approximate Bayes Model
• We are interested in the hypotheses: H0 : θ = 0,
evaluation of the Bayes factor
BF =
H1 : θ �= 0 and
b 0)
p(θ|H
.
b 1)
p(θ|H
• Using the approximate likelihood and normal prior we obtain:
„
«
1
Z2
Approximate Bayes Factor = √
exp − r ,
2
1−r
with Z =
Introduction
b
√θ
V
,r=
GLMs
W
V +W
.
Approximate Bayes
Conclusions
References
A Normal-Normal Approximate Bayes Model
• The approximation can be combined with a Prior Odds = (1 − π0 )/π0 to
give
BFDP
= ABF × Prior Odds
1 − BFDP
where BFDP is the Bayesian False Discovery Probability.
Posterior Odds on H0 =
• BFDP depends on the power, through r .
• For implementation,
all that we need from the data is the Z -score and the
√
standard error
V , or a confidence interval.
• Hence, published results that report confidence intervals can be converted
into Bayes factors for interpretation (see Lecture 8).
• The approximation relies on large sample sizes, so the normal distribution
of the estimator provides a good summary of the information in the data.
Introduction
GLMs
Approximate Bayes
Conclusions
References
Bayesian Logistic Regression: Estimation
> s o u r c e ( ” h t t p : / / f a c u l t y . w a s h i n g t o n . edu / j o n n o /BFDP . R” )
#
# 9 7 . 5 p o i n t o f p r i o r i s l o g ( 1 . 5 ) s o t h a t we w i t h p r o b
# 0 . 9 5 we t h i n k t h e t a l i e s i n ( 2 / 3 , 1 . 5 )
> Upper975 <− l o g ( 1 . 5 )
> W <− ( l o g ( Upper975 ) / 1 . 9 6 ) ˆ 2
> x <− c ( 0 , 1 , 2 )
> y <− c ( 6 , 8 , 7 5 )
> z <− c ( 1 0 , 6 6 , 1 6 3 )
> l o g i t m o d <− glm ( c b i n d ( y , z ) ˜ x , f a m i l y =” b i n o m i a l ” )
> t h e t a h a t <− l o g i t m o d $ c o e f [ 2 ]
> V <− v c o v ( l o g i t m o d ) [ 2 , 2 ]
> r <− W/ (V+W)
> r
[ 1 ] 0 . 7 7 1 7 7 1 8 # Not s o much d a t a h e r e , s o w e i g h t on p r i o r
# B a y e s i a n p o s t e r i o r median
> exp ( r ∗ t h e t a h a t )
x
1.446982
# S h r u n k t o w a r d s p r i o r median o f 1
# B a y e s i a n a p p r o x i m a t e 95% c r e d i b l e i n t e r v a l
> e x p ( r ∗ t h e t a h a t −1.96∗ s q r t ( r ∗V ) )
x
0.940091
> e x p ( r ∗ t h e t a h a t +1.96∗ s q r t ( r ∗V ) )
x
2.227186
Introduction
GLMs
Approximate Bayes
i s high .
Conclusions
Bayesian Logistic Regression: Hypothesis Testing
• Now we turn to testing using Bayes factors.
• We examine the sensitivity to the prior on the alternative, π1 .
> p i 1 <− c ( 1 / 2 , 1 / 1 0 0 , 1 / 1 0 0 0 , 1 / 1 0 0 0 0 , 1 / 1 0 0 0 0 0 )
> B F c a l l <− BFDPfunV ( t h e t a h a t , V ,W, p i 1 )
> BFcall
$BF
x
0.5110967
$pH0
x
0.256323
$pH1
x
0.5015156
$BFDP
[ 1 ] 0.3382290 0.9806196 0.9980453 0.9998044 0.9999804
• So data are twice as likely under the alternative as compared to the null.
• Apart from the 0.5 prior, under these priors the overall evidence is of no
association.
References
Introduction
Approximate Bayes
GLMs
Conclusions
References
Combination of data across studies
• Suppose we wish to combine data from two studies where we assume a
common log odds ratio θ..
√
• The √
estimates from the two studies are θb1 , θb2 with standard errors V 1
and V 2 .
• The Bayes factor is
p(θb1 , θb2 |H0 )
.
p(θb1 , θb2 |H1 )
• The approximate Bayes factor is
ABF(θb1 , θb2 ) = ABF(θb1 ) × ABF(θb2 |θb1 )
where
ABF(θb2 |θb1 ) =
and
(1)
p(θb2 |H0 )
p(θb2 |θb1 , H1 )
h
i
b
b
b
p(θ2 |θ1 , H1 ) = Eθ|θb1 p(θ2 |θ)
so that the density is averaged with respect to the posterior for θ.
• Important Point: The Bayes factors are not independent.
Introduction
Approximate Bayes
GLMs
Conclusions
References
Combination of data across studies
• This leads to an approximate Bayes factor (which summarizes the data
from the two studies) of
r

“
”ff
√
W
1
2
2
ABF(θb1 , θb2 ) =
exp −
Z1 RV2 + 2Z1 Z2 R V1 V2 + Z2 RV1
RV1 V2
2
where
• R = W /(V1 W + V2 W + V1 V2 )
b
• Z1 = √θ1
and
• Z2 = √
are the usual Z statistics.
V1
b
θ2
V2
• The ABF will be small (evidence for H1 ) when the absolute values of Z1
and Z2 are large and they are of the same sign.
Introduction
Approximate Bayes
GLMs
Conclusions
References
Example of Combination of Studies in a GWAS
• Frayling et al. (2007) report a GWAS for Type II diabetes.
• For SNP rs9939609:
Stage
1st
2nd
Combined
Estimate (CI)
1.27 (1.16–1.37)
1.15 (1.09–1.23)
–
p-value
6.4 × 10−10
4.6 × 10−5
–
− log10 BF
7.28
2.72
13.8
Pr(H0 |data)
1/5,000
0.00026
0.905
8 × 10−11
• Combined evidence is stronger than each separately since the point
with prior:
1/50,000
0.0026
0.990
8 × 10−10
estimates are in agreement.
• For summarizing inference the (5%, 50%, 95%) points for the RR are:
Prior
First Stage
Combined
Introduction
GLMs
1.00 (0.67–1.50)
1.26 (1.17–1.36)
1.21 (1.15–1.27)
Approximate Bayes
Conclusions
Conclusions
• Computationally GLMs and GLMMs can now be fitted in a relatively
straightforward way.
• INLA is very convenient and is being constantly improved.
• As with all analyses, it is crucial to check modeling assumptions (and
there are usually more in a Bayesian analysis).
• For binary observations INLA can produce inaccurate estimates. Markov
chain Monte Carlo provides an alternative for computation. WinBUGS is
one popular implementation.
• Other MCMC possibilities include: JAGS, BayesX.
References
Introduction
GLMs
Approximate Bayes
Conclusions
References
References
Breslow, N. and Clayton, D. (1993). Approximate inference in generalized linear
mixed models. Journal of the American Statistical Association, 88, 9–25.
Fong, Y., H.Rue, and Wakefield, J. (2010). Bayesian inference for generalized
linear mixed models. Biostatistics, 11, 397–412.
Frayling, T., Timpson, N., Weedon, M., Zeggini, E., Freathy, R., and et al.,
C. L. (2007). A common variant in the FTO gene is associated with body
mass index and predisposes to childhood and adult obesity. Science, 316,
889–894.
Karim, M. and Zeger, S. (1992). Generalized linear models with random
effects: Salamander mating revisited. Biometrics, 48, 631–644.
McCullagh, P. and Nelder, J. (1989). Generalized Linear Models, Second
Edition. Chapman and Hall, London.
Wakefield, J. (2007). A Bayesian measure of the probability of false discovery
in genetic epidemiology studies. American Journal of Human Genetics, 81,
208–227.
Wakefield, J. (2009). Bayes factors for genome-wide association studies:
comparison with p-values. Genetic Epidemiology , 33, 79–86.