Bayesian Cheat Sheets
July 22, 2026
Point Estimate
R
Posterior Mean: θ̂ = E(θ|y ) = θ θp(θ|y )dθ
R θ̂
Posterior Median:Choose θ̂ s.t. −∞ p(θ|y )dθ = 1
2
Posterior Mode:Choose θ̂ s.t. p θ̂|y
≥ p(θ|y )
∀θ
When P(θ|y ) is symmetric and unimodal, the 3 choices coincide.
Interval Estimate → posterior credible intervals 100(1 − α)%
h
i
”equal-tailed” posterior credible intervals (central intervals): θ ∈ θα/2 , θ1−α/2
”Highest Posterior Density”(HPD) region: s(y ) ⊂ Θ
1
Pr (θ ∈ s(y )|y ) = 1 − α
2
∀ θa ∈ s(y ) and θb ∈
/ s(y ), then p (θa |y ) > p θb |y
2 / 18
Beta (a(b): prior number of 1′ s (0′ s), a + b: prior sample size; Γ(n + 1) = n!)
θ ∼ B(a, b) =
Γ(a + b) a−1
b−1
θ
(1 − θ)
,
Γ(a)Γ(b)
(E, Var, Mode) :=
a
a+b
,
ab
(a + b)2 (a + b + 1)
,
!
a−1
(a − 1) + (b − 1)
Binomial model ((E, Var, Mode) := (nθ, nθ(1 − θ), ⌊(n + 1)θ⌋))
p(y |θ) ∝ θ y (1 − θ)n−y
p(θ) ∝ θ a−1 (1 − θ)b−1
p(θ|y ) ∝ θ a+y −1 (1 − θ)b+n−y −1
p(ỹ = 1|y )
↑
↑
↑
↑
Bionomial(y |θ, n)
B(θ|a, b)
B(θ|a + y , b + n − y )
E (θ|y )
Gamma and inverse gamma distributions
(
!
b a a−1 −bθ
a a
(a − 1)/b a ≥ 1
θ ∼ G(a, b) =
θ
e
,
(E, Var, Mode) :=
,
,
2
0
a<1
Γ(a)
b b
θ ∼ IG(a, b) =
ba
Γ(a)
θ
−(a+1) −b/θ
e
,
(E, Var, Mode) :=
b
a−1
a > 1,
b2
(a − 1)2 (a − 2)
a > 2,
b
!
a+1
Negative binomial distribution (x failures occur in a sequence of Bernoulli trials before n successes is reached (R version))
n+x −1 n
n(1 − p) n(1 − p)
p
x
x ∼ N B1 (n, p) =
p (1 − p) ,
(E, Var) :=
,
⇒
,
α = n, β =
x
p
p2
1−p
α x
x +α−1
β
1
α α(β + 1)
x ∼ N B2 (α, β) =
,
(E, Var) :=
,
α−1
β+1
β+1
β
β2
Poisson model (prior stats (E, Var, Mode) := (θ, θ, ⌊θ⌋))
p(y |θ) ∝ θ y e −θ
p(θ) ∝ θ a−1 e −bθ
↑
↑
P(y |θ)
G(θ|a, b)
p(yi |xi θ) ∝ θ yi e −xi θ
↑
p(θ) ∝ θ a−1 e −bθ
↑
p(θ|yy )
↑
G(θ|a + ny , b + n)
p(ỹ |yy )
↑
b+n
N B1 a + ny , b+n+1
p(θ|yy )
↑

n
n
X
X
G θ|a +
yi , b +
xi 
N B2 (a + ny , b + n)

P(yi |xi θ)
G(θ|a, b)
i=1
i=1
3 / 18
Prior predictive distribution for a single observation from Poisson distribution (a.2.7)
α x
P(x|θ)G(θ|α, β)
α+x −1
1
β
N B2 (x|α, β) =
=
G(θ|α + x, β + 1)
β+1
β+1
x
Means and variances of conditional distributions
E(u) = E(E(u|v )),
Var(u) = E(Var(u|v )) + Var(E(u|v ))
Jeffrey’s noninformative prior (invariant to parameterization)
#
" 2
" 2
d log p(y |θ)
d log p(y |ϕ = h−1 (θ))
1/2
p(θ) ∝ [J(θ)]
, J(θ) = −E
θ = −E
dθ 2
dϕ2
dϕ 2
dθ
#
θ
dϕ 2
= J(ϕ)
dθ
Normal with known variance
2
y |θ ∼ N (θ, σ ),
2
θ ∼ N (µ0 , τ0 ),
2
θ|yy ∼ N (µn , τn ),
2
2
ỹ |yy ∼ N (µn , τn + σ ),
µn =
1 µ + n y
0
σ2
τ2
0
,
1 + n
σ2
τ2
0
1
τn2
1
=
τ2
0
n
+
σ2
Scaled inverse-χ2 (different name of the same distribution as IG)
2
2
θ ∼ Inv-χ (ν, s ) =
(ν/2)ν/2 ν −(ν/2+1) −νs 2 /(2θ)
s θ
e
≡ IG
Γ(ν/2)
ν
2
,
νs 2
!
2
Normal with known mean
y |σ 2 ∼ N (θ, σ 2 )
ν =
n
1 X
2
(yi − θ)
n i=1
1
σ2
ν
∼ G( 20 ,
ν0 σ02
)
2
σ 2 ∼ Inv-χ2 (ν0 , σ02 )
2
ν +n ν σ +nν
1 |y
y ∼ G( 02 , 0 02
)
σ2
ν0 σ02 +nν
2 y
2
σ |y ∼ Inv-χ (ν0 + n, ν +n )
0
Normal-inverse-χ2 distribution
2
2
µ|σ ∼ N (µ0 , σ /κ0 ),
2
2
ν0 +1
+1
2
2
σ ∼ Inv-χ (ν0 , σ0 )
−
2
2
2
2
2
µ, σ ∼ N -Inv-χ (µ0 , σ0 /κ0 ; ν0 , σ0 ) ∝ σ
exp
⇒
−1 2
2
ν0 σ0 + κ0 (µ − µ0 )
2σ 2
4 / 18
Normal with unknown mean and variance (conjugate prior distribution)
µ|σ
2
σ
2
∼
Inv-χ (ν0 , σ0 )
µ|σ , y
∼
N (µn , σ /κn )
∼
2
N (µ0 , σ /κ0 )
2
µn =
2

2
σ |yy
2
∼
p(µ|yy )
=
2
2
2
Inv-χ (νn , σn )
=
=
N
κ0 µ0 + ny
νn
IG
0 + n
σ2
σ2
κ0 + n
2
,
νn σn2

2
2
2
νn σn = ν0 σ0 + (n − 1)s +
!
2
∼
2
p(σ |yy )
=
2
2
ỹy |yy ∼ tνn (µn 1m , σn M)
M = Im +
2
N (µ0 , τ0 ),
2
2
σ ∼ Inv-χ (ν0 , σ0 ),
2
∝
p(µ|σ 2 , y )
N (µ|µn , τn2 )
1
2
2
2
2
N (µn |µ0 , τ0 )Inv-χ (σ |ν0 , σ0 )
n
Y
2
N (yi |µn , σ )
2
y , σn
p(µ|yy ) = tn−1
2
y , sn
p(ỹ |yy ) = tn−1 y , 1 + n1 s 2
(n−1)s 2
σ 2 |yy ∼ IG n2 ,
2
5 / 18
σ 2 |yy ∼ Inv-χ2 (n − 1, s 2 )
µ|σ 2 , y ∼ N
1 + n
σ2
τ2
0
i=1
Normal with noninformative prior (1st from book [1], 2nd use conjugate prior formula)
2
(n−1)s 2
p(µ, σ 2 ) ∝ (σ 2 )−1 µ|σ 2 , y ∼ N y , σn
σ 2 |yy ∼ IG n−1
,
2
2
κ0 , ν0 → 0
1
2
τn =
i=1
τn
T
1m 1m
1 µ + n y
0
σ2
τ2
0
1 + n
σ2
τ2
0
µn =
∝
1
κn
2
µ|σ , y ∼ N (µn , τn )
p(µ, σ 2 |yy )
n
Y
2
N (µ|µ0 , τ02 )Inv-χ2 (σ 2 |ν0 , σ02 )
N (yi |µ, σ )
µ=µn
νn = ν0 + n
κ0 n
2
(y − µ0 )
κn
2
2
tνn (µ|µn , σn /κn )
1
2
ỹ |yy
∼
t νn µ n , 1 +
σn
κn
Normal with a semi-conjugate prior distribution
µ|σ
n
y
κ0 + n
κn = κ0 + n
µ0 +

1
, κ
κ0
κ0 + n
T distribution (a. Appendix A)
"
#− ν+1
Γ ν+1
2
1
1 (θ − µ)2
2
2
θ ∼ tν µ, σ
1+
= √
,
ν
σ2
Γ ν2
νπ σ
ν+k
2
ν
2
Γ
µ , Σ) =
θ ∼ tν (µ
Γ
k
(νπ) 2
|Σ|
−1
2
(E, Var, Mode) :=
µ (ν > 1),
− ν+k
1
2
θ − µ )T Σ−1 (θ
θ − µ)
1+
(θ
, (E, Var, Mode) :=
ν
Multivariate normal
−k
−1
2 |Σ| 2 exp
µ, Σ) = (2π)
y ∼ Nk (µ

ν
µ (ν > 1),
2
ν−2
σ (ν > 2), µ
ν
!
Σ (ν > 2), µ
ν−2
1
T −1
− (yy − µ ) Σ (yy − µ )
2

n
1 −1 1 X
−n
−n
T −1
2 exp −
(yy i − µ ) Σ (yy i − µ ) = |Σ| 2 exp − tr Σ S0
µ, Σ) ∝ |Σ|
p(yy 1 , y 2 , . . . , y n |µ
S0 =
n
X
2 i=1
T
(yy i − µ )(yy i − µ ) =
i=1
2
n
X
T
T
δδ T
(yy i − y )(yy i − y ) + n(yy − µ )(yy − µ ) ≜ Q0 + nδ
i=1
− n−1
2 exp
µ, Σ) ∝ |Σ|
p(yy 1 , y 2 , . . . , y n |µ
1 −1 n
−1
T −1
× |Σ| 2 exp − (yy − µ ) Σ (yy − µ )
− tr Σ Q0
2
2
Wishart & inverse Wishart distribution (W → precision, IW → variance)
 T


y1
y11
y12
...
y1k
y T 
y21
y22
...
y2k 
 


 2
Y = . =
.
. 
0k , Σ)
y i ∼ Nk (0
..
 ..
 . 
.
. 
 .
.
 . 
.
. 
yT
n
2
Wii ∼ Σii χn
Wj ∼ Wk (nj , Σ) →
m
X
Wj ∼ Wk (
j=i
n−k−1
−n
2 |W |
2
exp
p(W ) ∝ |Σ|
m
X
yn1
...
ynk
V =W
nj , Σ)
j=1
yn2
1 −1 − tr Σ W
2
−1
T
W = Y Y ∼ Wk (n, Σ)
E(W ) = nΣ
T
T
B WB ∼ W(n, B ΣB)
−1/2 T
−1/2
Σ
WΣ
∼ W(n, I )
−1
∼ IW(n, Σ
),
−1
E(V ) = (n − k − 1)
n
− n+k+1
2
exp
p(V ) ∝ |Σ| 2 |V |
Σ
1 −1
− tr ΣV
2
6 / 18
Normal-inverse-Wishart distribution
−1
Σ ∼ IW(ν0 , Λ0 ),
µ0 , Σ/κ0 )
µ |Σ ∼ N (µ
⇒
ν +k
κ0
1
− 0
−1
−1
2
µ, Σ) ∝ |Σ|
µ − µ 0 )′ Σ−1 (µ
µ − µ0 )
p(µ
(µ
exp − tr(Λ0 Σ ) −
2
2
Multivariate normal with known covariance matrix
µ
∼
µ0 , Λ0 )
Nk (µ
1
µ − µ0 )T Λ−1
µ − µ0 )
− (µ
0 (µ
2
1 T −1
T −1
∝
exp − µ Λ0 µ + µ Λ0 µ 0
2
1 T
T
−1
−1
=
exp − µ A0µ + µ b 0
(A0 = Λ0 , b 0 = Λ0 µ 0 )
2


n
 1 X

−nk/2
−n/2
T −1
µ, Σ)
p(yy 1 , y 2 , . . . , y n |µ
=
(2π)
|Σ|
exp −
(yy i − µ ) Σ (yy i − µ )
 2

i=1
1 T
−1
−1
T
∝
exp − µ A1µ + µ b 1
(A1 = nΣ , b 1 = nΣ y )
2
1 T
1 T
T
T
µ|yy 1 , y 2 , . . . , y n , Σ)
p(µ
∝
exp − µ A0µ + µ b 0 × exp − µ A1µ + µ b 1
2
2
1 T
T
=
exp − µ An µ + µ b n
(An = A0 + A1 , b n = b 0 + b 1 )
2
µ)
p(µ
=
−k/2
(2π)
−1/2
|Λ0 |
exp
µ|yy 1 , y 2 , . . . , y n , Σ) = Nk (µ
µ n , Λn )
p(µ
−1
−1 −1 −1
−1
µ n = (Λ0 + nΣ ) (Λ0 µ 0 + nΣ y ),
−1
Λn = (Λ0
−1 −1
+ nΣ
)
µn , Σ + Λn )
p(ỹy |yy ) = Nk (µ
µ, y )|yy ) = E(µ
µ|yy ) = µ n ,
E(ỹy |yy ) = E(E(ỹy |µ,
µ, y )|yy )+Var(E(ỹy |µ,
µ, y )|yy ) = E(Σ|yy )+Var(µ
µ|yy ) = Σ+Λn
Var(ỹy |yy ) = E(Var(ỹy |µ,
7 / 18
Multivariate normal with known mean vector
−1
Σ ∼ IW(ν0 , S0
p(Σ) ∝ |Σ|
),
−n
2 exp
µ, Σ) ∝ |Σ|
p(yy 1 , y 2 , . . . , y n |µ
p(Σ|yy 1 , y 2 , . . . , y n , µ )
Σ|yy 1 , y 2 , . . . , y n , µ
E(Σ|yy 1 , y 2 , . . . , y n , µ )
=
=
ν +k+1
1 − 0
−1
2
exp − tr S0 Σ
2
1 −1 − tr Σ Sµ
2
n
X
(Sµ =
T
(yy i − µ )(yy i − µ ) )
i=1
∝
µ, Σ)
p(Σ) × p(yy 1 , y 2 , . . . , y n |µ
∝
|Σ|
∼
IW(ν0 + n, [S0 + Sµ ]
ν +n+k+1
1 −1
− 0
2
exp − tr Σ [S0 + Sµ ]
2
−1
1
ν0 + n − k − 1
ν0 − k − 1
)
(S0 + Sµ )
1
ν0 + n − k − 1 ν0 − k − 1
S0 +
n
1
ν0 + n − k − 1 n
Sµ
If we are confident that the true covariance matrix is near some covariance matrix Σ0 , then we might choose ν0 to be large and
set S0 = (ν0 − k − 1)Σ0 , making the distribution of Σ concentrated around Σ0 . On the other hand, choosing ν0 = k + 2 and
S0 = Σ0 makes Σ only loosely centered around Σ0 .
8 / 18
Multivariate normal with unknown mean and variance
−1
Σ
∼
IW(ν0 , Λ0 )
µ |Σ
∼
µ0 , Σ/κ0 )
N (µ
µ, Σ)
p(µ
∝
|Σ|
−
exp
µn =
ν0 +k
+1
2
×
κ0
−1
µ − µ0 )′ Σ−1 (µ
µ − µ0 )
− tr Σ Λ0 −
(µ
2
2
κ0
κ0 + n
n
y
κ0 + n
κn = κ 0 + n
µ0 +
1
νn = ν0 + n
n
X
′
S =
(yy i − ȳ )(yy i − ȳ )
−1
Σ|y
∼
IW(νn , Λn )
µ |Σ, y
∼
µn , Σ/κn )
N (µ
µ|y )
p(µ
=
tνn −k+1
µn ,
i=1
Λn = Λ0 + S +
!
Λn
κ0 n
κ0 + n
′
(yy − µ 0 )(yy − µ 0 )
κn (νn − k + 1)
Noninformative prior distribution (Jeffreys prior)
µ, Σ) ∝ |Σ|−(k+1)/2
p(µ
Σ|y ∼ IW(n − 1, S
−1
)
(κ0 → 0, ν0 → −1, |Λ0 | → 0)
µ |Σ, y ∼ N (yy , Σ/n)
µ |y ∼ tn−k
y,
S
!
n(n − k)
9 / 18
Hierarchical Model (Book1 5.4)
2
2
yij |θj ∼ N (θj , σ ),
2
i = 1, . . . , nj , j = 1, . . . , J,
ȳ.j |θj ∼ N (θj , σj = σ /nj )
known variances
2
2
2
2
θj |µ, τ ∼ N (µ, τ ),
p µ, τ
= p(µ)p τ
,
p(µ) ∝ 1, p(τ ) ∝ 1
θ , µ, τ |yy )
p(θ
∝
θ |µ, τ )p(yy |θ
θ)
p(µ, τ )p(θ
∝
p(µ, τ )
J
Y
N
J
Y
2
2
θj |µ, τ
N ȳ.j |θj , σj
j=1

θj |µ, τ, y
p(µ, τ |yy )
∝
∼
N θ̂j =
ȳ.j /σj2 + µ/τ 2
1/σj2 + 1/τ 2
p(µ, τ )p(yy |µ, τ )
∝
j=1
, Vj =
p(µ, τ )
J
Y

1
1/σj2 + 1/τ 2
N

2
2
ȳ.j |µ, σj + τ
j=1

p(µ, τ |yy ) = p(µ|τ, y )p(τ |yy ),
J
ȳ.j
P

2
2

1
j=1 σj +τ
µ|τ, y ∼ N 
, Vµ =
µ̂ = P
J
J
P

1
1
2
2
j=1 σj +τ
p(τ |yy ) =
p(µ, τ |yy )
p(µ|τ, y )
p(τ )
=∝
2
2
j=1 σj +τ








2
2
J Y
ȳ.j − µ̂ 2
j=1 N ȳ.j |µ, σj + τ
µ=µ̂
1/2
2
2 −1/2

∝ p(τ )Vµ
σj + τ
exp − N µ|µ̂, Vµ
2 σ2 + τ 2
j=1
QJ
j

ȳ.j /σj2 + µ/τ 2
E(θj |τ, y ) = E[E(θj |µ, τ, y )|τ, y ] = E 
1/σj2 + 1/τ 2
Var(θj |τ, y ) = E[Var(θj |µ, τ, y )|τ, y ] + Var[E(θj |µ, τ, y )|τ, y ] =

ȳ.j /σj2 + µ̂/τ 2
τ, y  =
1/σj2 + 1/τ 2
1
1/σj2 + 1/τ 2

+
1/τ 2
1/σj2 + 1/τ 2
2
 Vµ
10 / 18
The Hierarchical Normal Model – Gibbs Sampling (Book2 8.3)
µ, τ 2
2
θ1
θ2
...
θm−1 θm
Y1
Y2
...
Ym−1 Ym
θj |µ, τ ∼ N
µ∼N
2
2
2
µ0 , γ0
1
,
τ2
2
µ, τ
,
2
yj |θj , σ ∼ N
η0
∼G
2
,
η0 τ02
!
1
,
2
2
θj , σ
σ2
∼G
ν0
2
,
ν0 σ02
!
2
σ2
2
2
p(θ1 , . . . , θm , µ, τ , σ |yy 1 , . . . , y m )
∝
=
2
2
n
o
2
µ|θ1 , . . . , θm , τ
∼N
n
2
1/τ |θ1 , . . . , θm , µ
θj |y1j , . . . , yn j , µ, σ
2
o
j

n
2
θ, y 1 , . . . , y m
1/σ |θ
o
2
j=1 i=1
j=1
n
2
p(µ, τ , σ )p(θ1 , . . . , θm |µ, τ , σ )p(yy 1 , . . . , y m |θ1 , . . . , θm , µ, τ , σ )



nj
m Y
m

Y
 Y
2
2
2
2
p(yij |θj , σ )
p(µ)p(τ )p(σ )
p(θj |µ, τ )



∼ G
o
i
mθ̄/τ 2 + µ0 /γ02 h
2
2 −1
, m/τ + 1/γ0
m/τ 2 + 1/γ02
η0 + m
∼G
2
,
η0 τ02 +
1
2
!
θj − µ 2
P
2
2
∼N
!
2
i
nj ȳj /σ + µ/τ h
2
2 −1
, nj /σ + 1/τ
nj /σ 2 + 1/τ 2

ν0 +
m
X
j=1

nj  ,
1
2

ν0 σ 2 +
0
!
n
j
m X
X

yij − θj
2

j=1 i=1
11 / 18
The Hierarchical
Normal Model (Cont)
2
µ, τ
2
θ1
θ2
...
Y1
Y2
...
Ym−1 Ym
σ12
σ22
...
2
σm−1
θj |µ, τ ∼ N
θm−1 θm
µ∼N
2
σm
2
η0
∼G
τ2
2
2
2
,
,
η0 τ02
ν0 σ02
2
!
2
yj |θj , σj ∼ N
,
2
θj , σj
!
,
2
2
2
2
σ0 ∼ G(a, b),
ν0 ∝ e
−αν0
2
p(θ1 , . . . , θm , µ, τ , σ1 , . . . , σm , ν0 , σ0 |yy 1 , . . . , y m )
2
2
2
2
2
2
j=1
j=1
µ|θ1 , . . . , θm , τ
n
2
o
2
2
2
∼N
2
1/τ |θ1 , . . . , θm , µ
o
j=1 i=1
i
mθ̄/τ 2 + µ0 /γ02 h
2
2 −1
, m/τ + 1/γ0
m/τ 2 + 1/γ02
∼G
η0 + m
2
η0 τ02 +
,
!
2
2

o
i
n
nj ȳj /σj2 + µ/τ 2 h
2
2
2 −1 
θj |y1j , . . . , yn j , µ, σj ∼ N 
, nj /σj + 1/τ
j
nj /σj2 + 1/τ 2

2
1/σj |θj , y1j , . . . , yn j ,
j
o
∼ G
ν 0 + nj
2

n
2
2
2
σ0 |σ1 , . . . , σm , ν0
o
∼ G a +
,
1
2
mν0
2

nj
ν0 σ 2 +
0
X

yij − θj
2
2
p(ν0 |σ0 , σ1 , . . . , σm ) ∝
!
θj − µ 2
P

n
ν0
∼G
p(µ, τ , ν0 , σ0 )p(θ1 , . . . , θm , σ1 , . . . , σm |µ, τ , ν0 , σ0 )p(yy 1 , . . . , y m |θ1 , . . . , θm , µ, τ , σ1 , . . . , σm , ν0 , σ0 )




nj
m
m Y
m
 Y

Y
 Y
2
2
2
2
2
2
p(σj |ν0 , σ0 )
p(yij |θj , σj )
p(µ)p(τ )p(ν0 )p(σ0 )
p(θj |µ, τ )




=
n
σj2
1
2
µ 0 , γ0 ,
ν0 , σ02
∝
1
2
µ, τ
,
2


ν /2 m 
ν /2+1
0
m
ν0 σ02 /2 0

 Y 1 


2
Γ (ν0 /2)
σ
j=1 j
exp



m


σ2 X
1

−ν0 α + 0

2
σ2 
j=1
j
i=1
,b +
m
ν0 X
1

2 j=1 σj2

12 / 18
Linear regression
β , σ 2 ∼ N (Xβ
β , σ 2 I ))
Noninformative prior (yy |β
2
−2
p β, σ
∼σ ,
2
σ |yy ∼ IG
n−k
2
,
(n − k)s 2
!
2
,
−1
′
′
X X
X y,
β , σ 2 Vβ ,
β̂
β =
β̂
p β , σ 2 |yy
2
p σ |yy =
,
p β |σ 2 , y
s =
2
β |σ , y ∼ N
2
Vβ =
−1
′
X X
′ β
β
y − X β̂
y − X β̂
n−k
β |yy ∼ tn−k
,
β , s 2 Vβ
β̂
2
β , σ 2 , y )|σ 2 , y ] + Var[E(ỹy |β
β , σ 2 , y )|σ 2 , y ] = E[σ 2 I |σ 2 , y ] + Var[X̃β
β |σ 2 , y ] = σ 2 (I + X̃ Vβ X̃ ′ )
Var(ỹy |σ , y ) = E[Var(ỹy |β
2
β , σ 2 , y )|σ 2 , y ] = E(X̃β
β |σ 2 , y ) = X̃ β̂
β,
β , s 2 (I + X̃ Vβ X̃ ′ )
E(ỹy |σ , y ) = E[E(ỹy |β
ỹy |yy ∼ tn−k X̃ β̂
β , σ 2 ∼ N (Xβ
β , σ 2 Qy ))
Noninformative prior (yy |β
2
−2
p β, σ
∼σ ,
2
β |σ , y ∼ N
2
n−k
σ |yy ∼ IG
Noninformative prior
2
β , Vβ σ 2 ,
β̂
,
(n − k)s 2
2
β =
β̂
−1
′ −1
′ −1
X Qy X
X Qy y ,
!
,
2
s =
Vβ =
−1
′ −1
X Qy X
′
β Qy−1 y − X β̂
β
y − X β̂
n−k
Qn
β , σ , ϕ ∼ N (Xβ
β , σ diag(w1−ϕ , . . . , wn−ϕ )),
(yy |β
i=1 wi = 1)
2
2
2
−2
p β , σ |ϕ ∼ σ ,
β , σ 2 , ϕ|yy ) ∝ p(ϕ)p(β
β , σ 2 |ϕ)
p(β
n
Y
2
−ϕ
β ) i , σ wi
N (yi |(Xβ
)
i=1
p(ϕ|yy ) =
β , σ 2 , ϕ|yy )
p(β
β , σ 2 |ϕ, y )
p(β
∝
Q
−ϕ
p(ϕ)σ −2 ni=1 N (yi |Xi β , σ 2 wi
)
β ,s 2
β̂
1/2 2 −(n−k)/2
∝ p(ϕ)|Vβ |
(s )
n−k (n−k)s 2
2
2
β |β̂
β , Vβ σ )
IG σ | 2 ,
N (β
2
13 / 18
Linear regression (cont)
β , σ12 , . . . , σI2 ∼ N (Xβ
β , diag(σ12 I1 , . . . , σI2 II )))
Noninformative prior (yy |β
β , Σy ) ∝
p(β

I
Y
−2
σi ,
β , σ12 , . . . , σI2 |yy ) ∝ 
p(β
i=1
1/2
p(Σy |yy ) ∝ p(Σy )|Vβ |
|Σy |
−1/2

I
Y
1
−n −2
 exp − (yy − Xβ
β )′ Σ−1
y − Xβ
β)
σi i
y (y
2
i=1
exp
1
β )′ Σ−1
y − X β̂
β)
− (yy − X β̂
y (y
2
Semi-conjugate prior distribution
ν0
2
β ∼ N (β
β 0 , Σ0 ),
σ ∼ IG
2
,
ν0 σ02
!
2
−1
1 ′
−1
Λ = Σ0 +
X X
,
2
σ
Unit information prior:
β |σ 2 , y ∼ N (µ
µ, Λ),
,
µ =Λ
−1
Σ0 β 0 +
−1
′
Σ0
2
1
′
σ2
= (X X )/(nσ )
X y
ν0 + n
2
β , y ∼ IG
σ |β
2
,
s =
β ols
β 0 = β̂
2
,
ν0 σ02 + ns 2
!
2
β )′ (yy − Xβ
β)
(yy − Xβ
n
2
ν0 = 1
2
σ0 = σ̂ols
g-prior distribution
β ∼N
2
′
−1
0 , g σ (X X )
,
2
2
2
p(σ |yy ) ∝ p(σ )p(yy |σ ),
2
σ |yy ∼ IG
2
σ ∼ IG
2
p(yy |σ ) =
ν0 + n
2
,
ν0
2
Z
,
ν0 σ02
2
,
2
β |σ , y ∼ N
g
g +1
′
−1
(X X )
g
′
X y,
−n
−k
−n
2 (1 + g ) 2 (σ 2 ) 2 exp
!
,
SSRg = y
′
I −
g
g +1
′
−1
X (X X )
X
′
2
g +1
β , σ 2 )p(β
β |σ 2 )dβ
β = (2π)
p(yy |β
ν0 σ02 + SSRg
2
!
′
−1
σ (X X )
SSRg
−
2
2σ
y
14 / 18
Bayesian model comparison
odds(zz a , z b |yy )
yi = z1 β1 xi,1 + . . . + zk βk xi,k + ϵi
p(zz a |yy )
p(zz b |yy )
=
p(zz a )
p(zz b )
×
p(yy |zz a )
p(yy |zz b )
posterior odds
=
prior odds
×
Bayes factor
=
zj ∈ {0, 1}
2
β z |zz , σ ∼ N
Z Z
p(yy |zz )
=
Z
=
=
2
β |zz , σ 2 )dβ
β
p(yy |zz , β , σ )p(β
2 ′
−1
0 , g σ [Xz Xz ]
2
Z
2
p(σ )dσ =

γSSRgz
k
n −
−n
− z
×
2
(2π) 2 (1 + g ) 2 γ 2 e

2
2
2
p(yy |zz , σ )p(σ )dσ =
! ν0
2
ν0 σ0
2
Γ
2
ν0
−1
2

ν0
Z
p(yy |zz , γ)p(γ)dγ
γ 2 −1 e −
γν0 σ02
 dγ
2

(ν0 σ02 )ν0 /2
− n Γ((ν0 + n)/2)
−k /2
(1 + g ) z
(2π) 2
Γ(ν0 /2)
(ν0 σ02 + SSRgz )(ν0 +n)/2

1/2

z (n+1)/2
2
sz2 + SSRgb
(k −kza )/2  sza 

= (1 + n) zb
× b
z
p(yy |zz b )
sz2
sz2 + SSRga
p(yy |zz a )
p(zj = 1|yy , z −j ) =
oj
1 + oj
,
2
2
(g = n, ν0 = 1, σ0 = s )
a
b
oj =
p(zj = 1|yy , z −j )
p(zj = 0|yy , z −j )
=
p(zj = 1)
p(zj = 0)
×
p(yy |zj = 1, z −j )
p(yy |zj = 0, z −j )
Sampling procedure
1 Set z = z (s) ;
2 For j ∈ {1, . . . , p} in random order, replace zj with a sample from p(zj |zz −j , y );
3 Set z (s+1) = z ;
4 Sample σ 2(s+1) ∼ p(σ 2 |zz (s+1) , y );
β |zz (s+1) , σ 2(s+1) , y );
5 Sample β (s+1) ∼ p(β
15 / 18
Markov Chain Monte Carlo (MCMC)
Metropolis-Hastings algorithm (special cases: Metropolis algorithm, Gibbs sampler)
1 Sample θ ∗ ∼ Js (θ|θ (s) )
Gibbs sampler
2 Compute the acceptance ratio
r =
p(θ ∗ |yy )Js (θ (s) |θ ∗ )
p(θ (s) |yy )Js (θ ∗ |θ (s) )
=
θ ∗ |θ
θ (s) ) =
Jj,s (θ
p(yy |θ ∗ )p(θ ∗ )Js (θ (s) |θ ∗ )
p(yy |θ (s) )p(θ (s) )Js (θ ∗ |θ (s) )
r
(
(s)
(s)
θ∗
θ −j , y )
p(θ
j |θ
if θ ∗
−j = θ −j
0
otherwise
θ ∗ |yy )/Jj,s (θ
θ ∗ |θ
θ (s) )
p(θ
=
θ (s) |yy )/Jj,s (θ
θ (s) |θ
θ∗ )
p(θ
3 Generate u ∼ U (0, 1) and set
(s)
(
θ
(s+1)
=
θ∗
θ (s)
θ ∗ |yy )/p(θ
θ∗
θ −j , y )
p(θ
j |θ
=
if u ≤ min(r , 1)
otherwise
(s) (s)
θ (s) |yy )/p(θ
θ j |θ
θ −j , y )
p(θ
(s)
θ −j |yy )
p(θ
=
Metropolis algorithm: use symmetric proposal distribution
(s)
=1
θ −j |yy )
p(θ
Js (θ ∗ |θ (s) ) = Js (θ (s) |θ ∗ )
Given stationary distribution p0 (θ) and p0 (θa )Js (θb |θa ) ≥ p0 (θb )Js (θa |θb )
Pr (θ
(s)
= θa , θ
(s+1)
Pr (θ
(s)
Pr (θ
= θb ) = p0 (θa )Js (θb |θa )
= θb , θ
(s+1)
(s+1)
= θ) =
p0 (θb )Js (θa |θb )
p0 (θa )Js (θb |θa )
= p0 (θb )Js (θa |θb )
Pr (θ
⇒
= θa ) = p0 (θb )Js (θa |θb )
X
θa
Pr (θ
(s+1)
= θ, θ
(s)
= θa ) =
(s)
Pr (θ
X
Pr (θ
(s+1)
= θa , θ
(s)
= θa , θ
(s)
(s+1)
= θb , θ
= θ) = Pr (θ
(s)
= θb ) =
(s+1)
= θa )
= θ)
θa
Irreducible, aperiodic and recurrent Markov Chain converges to desired stationary distribution.
r ↑ =⇒ move slowly and have high correlation; r ↓ =⇒ ”stuck” to the posterior modes and have low correlation.
Recommended rejections rates around 80% (when altering a vector of parameters) and 60% (when altering one parameter at a
time).
16 / 18
MCMC (inference and assessing convergence), DIC
Let ψij (i = 1 : n, j = 1 : m) are values from m chain after removing the burn in part and thinning the remaining
values.(neff :effective sample size, R̂ : potential scale reduction factor.)
B =
n
m
X
m − 1 j=1
ψ̄.j − ψ̄..
2
,
W =


X n−t
σ2
2σ 2 n−1

var ψ̄.. =
+
ρt  ,
mn
mn
n
t=1
m
1 X
2
sj ,
m j=1
c + (ψ|yy ) =
var
lim mn var ψ̄.. =
n→∞
n−1
1+2
n
∞
X
W +
1
n
s
B,
R̂ =
c + (ψ|yy )
var
W
!
ρt
var(ψ|yy ),
neff =
mn
∞
P
1+2
t=1
ρt
t=1
Recommended stopping rule: R̂ ≤ 1.1, n̂eff ≥ 10m.
Deviance Information Criterion (DIC = ’goodness of fit’ + ’complexity’), effective number of parameters (pD )
θ ) = −2 log p(yy |θ
θ ),
D(θ
θ ]) = D̄ − D(θ̄
θ ),
pD = Eθ |yy [D] − D(Eθ |yy [θ
θ ) ≈ D(θ̄
θ ) − (θ
θ − θ̄
θ )′
D(θ
(2)
pD =
1
2
d(D(θ
θ )) =
Var
∂2
θ2
∂θ
1
θ ) + 2pD
DIC = D̄ + pD = D(θ̄
!
θ − θ̄
θ ) ≈ D(θ̄
θ ) + χ2k
(θ
θ)
log p(yy |θ
θ
θ̄
1
L 2
X
l
D(yy , θ ) − D̂avg (yy )
2 L − 1 l=1
(2)
pD is the preferred one. pD is a number of ’unconstrained’ parameters, a parameter will count as:
1 if it is completely unconstrained (no information about it in the prior);
0 if it is completely constrained (all information about if is in the prior);
intermediate if information about it comes from the data and the prior.
17 / 18
Reference and useful web links
[1] Gelman A, Carlin JB. Bayesian Data Analysis (3rd Edition). Chapman and Hall/CRC, Boca Raton, FL, 2013.
[2] Jeffrey W. Miller Bayesian Methodology in Biostatistics. (Course material for BIOSTAT 249), Harvard University,
2021.
[3] Aki Vehtari Bayesian Data Analysis course
[4] Hoff PD. A First Course in Bayesian Statistical Methods. Springer, 2009.
18 / 18