Geometry of
Sensitivity
analysis
Paul Marriott
Objectives
SEM
Geometries
Computational Information Geometry:
Pain Relief
Example
Geometry of Model Choice
Least
informative
model
Paul Marriott
University of Waterloo
Information Geometry and its Applications III
Leipzig August 2010
Geometry of
Sensitivity
analysis
Big Picture
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Introduction to Computational Information Geometry
Geometry of
Sensitivity
analysis
Big Picture
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Introduction to Computational Information Geometry
• The overall objective is to construct diagnostic tools to
help understand sensitivity to model choice
Geometry of
Sensitivity
analysis
Big Picture
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Introduction to Computational Information Geometry
• The overall objective is to construct diagnostic tools to
help understand sensitivity to model choice
• Targeted at applications where Generalised Linear
Models are used
Geometry of
Sensitivity
analysis
Big Picture
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Introduction to Computational Information Geometry
• The overall objective is to construct diagnostic tools to
help understand sensitivity to model choice
• Targeted at applications where Generalised Linear
Models are used
• Our geometry is affine and convex, not manifold based:
non-constant support and moment structure
Geometry of
Sensitivity
analysis
Big Picture
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Introduction to Computational Information Geometry
• The overall objective is to construct diagnostic tools to
help understand sensitivity to model choice
• Targeted at applications where Generalised Linear
Models are used
• Our geometry is affine and convex, not manifold based:
non-constant support and moment structure
• Topology defined via duality
Geometry of
Sensitivity
analysis
Big Picture
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Introduction to Computational Information Geometry
• The overall objective is to construct diagnostic tools to
help understand sensitivity to model choice
• Targeted at applications where Generalised Linear
Models are used
• Our geometry is affine and convex, not manifold based:
non-constant support and moment structure
• Topology defined via duality
• Joint work with Karim Anaya-Izquierdo, Frank Critchley
and Paul Vos
Geometry of
Sensitivity
analysis
Big Picture
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Introduction to Computational Information Geometry
• The overall objective is to construct diagnostic tools to
help understand sensitivity to model choice
• Targeted at applications where Generalised Linear
Models are used
• Our geometry is affine and convex, not manifold based:
non-constant support and moment structure
• Topology defined via duality
• Joint work with Karim Anaya-Izquierdo, Frank Critchley
and Paul Vos
• Thanks to EPSRC Grant Number EP/E017878/1
Geometry of
Sensitivity
analysis
Problem of Interest
Paul Marriott
Objectives
SEM
Geometries
The data
Pain Relief
Example
• Question: what is the
Least
informative
model
• How do modelling assumptions
40
affect inference about mean?
• Can geometry of
20
‘space of all models’ give
a framework for
discussion?
0
Frequency
60
population mean?
0
5
10
Observed Data
15
Geometry of
Sensitivity
analysis
Problem of Interest
Paul Marriott
Objectives
SEM
Geometries
The data
Pain Relief
Example
• Question: what is the
Least
informative
model
• How do modelling assumptions
40
affect inference about mean?
• Can geometry of
20
‘space of all models’ give
a framework for
discussion?
0
Frequency
60
population mean?
0
5
10
Observed Data
15
Geometry of
Sensitivity
analysis
Problem of Interest
Paul Marriott
Objectives
SEM
Geometries
The data
Pain Relief
Example
• Question: what is the
Least
informative
model
• How do modelling assumptions
40
affect inference about mean?
• Can geometry of
20
‘space of all models’ give
a framework for
discussion?
0
Frequency
60
population mean?
0
5
10
Observed Data
15
Geometry of
Sensitivity
analysis
Paul Marriott
Structured Extended
Multinomials
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Extended multinomials are multinomial but allow cell
probabilities to be zero.
Geometry of
Sensitivity
analysis
Paul Marriott
Structured Extended
Multinomials
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Extended multinomials are multinomial but allow cell
probabilities to be zero.
• Discretizing continuous data gives categorical models
with structure on the cells
Geometry of
Sensitivity
analysis
Paul Marriott
Structured Extended
Multinomials
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Extended multinomials are multinomial but allow cell
probabilities to be zero.
• Discretizing continuous data gives categorical models
with structure on the cells
• Examples of structure:
Geometry of
Sensitivity
analysis
Paul Marriott
Structured Extended
Multinomials
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Extended multinomials are multinomial but allow cell
probabilities to be zero.
• Discretizing continuous data gives categorical models
with structure on the cells
• Examples of structure:
• numerical labels
• ordering
Geometry of
Sensitivity
analysis
Paul Marriott
Structured Extended
Multinomials
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Extended multinomials are multinomial but allow cell
probabilities to be zero.
• Discretizing continuous data gives categorical models
with structure on the cells
• Examples of structure:
• numerical labels
• ordering
• Structured Extended Multinomials (SEM) include this
sample space structure
Geometry of
Sensitivity
analysis
Paul Marriott
Structured Extended
Multinomials
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Extended multinomials are multinomial but allow cell
probabilities to be zero.
• Discretizing continuous data gives categorical models
with structure on the cells
• Examples of structure:
• numerical labels
• ordering
• Structured Extended Multinomials (SEM) include this
sample space structure
• SEM proxy for universal space of all distributions
Geometry of
Sensitivity
analysis
Structured Extended
Multinomials
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Extended multinomials are multinomial but allow cell
probabilities to be zero.
• Discretizing continuous data gives categorical models
with structure on the cells
• Examples of structure:
• numerical labels
• ordering
• Structured Extended Multinomials (SEM) include this
sample space structure
• SEM proxy for universal space of all distributions
• Theory for infinite dimensions; in practice, finite
dimensional.
Geometry of
Sensitivity
analysis
−1 -simplical structure
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
1.0
• Mean (−1) parameters can
0.6
• Union of exponential
0.2
• Different support sets
0.4
0.8
be on boundary
0.0
Least
informative
model
mean parametrisation
0.0
0.2
0.4
0.6
0.8
1.0
families each with
corresponding natural (+1)
parameters
Geometry of
Sensitivity
analysis
−1 -simplical structure
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
1.0
• Mean (−1) parameters can
0.6
• Union of exponential
0.2
• Different support sets
0.4
0.8
be on boundary
●
0.0
Least
informative
model
meanSupport
parametrisation
sets
●
0.0
●
0.2
0.4
0.6
0.8
1.0
families each with
corresponding natural (+1)
parameters
Geometry of
Sensitivity
analysis
−1 -simplical structure
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
1.0
• Mean (−1) parameters can
0.6
• Union of exponential
0.2
• Different support sets
0.4
0.8
be on boundary
●
0.0
Least
informative
model
meanSupport
parametrisation
sets
●
0.0
●
0.2
0.4
0.6
0.8
1.0
families each with
corresponding natural (+1)
parameters
Geometry of
Sensitivity
analysis
+1 simplex structure
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• How can we define +1 structure on the extended
multinomial?
Geometry of
Sensitivity
analysis
+1 simplex structure
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• How can we define +1 structure on the extended
multinomial?
• Problem: the support and the moment structure
changes across SEM
Geometry of
Sensitivity
analysis
+1 simplex structure
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• How can we define +1 structure on the extended
multinomial?
• Problem: the support and the moment structure
changes across SEM
• Need to glue together different exponential families
Geometry of
Sensitivity
analysis
+1 simplex structure
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• How can we define +1 structure on the extended
multinomial?
• Problem: the support and the moment structure
changes across SEM
• Need to glue together different exponential families
• Use the dual structure of information geometry
Geometry of
Sensitivity
analysis
Very Simple Example
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Sample space of 3 values: {t0 , t1 , t2 }
Geometry of
Sensitivity
analysis
Very Simple Example
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Sample space of 3 values: {t0 , t1 , t2 }
• Multinomial distribution
∆int = {(π0 , π1 , π2 )|πi > 0,
P
πi = 1}
Geometry of
Sensitivity
analysis
Very Simple Example
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Sample space of 3 values: {t0 , t1 , t2 }
• Multinomial distribution
P
∆int = {(π0 , π1 , π2 )|πi > 0,
πi = 1}
∆∗int = {(η1 , η2 )|ηi = log(πi /π0 )}
Geometry of
Sensitivity
analysis
Very Simple Example
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Sample space of 3 values: {t0 , t1 , t2 }
• Multinomial distribution
P
∆int = {(π0 , π1 , π2 )|πi > 0,
πi = 1}
∆∗int = {(η1 , η2 )|ηi = log(πi /π0 )}
• Extended Multinomial distribution
∆ = {(π1 , π2 , π3 )|πi ≥ 0,
P
πi = 1}
Geometry of
Sensitivity
analysis
Very Simple Example
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Sample space of 3 values: {t0 , t1 , t2 }
• Multinomial distribution
P
∆int = {(π0 , π1 , π2 )|πi > 0,
πi = 1}
∆∗int = {(η1 , η2 )|ηi = log(πi /π0 )}
• Extended Multinomial distribution
P
∆ = {(π1 , π2 , π3 )|πi ≥ 0,
πi = 1}
∆∗ = more complicated; but has been done.
• Likelihood and KL divergence defined on ∆int (and ∆)
Geometry of
Sensitivity
analysis
Very Simple Example
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Sample space of 3 values: {t0 , t1 , t2 }
• Multinomial distribution
P
∆int = {(π0 , π1 , π2 )|πi > 0,
πi = 1}
∆∗int = {(η1 , η2 )|ηi = log(πi /π0 )}
• Extended Multinomial distribution
P
∆ = {(π1 , π2 , π3 )|πi ≥ 0,
πi = 1}
∆∗ = more complicated; but has been done.
• Likelihood and KL divergence defined on ∆int (and ∆)
– These quantities do not depend on {t0 , t1 , t2 }.
Geometry of
Sensitivity
analysis
Very Simple Example
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Sample space of 3 values: {t0 , t1 , t2 }
• Multinomial distribution
P
∆int = {(π0 , π1 , π2 )|πi > 0,
πi = 1}
∆∗int = {(η1 , η2 )|ηi = log(πi /π0 )}
• Extended Multinomial distribution
P
∆ = {(π1 , π2 , π3 )|πi ≥ 0,
πi = 1}
∆∗ = more complicated; but has been done.
• Likelihood and KL divergence defined on ∆int (and ∆)
– These quantities do not depend on {t0 , t1 , t2 }.
– To use the structure in the sample space we need:
Geometry of
Sensitivity
analysis
Very Simple Example
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Sample space of 3 values: {t0 , t1 , t2 }
• Multinomial distribution
P
∆int = {(π0 , π1 , π2 )|πi > 0,
πi = 1}
∆∗int = {(η1 , η2 )|ηi = log(πi /π0 )}
• Extended Multinomial distribution
P
∆ = {(π1 , π2 , π3 )|πi ≥ 0,
πi = 1}
∆∗ = more complicated; but has been done.
• Likelihood and KL divergence defined on ∆int (and ∆)
– These quantities do not depend on {t0 , t1 , t2 }.
– To use the structure in the sample space we need:
• Structured Extended Multinomial
{∆, ∆∗ , (t0 , t1 , t2 )}
Geometry of
Sensitivity
analysis
Very Simple Example
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Sample space of 3 values: {t0 , t1 , t2 }
• Multinomial distribution
P
∆int = {(π0 , π1 , π2 )|πi > 0,
πi = 1}
∆∗int = {(η1 , η2 )|ηi = log(πi /π0 )}
• Extended Multinomial distribution
P
∆ = {(π1 , π2 , π3 )|πi ≥ 0,
πi = 1}
∆∗ = more complicated; but has been done.
• Likelihood and KL divergence defined on ∆int (and ∆)
– These quantities do not depend on {t0 , t1 , t2 }.
– To use the structure in the sample space we need:
• Structured Extended Multinomial
{∆, ∆∗ , (t0 , t1 , t2 )}
• We’ll take (t0 , t1 , t2 ) = (0, 1, 2)
Geometry of
Sensitivity
analysis
Dual Parameterisations
Paul Marriott
!10
!5
0
5
10
(b) !1!geodesics in +1!simplex
0.0
0.2
0.4
0.6
0.8
1.0
(c) +1!geodesics in !1!simplex
!10
!5
0
5
10
0
5
10
(d) +1!geodesics in +1!simplex
!5
Least
informative
model
(a) !1!geodesics in !1!simplex
!10
Pain Relief
Example
Natural Parameter(∆∗ )
Mean Parameter (∆)
0.0 0.2 0.4 0.6 0.8 1.0
SEM
Geometries
0.0 0.2 0.4 0.6 0.8 1.0
Objectives
0.0
0.2
0.4
0.6
0.8
1.0
!10
!5
0
5
10
Geometry of
Sensitivity
analysis
Paul Marriott
Computational Information
Geometry
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Want to be able to numerically compute in high
dimensional SEM
Geometry of
Sensitivity
analysis
Paul Marriott
Computational Information
Geometry
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Want to be able to numerically compute in high
dimensional SEM
• Need to get the topology and geometry right
Geometry of
Sensitivity
analysis
Paul Marriott
Computational Information
Geometry
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Want to be able to numerically compute in high
dimensional SEM
• Need to get the topology and geometry right
• Information Geometry sits naturally on these simplicial
structures – different from manifolds
Geometry of
Sensitivity
analysis
Paul Marriott
Computational Information
Geometry
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Want to be able to numerically compute in high
dimensional SEM
• Need to get the topology and geometry right
• Information Geometry sits naturally on these simplicial
structures – different from manifolds
• Almost all of the information geometry on SEM is
numerically easy
Geometry of
Sensitivity
analysis
Paul Marriott
Computational Information
Geometry
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Want to be able to numerically compute in high
dimensional SEM
• Need to get the topology and geometry right
• Information Geometry sits naturally on these simplicial
structures – different from manifolds
• Almost all of the information geometry on SEM is
numerically easy
• Hard part: the mixed parameterisation
Geometry of
Sensitivity
analysis
Paul Marriott
Computational Information
Geometry
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Want to be able to numerically compute in high
dimensional SEM
• Need to get the topology and geometry right
• Information Geometry sits naturally on these simplicial
structures – different from manifolds
• Almost all of the information geometry on SEM is
numerically easy
• Hard part: the mixed parameterisation
• This is our computational framework
Geometry of
Sensitivity
analysis
Pain Relief Example
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Paul Meier et al, Reconsideration of Methodology in
Studies of Pain Relief, Biometrics (1958), pp. 330-342
Geometry of
Sensitivity
analysis
Pain Relief Example
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Paul Meier et al, Reconsideration of Methodology in
Studies of Pain Relief, Biometrics (1958), pp. 330-342
• Response Y : number of hours for which pain relief was
greater than 50%
Geometry of
Sensitivity
analysis
Pain Relief Example
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Paul Meier et al, Reconsideration of Methodology in
Studies of Pain Relief, Biometrics (1958), pp. 330-342
• Response Y : number of hours for which pain relief was
greater than 50%
• Explanatory var X : Drug (Demerol, T 1mg, T 3mg)
Geometry of
Sensitivity
analysis
Pain Relief Example
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Paul Meier et al, Reconsideration of Methodology in
Studies of Pain Relief, Biometrics (1958), pp. 330-342
• Response Y : number of hours for which pain relief was
greater than 50%
• Explanatory var X : Drug (Demerol, T 1mg, T 3mg)
• 43 patients randomly assigned; double blind
Geometry of
Sensitivity
analysis
Pain Relief Example
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Paul Meier et al, Reconsideration of Methodology in
Studies of Pain Relief, Biometrics (1958), pp. 330-342
• Response Y : number of hours for which pain relief was
greater than 50%
• Explanatory var X : Drug (Demerol, T 1mg, T 3mg)
• 43 patients randomly assigned; double blind
• Inference question concerns differences of means
across groups
Geometry of
Sensitivity
analysis
Paul Marriott
Test Drug at 3 mg
Objectives
0.05
Density
Least
informative
model
0.00
Pain Relief
Example
0.10
0.15
SEM
Geometries
0
5
10
15
Hours of Relief
20
25
Geometry of
Sensitivity
analysis
Paul Marriott
Test Drug at 3 mg
Objectives
0.05
Density
Least
informative
model
0.00
Pain Relief
Example
0.10
0.15
SEM
Geometries
0
5
10
15
Hours of Relief
20
25
Geometry of
Sensitivity
analysis
Model Family Restriction
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• The universality of the SEM is very rich, it is desirable
to have restrictions
Geometry of
Sensitivity
analysis
Model Family Restriction
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• The universality of the SEM is very rich, it is desirable
to have restrictions
• Goodness-of-fit tests can reveal poor model familes;
many diverse model families remain
Geometry of
Sensitivity
analysis
Model Family Restriction
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• The universality of the SEM is very rich, it is desirable
to have restrictions
• Goodness-of-fit tests can reveal poor model familes;
many diverse model families remain
• Question: Among data-supported model families, how
does inference depend on the choice of model family?
Geometry of
Sensitivity
analysis
Model Family Restriction
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• The universality of the SEM is very rich, it is desirable
to have restrictions
• Goodness-of-fit tests can reveal poor model familes;
many diverse model families remain
• Question: Among data-supported model families, how
does inference depend on the choice of model family?
• C.f. Results of Copas and Eguchi
Geometry of
Sensitivity
analysis
Thought experiment
Paul Marriott
Objectives
−1−geometry
+1−geometry
5
0
η2
−5
−10
0.4
0.6
0.8
1.0
−10
−5
0
η1
π1
−1
0
Log−Likelihood
−2
0.2
Log−likelihood
0.0
−3
π2
●
●
−4
Least
informative
model
0.0
Pain Relief
Example
0.4
0.8
10
SEM
Geometries
1.5
2.0
2.5
mean
True value
3.0
3.5
5
10
Geometry of
Sensitivity
analysis
Thought experiment
Paul Marriott
Objectives
−1−geometry
+1−geometry
5
0
η2
−5
−10
0.4
0.6
0.8
1.0
−10
−5
0
η1
π1
−1
0
Log−Likelihood
−2
0.2
Log−likelihood
0.0
−3
π2
●
●
−4
Least
informative
model
0.0
Pain Relief
Example
0.4
0.8
10
SEM
Geometries
1.5
2.0
2.5
mean
3.0
3.5
5
10
Geometry of
Sensitivity
analysis
Thought experiment
Paul Marriott
Objectives
−1−geometry
+1−geometry
5
0
η2
−5
−10
0.4
0.6
0.8
1.0
−10
−5
0
η1
π1
−1
0
Log−Likelihood
−2
0.2
Log−likelihood
0.0
−3
π2
●
●
−4
Least
informative
model
0.0
Pain Relief
Example
0.4
0.8
10
SEM
Geometries
1.5
2.0
2.5
mean
3.0
3.5
5
10
Geometry of
Sensitivity
analysis
Thought experiment
Paul Marriott
Objectives
−1−geometry
+1−geometry
5
0
η2
−5
−10
0.4
0.6
0.8
1.0
−10
−5
0
η1
π1
−1
0
Log−Likelihood
−2
0.2
Log−likelihood
0.0
−3
π2
●
●
−4
Least
informative
model
0.0
Pain Relief
Example
0.4
0.8
10
SEM
Geometries
1.5
2.0
2.5
mean
3.0
3.5
5
10
Geometry of
Sensitivity
analysis
Thought experiment
Paul Marriott
Objectives
−1−geometry
+1−geometry
5
0
η2
−5
−10
0.4
0.6
0.8
1.0
−10
−5
0
η1
π1
−1
0
Log−Likelihood
−2
0.2
Log−likelihood
0.0
−3
π2
●
●
−4
Least
informative
model
0.0
Pain Relief
Example
0.4
0.8
10
SEM
Geometries
1.5
2.0
2.5
mean
3.0
3.5
5
10
Geometry of
Sensitivity
analysis
Thought experiment
Paul Marriott
Objectives
−1−geometry
+1−geometry
5
0
η2
−5
−10
0.4
0.6
0.8
1.0
−10
−5
0
η1
π1
−1
0
Log−Likelihood
−2
0.2
Log−likelihood
0.0
−3
π2
●
●
−4
Least
informative
model
0.0
Pain Relief
Example
0.4
0.8
10
SEM
Geometries
1.5
2.0
2.5
mean
3.0
3.5
5
10
Geometry of
Sensitivity
analysis
Thought experiment
Paul Marriott
Objectives
−1−geometry
+1−geometry
5
0
η2
−5
−10
0.4
0.6
0.8
1.0
−10
−5
0
η1
π1
−1
0
Log−Likelihood
−2
0.2
Log−likelihood
0.0
−3
π2
●
●
−4
Least
informative
model
0.0
Pain Relief
Example
0.4
0.8
10
SEM
Geometries
1.5
2.0
2.5
mean
3.0
3.5
5
10
Geometry of
Sensitivity
analysis
Thought experiment
Paul Marriott
Objectives
−1−geometry
+1−geometry
5
0
η2
−5
−10
0.4
0.6
0.8
1.0
−10
−5
0
η1
π1
−1
0
Log−Likelihood
−2
0.2
Log−likelihood
0.0
−3
π2
●
●
−4
Least
informative
model
0.0
Pain Relief
Example
0.4
0.8
10
SEM
Geometries
1.5
2.0
2.5
mean
3.0
3.5
5
10
Geometry of
Sensitivity
analysis
Thought experiment
Paul Marriott
Objectives
−1−geometry
+1−geometry
5
0
η2
−5
−10
0.4
0.6
0.8
1.0
−10
−5
0
η1
π1
−1
0
Log−Likelihood
−2
0.2
Log−likelihood
0.0
−3
π2
●
●
−4
Least
informative
model
0.0
Pain Relief
Example
0.4
0.8
10
SEM
Geometries
1.5
2.0
2.5
mean
3.0
3.5
5
10
Geometry of
Sensitivity
analysis
Thought experiment
Paul Marriott
Objectives
−1−geometry
+1−geometry
5
0
η2
−5
−10
0.4
0.6
0.8
1.0
−10
−5
0
η1
π1
−1
0
Log−Likelihood
−2
0.2
Log−likelihood
0.0
−3
π2
●
●
−4
Least
informative
model
0.0
Pain Relief
Example
0.4
0.8
10
SEM
Geometries
1.5
2.0
2.5
mean
3.0
3.5
5
10
Geometry of
Sensitivity
analysis
Least informative model
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Since rotation is through data generation process
(DGP)we can’t use goodness-of-fit tests to distinguish
between model families
Geometry of
Sensitivity
analysis
Least informative model
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Since rotation is through data generation process
(DGP)we can’t use goodness-of-fit tests to distinguish
between model families
• Rotation changes the mode and the shape of the
likelihood
Geometry of
Sensitivity
analysis
Least informative model
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Since rotation is through data generation process
(DGP)we can’t use goodness-of-fit tests to distinguish
between model families
• Rotation changes the mode and the shape of the
likelihood
• Smallest Expected Fisher information at DGP is when
exponential model is orthogonal to level sets of mean
Geometry of
Sensitivity
analysis
Least informative model
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Since rotation is through data generation process
(DGP)we can’t use goodness-of-fit tests to distinguish
between model families
• Rotation changes the mode and the shape of the
likelihood
• Smallest Expected Fisher information at DGP is when
exponential model is orthogonal to level sets of mean
• Families of Models with this orthogonality property we
call least informative model families
Geometry of
Sensitivity
analysis
Least informative model
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Since rotation is through data generation process
(DGP)we can’t use goodness-of-fit tests to distinguish
between model families
• Rotation changes the mode and the shape of the
likelihood
• Smallest Expected Fisher information at DGP is when
exponential model is orthogonal to level sets of mean
• Families of Models with this orthogonality property we
call least informative model families
• Information in inference comes from two sources: (i)
data and (ii) modelling assumptions. To be
conservative minimise (ii) relative to (i)
Geometry of
Sensitivity
analysis
Least informative models
Paul Marriott
select model which maximizes entropy for fixed
moments is a special case of a least informative model
Least
informative
model
π2
0.6
0.8
1.0
MCEM and Empirial Likelihood
0.4
Pain Relief
Example
• Moment Constrained Maximum Entropy (MCME)
●
0.2
SEM
Geometries
0.0
Objectives
0.0
0.2
0.4
0.6
π1
0.8
1.0
Geometry of
Sensitivity
analysis
Least informative models
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
• Moment Constrained Maximum Entropy (MCME)
select model which maximizes entropy for fixed
moments is a special case of a least informative model
Least
informative
model
0.4
π2
0.6
0.8
1.0
MCEM and Empirial Likelihood
0.0
0.2
●
0.0
0.2
0.4
0.6
0.8
1.0
π1
• Bootstrapping “least favourable models” Efron (1981)
Geometry of
Sensitivity
analysis
Least informative models
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
• Moment Constrained Maximum Entropy (MCME)
select model which maximizes entropy for fixed
moments is a special case of a least informative model
Least
informative
model
0.4
π2
0.6
0.8
1.0
MCEM and Empirial Likelihood
0.0
0.2
●
0.0
0.2
0.4
0.6
0.8
1.0
π1
• Bootstrapping “least favourable models” Efron (1981)
• Empirical likelihood: maximize likelihood for fixed
moments
Geometry of
Sensitivity
analysis
Effect of translation
Paul Marriott
Objectives
−1−geometry
+1−geometry
5
0
η2
−10
−5
0.4
0.4
0.6
0.8
1.0
−10
−5
0
η1
π1
−2
−1
0
Log−Likelihood
Log−likelihood
0.2
−3
0.0
−4
Least
informative
model
0.0
Pain Relief
Example
π2
0.8
10
SEM
Geometries
1.5
2.0
2.5
mean
3.0
3.5
5
10
Geometry of
Sensitivity
analysis
Effect of translation
Paul Marriott
Objectives
−1−geometry
+1−geometry
5
0
η2
−10
−5
0.4
0.4
0.6
0.8
1.0
−10
−5
0
η1
π1
−2
−1
0
Log−Likelihood
Log−likelihood
0.2
−3
0.0
−4
Least
informative
model
0.0
Pain Relief
Example
π2
0.8
10
SEM
Geometries
1.5
2.0
2.5
mean
3.0
3.5
5
10
Geometry of
Sensitivity
analysis
Effect of translation
Paul Marriott
Objectives
−1−geometry
+1−geometry
5
0
η2
−10
−5
0.4
0.4
0.6
0.8
1.0
−10
−5
0
η1
π1
−2
−1
0
Log−Likelihood
Log−likelihood
0.2
−3
0.0
−4
Least
informative
model
0.0
Pain Relief
Example
π2
0.8
10
SEM
Geometries
1.5
2.0
2.5
mean
3.0
3.5
5
10
Geometry of
Sensitivity
analysis
Effect of translation
Paul Marriott
Objectives
−1−geometry
+1−geometry
5
0
η2
−10
−5
0.4
0.4
0.6
0.8
1.0
−10
−5
0
η1
π1
−2
−1
0
Log−Likelihood
Log−likelihood
0.2
−3
0.0
−4
Least
informative
model
0.0
Pain Relief
Example
π2
0.8
10
SEM
Geometries
1.5
2.0
2.5
mean
3.0
3.5
5
10
Geometry of
Sensitivity
analysis
Effect of translation
Paul Marriott
Objectives
−1−geometry
+1−geometry
5
0
η2
−10
−5
0.4
0.4
0.6
0.8
1.0
−10
−5
0
η1
π1
−2
−1
0
Log−Likelihood
Log−likelihood
0.2
−3
0.0
−4
Least
informative
model
0.0
Pain Relief
Example
π2
0.8
10
SEM
Geometries
1.5
2.0
2.5
mean
3.0
3.5
5
10
Geometry of
Sensitivity
analysis
Effect of translation
Paul Marriott
Objectives
−1−geometry
+1−geometry
5
0
η2
−10
−5
0.4
0.4
0.6
0.8
1.0
−10
−5
0
η1
π1
−2
−1
0
Log−Likelihood
Log−likelihood
0.2
−3
0.0
−4
Least
informative
model
0.0
Pain Relief
Example
π2
0.8
10
SEM
Geometries
1.5
2.0
2.5
mean
3.0
3.5
5
10
Geometry of
Sensitivity
analysis
Effect of translation
Paul Marriott
Objectives
−1−geometry
+1−geometry
5
0
η2
−10
−5
0.4
0.4
0.6
0.8
1.0
−10
−5
0
η1
π1
−2
−1
0
Log−Likelihood
Log−likelihood
0.2
−3
0.0
−4
Least
informative
model
0.0
Pain Relief
Example
π2
0.8
10
SEM
Geometries
1.5
2.0
2.5
mean
3.0
3.5
5
10
Geometry of
Sensitivity
analysis
Effect of translation
Paul Marriott
Objectives
−1−geometry
+1−geometry
5
0
η2
−10
−5
0.4
0.4
0.6
0.8
1.0
−10
−5
0
η1
π1
−2
−1
0
Log−Likelihood
Log−likelihood
0.2
−3
0.0
−4
Least
informative
model
0.0
Pain Relief
Example
π2
0.8
10
SEM
Geometries
1.5
2.0
2.5
mean
3.0
3.5
5
10
Geometry of
Sensitivity
analysis
Sensitive perturbations
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• There exists large perturbations of models which have
‘no effect’ on inference
Geometry of
Sensitivity
analysis
Sensitive perturbations
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• There exists large perturbations of models which have
‘no effect’ on inference
• Limit of these translation exists-use the correct topology
Geometry of
Sensitivity
analysis
Sensitive perturbations
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• There exists large perturbations of models which have
‘no effect’ on inference
• Limit of these translation exists-use the correct topology
• Limit is Empirical Likelihood
Geometry of
Sensitivity
analysis
Sensitive perturbations
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• There exists large perturbations of models which have
‘no effect’ on inference
• Limit of these translation exists-use the correct topology
• Limit is Empirical Likelihood
• Shows link between least informative parametric
inference and non-parametric inference
Geometry of
Sensitivity
analysis
Sensitive perturbations
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• There exists large perturbations of models which have
‘no effect’ on inference
• Limit of these translation exists-use the correct topology
• Limit is Empirical Likelihood
• Shows link between least informative parametric
inference and non-parametric inference
• The number of perturbations which matter for inference
about the mean can be surprising small
Geometry of
Sensitivity
analysis
Sensitive perturbations
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• There exists large perturbations of models which have
‘no effect’ on inference
• Limit of these translation exists-use the correct topology
• Limit is Empirical Likelihood
• Shows link between least informative parametric
inference and non-parametric inference
• The number of perturbations which matter for inference
about the mean can be surprising small
Geometry of
Sensitivity
analysis
Exploring Model space
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Our geometry allows us to explore the range of
inferences about mean across different data-plausible
models
Geometry of
Sensitivity
analysis
Exploring Model space
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Our geometry allows us to explore the range of
inferences about mean across different data-plausible
models
• The inferences drawn depend upon (i) data through
choice of sufficient statistics (rotations) (ii) other
modelling assumptions (translations)
Geometry of
Sensitivity
analysis
Exploring Model space
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Our geometry allows us to explore the range of
inferences about mean across different data-plausible
models
• The inferences drawn depend upon (i) data through
choice of sufficient statistics (rotations) (ii) other
modelling assumptions (translations)
• If number of sufficient statistics is greater than the
dimension of interest parameter then need to select
way of drawing marginal inference e.g. plug-in, profile
likelihood, marginal posterior . . .
Geometry of
Sensitivity
analysis
Exploring Model space: pain
example
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• In this example a GLM approach is often taken in
practice
Geometry of
Sensitivity
analysis
Exploring Model space: pain
example
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• In this example a GLM approach is often taken in
practice
• This selects a two dimensional sufficient statistic and
corresponding exponential family
Geometry of
Sensitivity
analysis
Exploring Model space: pain
example
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• In this example a GLM approach is often taken in
practice
• This selects a two dimensional sufficient statistic and
corresponding exponential family
• Gamma family
• Log-Normal family
• Moment Constrained Maximum Entropy family
Geometry of
Sensitivity
analysis
Exploring Model space: pain
example
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• In this example a GLM approach is often taken in
practice
• This selects a two dimensional sufficient statistic and
corresponding exponential family
• Gamma family
• Log-Normal family
• Moment Constrained Maximum Entropy family
• Fixes a mean-variance relationship through
specification of the variance function φV (µ)
Geometry of
Sensitivity
analysis
Exploring Model space: pain
example
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• In this example a GLM approach is often taken in
practice
• This selects a two dimensional sufficient statistic and
corresponding exponential family
• Gamma family
• Log-Normal family
• Moment Constrained Maximum Entropy family
• Fixes a mean-variance relationship through
specification of the variance function φV (µ)
• Often using the plug-in, φ̂, for marginal inference
Geometry of
Sensitivity
analysis
Exploring Model space: pain
example
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• In this example a GLM approach is often taken in
practice
• This selects a two dimensional sufficient statistic and
corresponding exponential family
• Gamma family
• Log-Normal family
• Moment Constrained Maximum Entropy family
• Fixes a mean-variance relationship through
specification of the variance function φV (µ)
• Often using the plug-in, φ̂, for marginal inference
• Our geometric tools allow us to explore the sensitivity to
these kinds of assumptions
Geometry of
Sensitivity
analysis
Range of Inference
Paul Marriott
Objectives
SEM
Geometries
−6
−5
log−lik
−4
−3
−2
0.3
0.2
0.1
0.0
Probability
−1
0.4
Least
informative
model
0
Pain Relief
Example
5
10
15
Bin
20
0
5
10
mean
15
20
Geometry of
Sensitivity
analysis
Range of Inference
Paul Marriott
Objectives
log−lik
log−lik
55
10
15
Bin
20
−6
−6 −5
−5 −4
−4 −3−3 −2−2 −1 −1 0 0
0.1 0.10
0.2
0.0
0.00
Least
informative
model
Probability
Probability
Pain Relief
Example
0.3
0.20 0.4 0.30
SEM
Geometries
0
55
10
10
mean
15
15
20
20
Geometry of
Sensitivity
analysis
Summary
Paul Marriott
Objectives
SEM
Geometries
Pain Relief
Example
Least
informative
model
• Computational Information Geometry
• The overall objective is to construct diagnostic tools to
help understand sensitivity to model choice
• Targeted at applications where Generalised Linear
Models are used
• SEM geometry is affine and convex, not manifold
based: non-constant support and moment structure
• Topology defined via duality