Meta-Analysis in Medicine
and Health Policy
Dalene Stangl
Institute of Statistics and Decision Sciences
Duke University
Two Topics


Directions of current research
Gaps in current methologies
Current Research






shifting paradigms from fixed-effect to
random-effect models
incorporating inconsistency between
study designs and between study
outcomes
assessing model uncertainty
assessing and including measures of
study quality and publication bias
incorporating study-level covariates
developing software
Major Gap


Decision process usually remains
informal and ad hoc
Two suggestions



Train researchers and policy makers to
specify utilities, or at least not use automatic
0/1 loss functions
Demand analyses that demonstrate
robustness to utility functions
Example - hospital profiling
Part I: Research Review
Meta-Analysis in Medicine and
Health Policy, Marcel Dekker
Berry and Stangl (Eds.)

Paradigm Shift
 10
of 16 Chapters

Meta-Analysis of Heterogeneously Reported Study Results – A
Bayesian Approach
Keith Abrams, Paul Lambert, Bruno Sanso, Chris Shaw, Theresa Marteau

Meta-Analysis vs Large Trials: Resolving the Controversy
Scott M. Berry

A Bayesian meta-analysis of randomized mega-trials for the
choice of thrombolytic agent in acute myocardial infarction
James Brophy, Lawrence Joseph

Combining Studies with Continuous and Dichotomous
Responses: A Latent Variables Approach
Francesca Dominici, Giovanni Parmigiani

Computer Modeling and Graphical Strategies for Meta-Analysis
William H. DuMouchel, Sharon-Lise T. Normand

A Bayesian Meta-Analysis of the Relationship between Duration
of Estrogen Exposure and Occurrence of Endometrial Cancer
Daniel T. Larose

Modeling and Implementation Issues in Bayesian Meta-Analysis
Donna Pauler, Jon Wakefield

An Application of Meta-Analysis to Population Pharmacokinetic
data
Nargis Rahman and Jonathan Wakefield

Meta-analysis of Individual Patient Survival Data Using Random
Effects Models
Daniel J. Sargent, Benny Zee, Chantal Milan, Valter Torri, Guido Francini

Adjustment for Publication and Quality Bias in Bayesian Metaanalysis
D. D. Smith, Geof H. Givens, R. L. Tweedie
Part I: Research Review

Inconsistent Outcomes and
Result Reporting

Meta-Analysis of Heterogeneously Reported
Study Results – A Bayesian Approach
K. Abrams, P. Lambert, B. Sanso, C. Shaw, T. Marteau

Combining Studies with Continuous and
Dichotomous Responses: A Latent Variables
Approach
Francesca Dominici, Giovanni Parmigiani
Part I: Research Review

Model Uncertainty

Computer Modeling and Graphical Strategies for
Meta-Analysis
William H. DuMouchel, Sharon-Lise T. Normand
Part I: Research Review

Study-level Covariates

A Bayesian Meta-Analysis of Randomized Mega-Trials for the
Choice of Thrombolytic Agent in Acute Myocardial Infarction
James Brophy, Lawrence Joseph

Computer Modeling and Graphical Strategies for MetaAnalysis
William H. DuMouchel, Sharon-Lise T. Normand

A Bayesian Meta-Analysis of the Relationship between
Duration of Estrogen Exposure and Occurrence of
Endometrial Cancer
Daniel T. Larose

Modeling and Implementation Issues in Bayesian MetaAnalysis
Donna Pauler, Jon Wakefield

Meta-Analysis of Individual Patient Survival Data Using
Random-Effects Models
D. Sargent, B. Zee, C. Milan, V. Torri, G. Francini
Part I: Research Review

Measures of study quality and
publication bias

Computer Modeling and Graphical Strategies
for Meta-Analysis
William H. DuMouchel, Sharon-Lise T. Normand

Modeling and Implementation Issues in
Bayesian Meta-Analysis
Donna Pauler, Jon Wakefield

Adjustment for Publication and Quality Bias in
Bayesian Meta-analysis
D. D. Smith, Geof H. Givens, R. L. Tweedie
Part I: Research Review

Software

Meta-Analysis in Practice: ACritical Review of
Available Software
Alexander J. Sutton, Paul C. Lambert, Martin A. G.
Hellmich, Keith R. Abrams, David. R. Jones
Part II: Decision Analysis Gap

Sir Claus Moser, 1975


Dorothy Price, Director, NCHS, 1976


“…. As in other areas of social policy, health statisticians and
health data are increasingly expected to provide keys to rational
decision making. To accomplish this goal, the statistician and
decision maker need to interact to an increasing degree.”
WHO, 1974


“foremost responsibility (of the statistician) is to contribute to more
enlightened and efficient ‘decision making’ … through the fullest
possible exploitation of our skills in analyzing and interpreting the
data.”
“The statistician must concern himself with policy-making. …
supplier of the statistical information most relevant to the problem
under consideration and as a skilled interpreter of the information,
indicating the possible policy choices and the probable outcome of
different strategies.”
John Tukey, 1976
 “…those statisticians for whom opportunity and a natural bent
combine to offer experience and the development of expertise
ought, in the public interest, become as much policy makers as
their roles allow.”
Part II: Decision Analysis Gap

Commonalities



All refer to importance of decision making
Recommend more involvement of statisticians
All statements were made 20-25 years ago
Part II: Decision Analysis Gap

1970s-1980s
 development
of a coordinated,
systematic data base

1990s
 “Statistics
and Policy”
(Spencer, ed.)
 10
chapters - data collection and
descriptive statistics
 4 chapters - statistics for policy
analysis
 0 chapters - statistics for decision
making.
Part II: Decision Analysis Gap

Example - Hospital Profiling
 Paradigm
Shift: Hierarchical
Logistic Regression
 Outlier Identification

Posterior probability of excess
mortality for average patient adjusting
for case mix.

Posterior probability of a large
difference between a hospital’s
adjusted and standardized mortality.

Z-score (HCFA’s algorithm) –
hospitals with z-scores above 1.645
were classified as aberrant.
Part II: Decision Analysis Gap

Example - Hospital Profiling
 Embedded
Utilities
chosen outcome: mortality
 time point - 30 days
 relative performance measure
 median times 1.5
 posterior probabilities – constant
utilities

 Robustness???????
Part II: Decision Analysis Gap

2 Proposals
 relinquish
the automatic
constant utility embedded in
p-values and posterior
probabilities
 present statistical output in a
way that increases the
possibility and probability of
applying a wide diversity of
utility functions
Part II: Decision Analysis Gap
 relinquish
the automatic
constant utility embedded in
p-values and posterior
probabilities
    i ,
Figure 1.
i
sufficient
not sufficient
Part II: Decision Analysis Gap

Two decisions:
 d0:
accept that the hospital
is of sufficient quality
 d1: reject that the hospital is
of sufficient quality


U(d,) which measures utility
of d when the uncertain
value is 
U(d1,)>U(d0,) for >
Part II: Decision Analysis Gap
Figure 2. Constant Utility Function
U(d0,)
U(d1,)

sufficient
i
not sufficient
Figure 3. Linear Utility Function
U(d0,)
U(d1,)
 i

sufficient
not sufficient
Figure 4. Compromise Utility Function
U(d0,)
U(d 1,)

i

sufficient
not sufficient
Part II: Decision Analysis Gap

L()=U(d1,)-U(d0,)
Figure 5. Constant Loss
L()=(U(d1,)-U(d0,)

sufficient
Simplify Elicitations
i
Figure 6. Linear Loss
L()=U(d1,)-U(d0,)
not sufficient
 i

sufficient
Figure 7. Compromise Loss
L()=U(d1,)-U(d0,)
not sufficient
i

sufficient
not sufficient
Part II: Decision Analysis Gap

Comparison of Loss
Functions
 continuous/discontinuous
derivative at  is key
for decision making
 second
 must
elicit this quantity
Part II: Decision Analysis Gap

Extension to multivariate
outcomes
 Tan
& Smith (1998)
Part II: Decision Analysis Gap

2nd Proposal
 present
statistical output in a
way that increases the
possibility and probability of
applying a wide diversity of
utility functions

predictive distributions for
general outcomes
• profiling example - predictive survival
distributions
• requires more sophisticated modeling
• allows examination of diverse utilities
References
Berger, J.O. and Delampady, M. (1987). Testing precise hypotheses (with discussion). Statistical
Science 2: 317-352.
Berger, J.O. and Sellke, T. (1987). Testing a point-null hypothesis: the irreconcilability of p-values
and evidence ( with discussion). J. Amer. Statistica Assoc. 82:112-139.
Berger. R.L. and Hsu, J.C. (1996). Bioequivalence trials, intersection-union tests and equivalence
confidence sets (with discussion). Statist. Sci. 11:283-319.
Bernardo, J.M. and Smith, A.F.M. (1994). Bayesian Theory. Wiley. Chichester.
Casella, G. and Berger, R.L. (1987). Reconciling Bayesian and frequentist evidence in the one-sided
testing problem. J. Amer. Statist. Assoc. 82: 106-111.
Cochran, W.G. (1976). The role of statistics in national health policy decisions. American Journal of
Epidemiology 104(4):370-379.
Lilford, R.J. and Thornton J.D. (1992). Decision logic in medical practice. Journal of the Royal
College of Physicians of London, 26(4):400-412.
Lilford, R.J. and Braudholtz, D. (1996). The statistical basis of public policy: a paradigm shift is
overdue. British Medical Journal 313(7057):603-607.
Lindley, D.V. (1985). Making Decisions. Wiley, Chichester.
Lindley, D.V. (1997). The choice of sample size (with discussion). The Statistician 46: 129-166.
Lindley, D.V. (1998). Decision Analysis and Bioequivalence Trials. Statistical Science 13(2): 136-141.
Lindley, D.V. and Singpurwalla, N.D. (1991). On the evidence needed to reach agreed action between
adversaries, with application to acceptance sampling. J. Amer. Statist. Assoc. 86: 933-937.
Normand, S., Glickman, M., Gatsonis, C. (1997). Statistical methods for profiling providers of medical
care: issues and applications. J. Amer. Statist. Assoc. 92(439):803-814.
Paltiel, A.D. and Stinnett A.A. (1996). Making health policy decisions: Is human instinct rational? Is
rational choice human? Chance 9(2):34-39.
Rice, D. (1977). The role of statistics in the development of health care policy. The American
Statistician 31(3):101-106.
Tan S.B. and Smith, A.F.M. (1998). Exploratory thoughts on clinical trials with utilities. Statistics in
Medicine 17:2771-2791.
Tukey, J.W. (1976). Discussion of: “Role of statistics in national health policy decisions.” American
Journal of Epidemiology 104(4):380-385.
Slides for this talk will be at my website.
www.stat.duke.edu
-> Faculty
-> Dalene Stangl
-> Recent Talks
->Meta-Analysis in Medicine and Health Policy