Introduction to Research Methods
In the Internet Era
Critically Reviewing an
Epidemiologic Study
Thomas Songer, PhD
Learning Objectives:
1. Describe the approach to reviewing a manuscript
2. Identify the research hypothesis of a manuscript
3. Identify the quality of the research and the
validity of the findings of a manuscript
4. Describe the factors which may raise concern
about the truth of a research finding
2
Reviewing a Paper is about
Asking Questions
3
What level of measurement error exists?
Are there unstated confounding factors?
What is the study hypothesis?
What is my overall
impression?
Is the study adequately powered?
Were appropriate statistical
procedures used?
What population do the study subjects originate from?
4
Key Areas of Focus
• Critique of data collection
• Critique of data analysis
• Critique of data interpretation
5
Example Manuscript
• Fitzpatrick AL, Kuller LH, Lopez OL, Diehr P,
O’Meara ES, Longstreth Jr. WT, Luchsinger JA.
Midlife and Late-Life Obesity and the Risk of
Dementia. Archives of Neurology 66(3):336342, 2009.
6
What to Examine When Evaluating
Data Collection
• Study Context
• Study Objectives
– Is a hypothesis stated?
• Exposure and Outcome
Variables
• Study Design
• Study Population
• Potential for
– Selection Bias
– Information Bias
– Confounding
7
Study Context
• Several Issues to consider
–What is the public health significance of
this study?
–Does this study generate new hypotheses
or confirm previous results with improved
methods?
–Is the study hypothesis biologically
plausible?
8
Study Context – Fitzpatrick, et. al.
- What is the public health significance of
this study?
• Rising obesity levels in US
• Dementia increasing in US
• Aging population
–Does this study generate new
hypotheses or confirm previous results
with improved methods?
• Seeks to clarify conflicting results in the literature
by examining a large study sample longitudinally
–Is the study hypothesis biologically
plausible?
9
Study Objectives
• What do the investigators want to
achieve in this research?
–What is the hypothesis of this study?
• There may be more then one
• Is the hypothesis specific or too general to
refute?
10
Exposure and Outcome Variables
• Primary exposure
– How was variable defined?
• How was information on
exposure collected?
– Best method?
– Sensitivity/specificity of this
method?
– Potential for
misclassification?
• Primary outcome
– Conceptual vs. operational
outcome?
• e.g. breast cancer vs. malignant
neoplasm of the breast tissue
• How was information on
outcome collected?
– Best method?
– Sensitivity/specificity of this
method?
– Potential for misclassification?
11
Type of Study
• What study design was employed?
• Is it an appropriate design?
– Exposure or outcome rare?
– New hypothesis?
• What are the limitations and strengths of
this design?
12
Type of Study -- Fitzpatrick
• What study design was employed?
– Cohort study (retrospective)
• Is it an appropriate design?
– Exposure or outcome rare? Neither
– New hypothesis? No, but conflicting study results
• What are the limitations and strengths of this
design?
– Strengths: longitudinal assessment, incidence of
dementia, uses previously collected data
– Limitations: short period of assessment
13
The Study Population
• What was the source of study population?
– How does the study population compare to the
general population?
• How were subjects selected?
– Could this method introduce selection bias?
• What was the sample size?
– Is the statistical power of the study identified?
– Out of the projected study sample, how many persons
participated?
14
The Study Population -- Fitzpatrick
• What was the source of study population?
– Community dwellers who were Medicare eligible
over age 65 years - 4 sites
• How were subjects selected? Not stated
– Could this method introduce selection bias?
• What was the sample size? 2798 out of original
cohort of 5888 adults
– Is the statistical power of the study identified? Yes
– Out of the projected study sample, how many persons
participated? Unknown from original study.
15
Potential for Bias
SELECTION BIAS
• Could there have been
bias in the selection of
subjects?
• What type of bias would
this be?
– e.g. healthy worker bias
• In which direction would
this bias affect the
measure of association?
INFORMATION BIAS
• Could there have been
bias in the collection of
information?
• What type of bias would
this be?
– e.g. interviewer bias
• In which direction would
this bias affect the
measure of association?
16
Potential for Confounding
• What factors were potentially
confounding the study relationship?
• What methods did the authors use to
minimize the influence of confounding
when planning the study?
–E.g. restriction, matching, randomization,
etc.
• Is there still residual confounding?
17
Potential for Confounding - Fitzpatrick
• What factors were potentially
confounding the study relationship?
–Table 1, others
• What methods did the authors use to
minimize the influence of confounding
when planning the study?
–E.g. restriction, matching, randomization,
etc.
• Is there still residual confounding? likely
18
What to Examine When Evaluating
Data Analysis
• Confounding
• Measures of Association
• Measures of Statistical Stability
19
Data Analysis -- Confounding
• What methods were used to control confounding?
-Standardization: Indirect and Direct. Usually used to control for
differences in age distribution among populations.
-Stratification: Allows you to examine data more closely.
However, it is difficult to control for more than 1 confounder.
-Matching: Done in Case-Control Studies.
-Multivariate Analysis: Linear Regression, Logistic Regression,
Poisson Regression, Cox Proportional Hazards model. Allows
you to control for multiple confounders simultaneously.
20
Data Analysis – Measures of
Association
• What Measures of Association were reported in the
study? Was the correct measure used?
• Cohort Study: Relative Risk (RR), Odds Ratio (OR),
Hazard Ratio (HR), Incidence Rate Ratio (IRR.
• Case-Control: Exposure or Disease OR (if nested).
Can not use RR. However, the OR is a good estimate
of the RR when the prevalence of the disease in the
study population is very low.
• Cross-sectional Study: Prevalence Ratio.
• Ecologic Study: Correlation coefficient.
21
Data Analysis – Statistical Stability
• How was the potential for random error accounted for
in the study?
• Hypothesis Testing: Can use p-values or confidence
Intervals (CI) to test the null hypothesis.
• P-value: The probability of observing the study
results given that the null hypothesis is true. P<0.05 is
a standard value that investigators use to reject the null
hypothesis of no association and declare that there is a
significant relationship between 2 variables.
22
Data Analysis – Statistical Stability
• 95% CI: This measure can be used for hypothesis
testing and interval estimation. Can be defined as, if
one will repeat the study 100 times the true
association will lie inside the interval 95% of the
time.
• We fail to reject the null hypothesis when a
confidence interval contains the null value of 1
between its lower and upper limits for relative
measures.
23
Data Analysis – Statistical Stability
• Large confidence intervals indicate that the
standard error is high. A high standard error is
often related to a small sample size.
Underpowered studies normally have wider
confidence intervals and thus difficulty in
rejecting the null hypothesis.
• The problem, therein, lies that it is difficult to
know if the non-association is real or false.
24
What to Examine When Interpreting
the Results of the Study
• Major findings of the research
• Influence (on the results) of:
– Bias and confounding
– misclassification
25
Major Findings
• The first paragraph of the discussion section
in a manuscript should summarize the main
findings of the study.
• Example: Sedentary individuals in this
study have 3 (95% CI:1.5-4.9) times the risk
of developing a Myocardial (MI) compared
to active individuals after controlling for
potential confounders.
26
Influence of Bias and Confounding
• Reader should be able to recognize information
bias, selection bias, or confounding in the
study and assess their magnitude and direction
in the study.
• Bias or confounding that is large in magnitude
signals that the findings in this sample may not
approximate what you would expect to see in
the population.
27
Misclassification
• Misclassification of the exposure or the
outcome (or both) can influence study results
Non-Differential: Misclassification is similar in
the exposure or outcome groups. This would
bias the results to the null making it unlikely
for investigators to reject the null hypothesis.
28
• Differential: Misclassification occurs at a
different rate in exposure or outcome
groups.
• Example of differential misclassification, a
larger number of individuals are classified
as high stress instead of medium stress than
individuals classified as medium stress
instead of high stress. This type of
misclassification can bias results away or
towards the null hypothesis.
29
Formulating an Overall Impression
of the Manuscript
• What are the strengths and limitations of
the report?
• How do these balance?
• Can the results be generalized to the
whole population?
30
Strengths and Limitations
• Examine the overall issues related to data collection,
data analysis, and data interpretation.
• What conclusions do you draw from the results based
upon your interpretation of the strengths and
limitations of the study?
• Do the strengths outweigh the limitations?
• They are often mentioned in the discussion section of
a manuscript.
31
Generalizability
• Goal is to have a study where the results can be
used to infer what is going on in the population
• Major problems with the internal validity of the
study make it difficult to for the results to be
generalized to any population.
• Example, the study population excluded a certain
groups, minorities, women, blacks, or low income
individuals. The results would not be
generalizable to these groups.
32
Conclusions and Justification
• The conclusions are a brief summary of the findings.
• Authors tend to include recommendations for future
studies or policy.
• It is essential that the recommendations do not stray
far from the study findings. Recommendations
should be made in the context of the findings or the
readers may be deceived and make incorrect
conclusions about the actual results of the study.
33
The Big Picture of Research Findings
• Publication bias
• John P.A. Ioannidis
– Why Most Published Research Findings
are False. PLoS Medicine 2(8):e124,
2005.
34
Publication bias
• Definition:
“Publication bias refers to the greater
likelihood that studies with positive
results will be published”
JAMA 2002;287:2825-2828
Abbasi
35
Publication bias may ….
• Distort the scientific record
• Hide the “truth” of association/no
association
• Influence doctors’ decision making
• Mislead policy makers
• Etc.
36
Ioannidis Corollaries
• The smaller the studies conducted, the less
likely the research findings are to be true
• The smaller the effect size, the less likely the
research findings are to be true
• The greater the financial interest and
prejudice, the less likely the research findings
are to be true
• The hotter a topic interest, the less likely the
research findings are to be true
37