External validity:
to what populations do our
study results apply?
Epidemiology matters: a new introduction to methodological foundations
Chapter 12
Seven steps
1.
Define the population of interest
2.
Conceptualize and create measures of exposures and health
indicators
3.
Take a sample of the population
4.
Estimate measures of association between exposures and health
indicators of interest
5. Rigorously evaluate whether the association observed
suggests a causal association
6.
Assess the evidence for causes working together
7.
Assess the extent to which the result matters, is externally valid,
to other populations
Epidemiology Matters – Chapter 1
2
Generalizability or external validity refers to our
capacity to generalize our results beyond our study
sample
Epidemiology Matters – Chapter 12
Question: How can we assess the extent to which
results of a study are applicable in populations outside
of the underlying population base of particular study?
Answer: Think through characteristics of population of
interest and determine how robust study findings
might be across populations with similar or different
characteristics.
Epidemiology Matters – Chapter 12
1.
2.
3.
4.
5.
6.
7.
8.
Validity, four stages
Introduction to external validity
Prevalence of component causes
Causation and study design
External versus internal validity
Randomized control trials
Representative samples
Summary
Epidemiology Matters – Chapter 12
1.
2.
3.
4.
5.
6.
7.
8.
Validity, four stages
Introduction to external validity
Prevalence of component causes
Causation and study design
External versus internal validity
Randomized control trials
Representative samples
Summary
Epidemiology Matters – Chapter 12
Four types of validity
1. Measurement validity
2. Statistical conclusion validity
3. Internal validity
4. External validity
Epidemiology Matters – Chapter 12
1. Measurement validity
 An association cannot be valid beyond the study
sample unless it is valid within the study sample
 Accuracy and precision are key measurements
 Have we measured what we wanted to measure?
Epidemiology Matters – Chapter 12
2. Statistical conclusion validity
 Is the association observed due to chance?
 We assess this via confidence intervals around
estimates of association to describe role of sampling
variability
 We aim to rule out the potential that our results
arose by chance in the sampling process from an
underlying population of interest
Epidemiology Matters – Chapter 12
3. Internal validity
 Assessment of non-comparability between exposed
and non-exposed in any study
Epidemiology Matters – Chapter 12
4. External validity
 Explore external validity after assessing and ensuring
 Measurement validity
 Statistical conclusion validity
 Internal validity
Epidemiology Matters – Chapter 12
1.
2.
3.
4.
5.
6.
7.
8.
Validity, four stages
Introduction to external validity
Prevalence of component causes
Causation and study design
External versus internal validity
Randomized control trials
Representative samples
Summary
Epidemiology Matters – Chapter 12
External validity
External validity: the applicability of study findings beyond the study
sample
In an epidemiologic study we
1.
Identify a population of interest
2.
Sample from population – random or purposive
3.
Conduct study
4.
Sample result should reflect underlying association in population
of interest
Therefore, identifying population of interest is central to exploring
external validity once we have our findings
Epidemiology Matters – Chapter 12
1.
2.
3.
4.
5.
6.
7.
8.
Validity, four stages
Introduction to external validity
Prevalence of component causes
Causation and study design
External versus internal validity
Randomized control trials
Representative samples
Summary
Epidemiology Matters – Chapter 12
Prevalence of component causes
To understand external validity we must understand
the prevalence and distribution of component causes
across populations.
Epidemiology Matters – Chapter 12
Prevalence of component causes, example
Question: Does exposure to ambient air pollution cause
lung cancer?
 Component cause A: Ambient air pollution and smoking;
therefore, smoking will cause lung cancer only among
individuals exposed to ambient air pollution
 Component cause B: Genetics
Epidemiology Matters – Chapter 12
Non-diseased
Non-exposed
Diseased
Exposed air
pollution
Exposed genetic,
diseased
Smoker
Epidemiology Matters – Chapter 12
Exposed genetic,
diseased, exposed
air pollution, smoker
Exposed air
pollution, diseased,
smoker
Prevalence of component causes
example lung cancer
Population 1
Black = exposed to air pollution
Dots = genetically determined
Hat = smoker
Exposed to air pollution and genetic
(regardless of disease status and smoking status)
2
Un-exposed to air pollution and exposed to genetic
(regardless of disease status and smoking status)
2
Exposed to air pollution and smoker
(regardless of disease status and genetic status)
Un-exposed to air pollution and smoker
(regardless of disease status and genetic status)
6
6
Epidemiology Matters – Chapter 12
Prevalence of component causes
example lung cancer
Population 2
Black = exposed to air pollution
Dots = genetically determined
Hat = smoker
Exposed to air pollution and genetic
(regardless of disease status and smoking status)
2
Un-exposed to air pollution and exposed to genetic
(regardless of disease status and smoking status)
2
Exposed to air pollution and smoker
(regardless of disease status and genetic status)
Un-exposed to air pollution and smoker
(regardless of disease status and genetic status)
3
3
Epidemiology Matters – Chapter 12
Prevalence of component causes
example lung cancer risk difference
Population 1
Population 2
Black = exposed air pollution
Dots = genetically determined
Hat = smoker
Exposed diseased = 6 = 60% risk
Unexposed diseased = 2 = 20% risk
Risk difference: 60% – 20% = 40%
Exposed diseased = 3 = 30% risk
Unexposed diseased = 2 = 20% risk
Risk difference: 30% – 20% = 10%
Interpretation: 40 cases of lung cancer are
associated with ambient air pollution per
100 exposed
Interpretation: 10 cases of lung cancer are
associated with ambient air pollution per
100 exposed
Epidemiology Matters – Chapter 12
Prevalence of component causes
example lung cancer interpretation

Two studies asked the same question

Both are internally valid studies because the exposed and unexposed are
comparable on genetic determinism
 Population 1: the causal effect is a risk difference of 40%
 Population 2 : the causal effect is a risk difference of 10%
Why do these two causal effects differ?

Prevalence of people exposed to ambient air pollution is the same in both studies

Prevalence of genetic determinism is same in both studies

The reason that these two causal effects diverge is the different prevalence of
smoking between the two populations
Epidemiology Matters – Chapter 12
Prevalence of component causes
example lung cancer interpretation

When two causes interact the measure of association for the effect of one cause
on the outcome will differ across levels of the second cause

Air pollution example
 Ambient air pollution and smoking are causal partners within the same
sufficient cause
 Prevalence of one of them (smoking) influences the causal effect of the other
(ambient air pollution) on the outcome (lung cancer)
 We would therefore expect the causal effect of ambient air pollution on lung
cancer to differ across population where prevalence of smoking also differs
 Therefore, there is no one causal effect for all populations; the causal effect is
dependent on prevalence of component causes in each population
 Therefore, the result from one study will be externally valid to populations in
which the distribution of component causes of exposure is similar to the study
sample
Epidemiology Matters – Chapter 12
1.
2.
3.
4.
5.
6.
7.
8.
Validity, four stages
Introduction to external validity
Prevalence of component causes
Causation and study design
External versus internal validity
Randomized control trials
Representative samples
Summary
Epidemiology Matters – Chapter 12
Causation and study design
 The magnitude of an association will be applicable
beyond our study to the extent that the distribution
of causal partners of exposure is similar in the
population
 If we want to identify a cause of disease, should it be
a cause absolutely and in all types of populations?
Epidemiology Matters – Chapter 12
Causation and study design
 Cause: a factor that was necessary for that disease to occur in an
individual at that time; most causes are insufficient and
unnecessary in isolation
 Causal effect: epidemiology studies populations; therefore we
focus on the effect of causes
 We document an association between those who embody cause
(exposed) and those who do not (unexposed); this is context
specific and dependent on prevalence of component causes
 Therefore, understanding a cause in context of causal partners is
central to theory, design, and analysis
Epidemiology Matters – Chapter 12
1.
2.
3.
4.
5.
6.
7.
8.
Validity, four stages
Introduction to external validity
Prevalence of component causes
Causation and study design
External versus internal validity
Randomized control trials
Representative samples
Summary
Epidemiology Matters – Chapter 12
External vs. internal validity
 Internal validity is a prerequisite to external validity
 To achieve internal validity we need to design a study with narrow
population of interest and minimize non-comparability
 The resulting sample may not reflect broader swath of population
beyond underlying population of interest
 The more narrow a sample becomes - due to strict inclusion and
exclusion criteria for internal validity - the less external validity it
may have if causal partners of exposure have differing prevalence in
study compared with other populations
Epidemiology Matters – Chapter 12
External vs. internal validity
 Balancing external and internal validity is a a trade off
 To build a scientific argument for causal effect of
exposure on outcome, we select study design and assess
internal validity of causal question
 After causal effect of exposure is established in narrow
population, we expand the causal question to ask
 How often?
 Among whom?
 Under what conditions?
Epidemiology Matters – Chapter 12
1.
2.
3.
4.
5.
6.
7.
8.
Validity, four stages
Introduction to external validity
Prevalence of component causes
Causation and study design
External versus internal validity
Randomized control trials
Representative samples
Summary
Epidemiology Matters – Chapter 12
An example, RCT
Question: Does weight-loss drug reduce obesity among school-age children?
Study details:

Recruit children for randomized drug trial with body mass index 25< (BMI) < 40

Exclude children with diabetes

Parents must be fully participatory (monitoring children’s drug regime and attend
study clinic once per week)

Baseline survey and monthly measurements

Children randomized to receive weight loss drug or placebo

Follow-up over two years
Study results:

Mean BMI among drug group declines 31.5 to 26.7 (4.8 points BMI)

Mean BMI among placebo group declines from 31.4 to 28.5 (2.9 points BMI)

Reduction of 1.9 (95% CI 0.9 – 2.9) more points of BMI in drug group than placebo
group
Conclusion: Weight-loss drug reduced obesity among school-age children
Epidemiology Matters – Chapter 12
An example, RCT
Questions to ask about external validity
 Are these results externally valid to a broader population all
overweight children in Farrlandia?
 What about overweight children in other places?
 What information would we need to know in order to inform
this issue?
Epidemiology Matters – Chapter 12
An example, RCT
Are we confident of a causal effect in the study?
 Can only be externally valid if internally valid
 Good reason to conclude that results obtained are approximation of causal
effect of drug for population
Consider characteristics of population from which participants were drawn
 Good adherers to study protocol
 Diabetes-free
 Actively participating parents
 There is evidence that drug is effective in reducing BMI
Consider characteristics of larger population to assess external validity
 Does action of drug interact with other factors?
 Do other factors have a different prevalence in general the general population
of overweight children who may be prescribed the drug?
1.
2.
3.
4.
5.
6.
7.
8.
Validity, four stages
Introduction to external validity
Prevalence of component causes
Causation and study design
External versus internal validity
Randomized control trials
Representative samples
Summary
Epidemiology Matters – Chapter 12
Example, representative sample
Question: Do sales tax on sugar-sweetened beverages reduce obesity among children
aged 7 to 13?
Study details:

Enumerate all school-age children in Farrlandia

Take random sample of 1,000 eligible Farrlandians who are between age 7 to 13
and live in Farrlandia

Measure BMI before tax goes into effect

Measure BMI after tax across a two-year period
Study results:

Mean BMI school-age children prior to the tax = 26.7 (95% C.I. 24.2-29.3)

Mean BMI of school-age children was 24.3 (95% C.I. 23.7-24.9) two years after tax
Conclusion: Tax lowered mean BMI among school-age children
Epidemiology Matters – Chapter 12
Example, representative sample
Snowtown is considering a similar tax
 Are Farrlandian results externally valid to Snowtown?
 What information would we need to know about Farrlandians
and Snowtownians?
Epidemiology Matters – Chapter 12
Example, representative sample
Are we confident of a causal effect in the study?
 Internal validity: If school lunches changed to healthier offerings
during study period we would not make a causal claim that tax
reduced BMI
What are potential causal partners of soda tax?
 External validity
 Soda availability is component cause
Is the distribution of causal partners similar across populations?
 Soda is plentiful in Farrlandia
 Soda is hard to find in Snowtown
Epidemiology Matters – Chapter 12
1.
2.
3.
4.
5.
6.
7.
8.
Validity, four stages
Introduction to external validity
Prevalence of component causes
Causation and study design
External versus internal validity
Randomized control trials
Representative samples
Summary
Epidemiology Matters – Chapter 12
Seven steps
1.
Define the population of interest
2.
Conceptualize and create measures of exposures and health
indicators
3.
Take a sample of the population
4.
Estimate measures of association between exposures and health
indicators of interest
5. Rigorously evaluate whether the association observed
suggests a causal association
6.
Assess the evidence for causes working together
7.
Assess the extent to which the result matters, is externally valid,
to other populations
Epidemiology Matters – Chapter 1
38
epidemiologymatters.org
Epidemiology Matters – Chapter 1
39