Lecture 20: Non-experimental
studies of interventions
•
•
•
•
•
•
•
•
Describe the levels of evaluation (structure, process,
outcome) and give examples of measures of each level
Describe the applications of cohort and case-control
designs to the evaluation of interventions.
Describe advantages and disadvantages of randomization
versus:
- Historical controls
- Simultaneous, non-randomized controls
Describe the following quasi-experimental designs:
- Time series (trend) design
1
- Non-equivalent control group design
Design of an intervention study
• Study objectives:
– Define intervention
– Define target population
– Define evaluation measures
• Study design:
– Experimental
– Non-experimental
2
Levels of evaluation
• STRUCTURE:
– Drugs, devices, staff, equipment needed to provide
intervention
• PROCESS:
– Interaction between structure and patient/client
– Adherence/compliance
• OUTCOMES:
– Expected or unexpected results, positive or negative,
e.g.:
• Death, disease, disability
• Attitudes, behaviors
• Costs
3
Levels of evaluation
• Create hypothetical diagram linking
structure, process, and outcome
• Based on goals of study, select measures of
structure, process, and/or outcome
4
Levels of evaluation: example
• Hypothetical diagram:
– HIV/AIDS educational intervention for drug
injectors (describe planned structure)
 Process (attendance/quality of participation)
Outcome 1: Improved knowledge/attitudes
Outcome 2: Lower risk behaviour
Outcome 3: Lower HIV incidence rate
5
Levels of evaluation
• Example:
– Exercise program to reduce CHD risk
• STRUCTURE?
• PROCESS?
• OUTCOMES?
6
Epidemiological observational study
designs
• Cohort and case-control studies
• Independent and dependent variables:
• Studies of risk factors:
– independent variable (exposure): risk factor
– dependent variable: disease
• Studies of interventions:
– independent variable (exposure): intervention
– dependent variable(s): selected “outcomes” (could be
measures of process and/or outcomes)
7
Cohort study
•
Study population:
–
–
Cohorts with and without “exposure” to
intervention (or different levels of exposure)
Control (unexposed) cohort - concurrent or
historical
•
•
confounding by changes over tine in patient
population, aspects of treatment other than
intervention; measurement of confounders
Follow-up to measure outcomes
8
Cohort study
• Selection of controls: could they receive
either treatment?
• Example: medical vs surgical treatment of
CHD
• Some sources of bias:
– Selection bias
– Information bias: detection bias, other
– Confounding: by indication, other
9
Examples of cohort studies
• Effectiveness of new cancer treatment
– Historical controls
• Do HMOs reduce hospitalization in terminal cancer
patients, during 6 months before death?
– Administrative databases and tumor registry from
Rochester NY
– Cancer deaths in 100 pairs of HMO members and nonmembers
– Matched by age, cancer site, months from diagnosis to
death
10
11
Case-control study
• Study population:
– Cases (with outcome)
– Controls (without outcome) Limited to single,
categorical outcome
• Data collected on prior “exposure” to intervention
• Some sources of bias
– Selection bias
– Information bias
– Confounding: by indication, other
12
Case-control study: Examples
• Screening programs:
– screening Pap test and invasive cervical cancer
– screening mammography and breast cancer
deaths
– screening sigmoidoscopy and colon cancer
deaths
• Vaccine effectiveness (e.g., BCG)
• Neonatal intensive care and neonatal deaths
13
Quasi-experimental study designs
• Investigator has “some control” over timing
or allocation of intervention
– Non-randomized or quasi-randomized trials
– Non-equivalent control group designs:
• pre-test and post-test
• post-test only
– Time series designs
• single or muliple
14
Diagramming Intervention Study
(Evaluation) Designs
Campbell and Stanley
•
X = program
•
O = measurement
•
R = randomization
15
Randomized (Experimental)
Designs
• Randomized pre-test post-test control group
design
R O1 X O2
R O3
O4
• Post-test only control group design
R X O1
R
O2
16
Some Weak Observational
Designs: Cross-sectional
• One-shot case-study
X O
• Static group comparison:
X O1
O3
17
Some Weak Observational
Designs: Longitudinal
• Before-after (pre-post) study
O1 X O2
18
Some quasi-experimental designs:
with control/comparison group
Pre-test post-test non-equivalent control
group design
O1 X O2
O4
O3
Recurrent institutional cycle
X O1
O2 X O3
19
Health insurance in Quebec
• 1961: universal hospital insurance
– included ER care for accidents
• 1970: universal health insurance (Medicare)
– added MD care including hospital outpatient clinics and
ERs
• Population surveys before and after
• Effects on:
– use of physician services by general population
– physician workload
– use of emergency rooms
– hospitalization and surgery
20
MD visits/person/year by income
(household surveys)
8
7
6
5
Pre
Post
4
3
2
1
0
All visits <3000
3000-
5000-
9000-
15000+
21
MD visits/person/year (household surveys)
6
5
4
Pre
Post
3
2
1
0
All visits
Office
ODP/ER
Home
22
MD visits/person/year by income
(household surveys)
8
7
6
5
Pre
Post
4
3
2
1
0
All visits <3000
3000-
5000-
9000-
15000+
23
% adults with cough 2+ weeks who consulted
MD (household surveys)
70
60
50
40
Pre
Post
30
20
10
0
<$5000
$5000-
$9,000
Total
24
% children (<17) with tonsilitis or sore throat
and fever who consulted MD
(household surveys)
80
70
60
50
Pre
Post
40
30
20
10
0
<$5000
$5000-
$9,000
Total
25
% pregnancies with visit in first trimester
(household survey)
60
50
40
Pre
Post
30
20
10
0
<$5000
$5000-
$9,000
Total
26
Unsuccessful
Office/answering
machine
Spoke to MD
70
60
50
40
30
20
10
0
Tried to contact
% Tried to contact MD before ED visit;
of these, % successful (6 hospital sample)
Pre
Post
27
Examples of pre-post nonequivalent control group design
• Stanford 5-city study of CHD prevention
• Intervention included mass media education
and group interventions for high-risk
• 5 cities selected - similar characteristics
– those with shared media market were allocated
to intervention
– isolated cities allocated to control group
28
Time series designs
Time series desgn
O1 02 O3 X O4 O5 O6
Multiple time series design
O1 O2 O3 X O4 O5 O6
O7 O8 O9
O10 O11 O12
29
Example of time series study:
Tamblyn et al, 2001
• Evaluation of prescription drug cost-sharing
among poor and elderly
• Methods:
– Trend study: Multiple pre- and postmeasurements
– Cohort study:
30
Source: Tamblyn et al, JAMA 2001, 285(4): 421-429
31
Source: Tamblyn et al, JAMA 2001, 285(4): 421-429
32
Time-series design: Home care in
terminal cancer
• Evaluation of home-hospice programme in
Rochester, NY
• Expansion of home-care benefits in 1978
• Hypothesis: home-hospice care in last
month of life reduces hospital days and
costs
• Data sources: Linkage of tumor registry and
health insurance claims databases
33
34
35
Differences between quasi-experimental and
epidemiological cohort study designs
• Quasi-experimental designs often use ecological
rather than individual level of measurement
• Serial cross-sectional studies over time vs followup of individuals:
– advantages and disadvantages?
36