Foundations of
Research 
Quasi-experimental & field studies
This is a PowerPoint Show

Click “slide show” to start it.

Click through it by pressing any key.

Focus & think about each point; do not
just passively click.

To print:

Click “File” then “Print…”.

Under “print what” click “handouts (6
slides per page)”.
© Dr. David J. McKirnan, 2014
The University of Illinois Chicago
McKirnanUIC@gmail.com
Do not use or reproduce without
permission
Ongoing field studies show Baboons
to learn and transmit “culture”.
Click for NY Times article.
1
Foundations of
Research
Many important issues cannot be studied with true
experiments.
This module addresses four topics best addressed with
quasi-experiments.
Naturally occurring events / case studies
Interventions within a single group
Experiments with non-equivalent groups
Time series designs & field observations
2
Foundations of
Research
3
Quasi-experiments: how do we research…
Naturally occurring events / case studies
Interventions within a single group
Experiments with non-equivalent groups
Time series designs & field observations
Group
Naturally occurring event or social change
Observe
Foundations of
Research
Group
4
Naturally occurring events
Naturally occurring event or social change
Observe
 It is important to understand the consequences of
phenomena that are naturally occurring.
 It is easy to think of examples:



Social, cultural or political events

War, 9/11-like events, immigration, torture.

Policy changes; educational or civic policies, gay marriage

Rapid economic change…
Natural events

Weather-related or other natural disasters

Climate change…
Behavioral events

Crime victimization

Childbirth…
Foundations of
Research
Group
5
Naturally occurring events
Naturally occurring event or social change
Observe
 Research on naturally occurring events roughly falls
into three categories:

Simple description

Predictors (correlates) of outcomes or consequences

Potential interventions or policy changes
6
Naturally occurring events
Foundations of
Research
Group
Naturally occurring event or social change
Observe
 Research on naturally occurring events…

Simple description

What is the nature – size, scope, intensity – of the
event?

It is ‘positive’ or ‘negative’?

What are the consequences for those affected?

Economic, social or cultural; are social structures
damaged or enhanced?

Individual; psychological adjustment or growth,
physical harm, trauma
Foundations of
Research
Group
7
Naturally occurring events
Naturally occurring event or social change
Observe
 Research on naturally occurring events roughly falls
into three categories:

Simple description

Predictors (correlates) of outcomes


What personal or social factors allow individuals or
communities to benefit from ‘positive’ events?

What personal or social factors create vulnerability to
adverse outcomes?

What confers resilience in the face of uncontrollable
events; e.g. community support, economic resources
Potential interventions or policy changes

To prevent future occurrences

To lessen adverse outcomes
Foundations of
Research
Example: Child soldiers in Sierra Leone.
8
Description
EXAMPLE
 During Sierra Leone’s civil war children as
young as 7 were recruited as soldiers.
•
In this sample M age of captivity was 10, average duration = 5 years.
•
88% witnessed violence or warfare
•
28% killed a stranger or loved one.
•
37% of females and 6% of males were victims of rape.
•
7% experienced all three exposures.
 Most were resettled in their original villages or with relatives.
 Many returned child soldiers reported PTSD or other
psychological symptoms
• Anxiety, depression, hostility
• Low confidence and prosocial attitudes.
Betancourt, Theresa et al. (2010). Sierra Leone’s Former Child Soldiers: A Follow-Up Study of Psychosocial
Adjustment and Community Reintegration. Child Development, 81 (4), Pp. 1077–1095. Click for article.
Foundations of
Research
Child soldiers in Sierra Leone.
Description
Predictors / correlates of outcomes
EXAMPLE
 Actions during the war were strongly related to negative
outcomes
•
Wounding or killing another person
•
Witnessing violence
•
Victim of rape
 Family acceptance upon reentry was the key variable
associated with resilience & recovery.
•
Community acceptance was less important short-term, more
important long-term
 Family acceptance was weaker for those who were…
•
Older, male, engaged in or witnessed violence, or were raped.
•
Thus, acceptance was lower for those who needed it most.
9
Foundations of
Research
Child soldiers in Sierra Leone.
10
Note:
 “Predictors’ here are simply variables that
Description
correlate with the outcome.
Predictors / correlates
outcomes
 Causeofand
effect can be difficult to
EXAMPLE
 Actions during the
war were strongly related to negative
determine.
outcomes
 Did family acceptance cause some
•
formerperson
soldiers to have better outcomes?
Wounding or killing another
•

Witnessing violence
•
Victim of rape
Or, are soldiers who are doing better
more easily accepted by their families?
 Family acceptance upon reentry was the key variable
associated with resilience & recovery.
•
Community acceptance was less important short-term, more
important long-term
 Family acceptance was weaker for those who were…
•
Older, male, engaged in or witnessed violence, or were raped.
•
Thus, acceptance was lower for those who needed it most.
Foundations of
Research
Child soldiers in Sierra Leone.
11
EXAMPLE
Description
Predictors of outcomes
Potential interventions or policy changes
 The predictor / correlation evidence has shown some paths
toward interventions
•
International Rescue Committee has developed programs specifically
targeting family & community integration
•
Programs for former child soldiers include Burundi, Democratic
Republic of Congo, Ivory Coast, Liberia and Uganda.
•
UNICEF, the UN and other organizations are resettling soldiers
 There are only scattered data on the long-term effect of these
programs.
 Here are interviews with Ishmael Beah, a child soldier in Sierra
Leone who wrote a biography of his experiences.
An excerpt.
The complete interview.
Foundations of
Research
Group
12
Naturally occurring events
Naturally occurring event or social change
Observe
 Let’s think about the study of naturally occurring
events from a research perspective
 “Naturally Occurring” means events that are not
controlled by a researcher.
Group
13
Naturally occurring events
Foundations of
Research
Naturally occurring event or social change
 No control over
who is exposed
to the event
 Possible
control over
who is selected
for the research
sample
May compromise
both internal and
external validity.
 Event not controlled /
manipulated.
 Not a true Independent
Variable
 Often no control group
Significant threat to
internal validity
Observe
 May or may not have
control over
measures.
 Archival measures –
medical records,
climate data, crime
reports – may not
assess exactly what
the study needs.
 Survey or other posthoc measures can
address hypotheses.
 Heuristic value of field studies: generating hypotheses
for later experimental study or to confirm controlled
data in a “real world” setting.
Foundations of
Research
Examples of naturally occurring
events: Descriptive data
Events
Outcomes
Natural disaster / stressor
The 1984 San Francisco
earthquake
Coping responses to a major
stressor
Trauma
EXAMPLE
Iraq / Afghanistan service
PTSD & response to
treatment
Historical event
2007 – 2010 economic
contraction
Voting patterns
Publicity / cultural event
Negative results from The
Women’s Health Initiative
study of hormone
replacement
Health decisions
14
Foundations of
Research
Examples of naturally occurring events:
Post-hoc ‘predictors’ (correlates)
Events
Outcomes
Natural disaster / stressor
The 1984 San Francisco
earthquake
Coping responses to a major
stressor
EXAMPLE
Post-hoc surveys showed that:
Trauma
PTSDpredictor
& response
treatment

Proximity
to the
event was the best
oftostress
Iraq
/ Afghanistan
service
reactions
Historical
event
 Previous
psychiatric history was also predictive
2007
2010 economic
 – Those
with anxiety or depressionVoting
histories
had a significantly
patterns
contraction
stronger stress reaction.
 Social/ cultural
support “buffered”
the effect of the earthquake on
Publicity
event
stress
Negative results from The
Health decisions
 People
who Initiative
perceivedstudy
themselves to have strong social support
Women’s
Health
networks
fared better than those who did not.
of hormone
replacement
15
Foundations of
Research
Examples of naturally occurring events: Post-hoc ‘predictors’
(correlates)
Events
Outcomes
Natural disaster / stressor
The 1984 San Francisco
earthquake
Coping responses to a major
stressor
Trauma
EXAMPLE
Iraq / Afghanistan service
PTSD & response to
treatment
Historical
Post-hoc event
studies have shown that:

As–with
earthquake data, both proximity to the event
2007
2010the
economic
Voting patterns
contraction
previous psychiatric history were strong predictors of
PTSD.
Publicity
/ cultural event
 Traumatic brain injury is a major co-factor in symptom
Negative results from The
Health decisions
severity.
Women’s Health Initiative study
 of
Immediate
‘tela-treatment’ (web links from mental health
hormone replacement
professionals to the field) could moderate symptoms
16
Foundations of
Research
Examples of naturally occurring events: Post-hoc ‘predictors’
(correlates)
EXAMPLE
Descriptive & hypothesis-testing research has shown that:
Outcomes
Events
Voters generally vote more conservative
during times of
economic stress.
Natural
disaster / stressor
 <15% of voters correctly estimate the huge gulf between
The 1984 San Francisco
Coping responses to a major
the economic status of the top 10% vs. everyone else.
earthquake
stressor
 Voting rates among those who have lost their economic
Trauma
footing – ethnic minorities, the poor, young people – have
PTSD & response to treatment
gone
down, notservice
up.
Iraq / Afghanistan
Historical event
2007 – 2010 economic
contraction
Voting patterns
Publicity / cultural event
Negative results from The
Women’s Health Initiative study
of hormone replacement
Health decisions
17
EXAMPLE
Foundations of
Research
Examples of naturally occurring events: Post-hoc ‘predictors’
(correlates)
Post-hoc surveys & prescription patterns have shown that:

The great majority of post-menopausal
women stopped
Events
Outcomes
taking hormone replacement therapy altogether.
Natural
disaster
/ stressor
 Secondary
analyses
showed that;
The
San Francisco
Coping
responses
to a major
1984
Estrogen
was associated with
a variety
of health
risks
earthquake
stressor
 Progesterone is protective for cancer
and other
diseases.
Trauma
& response
to treatment
Iraq
Some
researchers
and cliniciansPTSD
question
whether
the low
/ Afghanistan
service
absolute risk of estrogen justifies discontinuation in
Historical
event
severe menopause.
– 2010
economicmay identify which women can(not)
2007
Further
research
Voting patterns
contraction
tolerate estrogen.
Publicity / cultural event
Negative results from The
Women’s Health Initiative
study of hormone
replacement
Health decisions
18
Foundations of
Research
‘One-shot’ survey study; Consumer Reports
psychotherapy survey.
19
Seligman, M. E. P. (1995). The effectiveness of psychotherapy: The Consumer Reports study.
American Psychologist, 50, 965-974.
 Despite experimental research showing that psychotherapy
EXAMPLE
is effective, there are few field studies of actual consumers.
 Many experimental studies from the 90s to today use small,
carefully selected samples who are given ‘lab’-like therapy
interventions.
This is a major challenge to external validity.
 Seligman set out to answer:
 Does psychotherapy “work” from a consumer view?
 Who gets therapy / what does it consist of?
 Do consumer responses vary by type of treatment?

This study remains one of the most often cited and
controversial studies in the therapy outcome literature.
Foundations of
Research
Consumer reports survey, 2
20
Research approach:
 One shot case study / survey
 Consumers given post-hoc survey of any therapy
EXAMPLE
experiences
Sampling frame:
 Any therapy or psychological service user in the
previous year
 We have little information about the general population
of therapy users.
 Thus, we can never know if this sample is
representative
Sampling procedure:
 4,100 Consumer reports readers responding to “in
magazine” mail-back survey form
Foundations of
Research
21
Consumer reports survey, 3
Entire population of therapy clients
U.S. Therapy clients
Proportion of therapy clients who
read Consumer Reports
EXAMPLE
Proportion of Consumer reports
readers willing to return the survey
CR Readers
Key self-selection biases
Returned
survey
 Of the complete therapy client
population, only a small % join
Consumer Reports.
 Of therapy clients who join Consumer Reports, only a %
were willing to complete a detailed survey.
 Neither of these percentages can be known, since we do not
have data on people who did not return the survey.
 This source of bias may threated External Validity, since we
cannot know how well these data generalize.
 The very large sample provides confidence, however.
Foundations of
Research
Consumer reports survey, 3
Potential predictors of outcomes

Gender, type & duration of treatment, medications.
Negatives:
EXAMPLE
 Cursory outcome measures: “satisfaction” & “helped
with my problem”
 This may threaten Internal Validity, since we cannot
ensure we are capturing the outcomes (e.g., change in
mental health) we are interested in.
Positives:
 Huge, national sample
 Anonymous, 3rd party data collection
 “Real world” assessment of product quality
22
Foundations of
Research
Consumer reports survey, 4.
Descriptive findings:
 People generally felt
EXAMPLE
positive about their
therapy experience.
Analysis of predictors
 People who got more
treatment did better.
 Demographic predictors
(age, gender…had modest
or no effects.
 For peoples’ presenting
problem(s) all specialists
did about the same.
23
Foundations of
Research
Consumer reports survey, 4.
 Analyses of predictors
EXAMPLE
For work and social
domains:
 Marriage counselors
were the least
effective
 Physicians were only
slightly better.
24
Foundations of
Research
Consumer reports survey, 5.
 Analyses of predictors
EXAMPLE
For personal domains
(self-esteem…):
 Again marriage
counselors were the
least effective
25
Foundations of
Research
Consumer reports survey, 6.
Bottom line
 Most therapy clients are satisfied
EXAMPLE
with their treatment.
 More treatment is better.
 Outcomes are not affected by
age, gender, ethnicity…
 Mental health professionals get
significantly better results than
do marriage counselors or
physicians
26
Foundations of
Research
Consumer reports survey, 7.
Interpreting survey findings:
 Since these are simple descriptive
EXAMPLE
data we cannot determine why
these outcomes occurred.
Why might marriage counselors
have done worse?
 Could marriage counselors be
more poorly trained?
 Might less competent people enter the marriage
counseling profession?
 Might marriages be really hard to fix?
What data would allow you to test each of these
alternate hypotheses?
27
Foundations of
Research
Naturally occurring events
28
Useful when:


SUMMARY

One An experiment is not possible
There cannot be a control group
“Pre-” measures not possible or practical
Chief virtues:




Describe naturally occurring or uncontrollable socially or
politically important events
Provides “real world” look at processes that are typically
studied in experiments
Archival data on potential predictors can help interpret the
findings / “control” some alternate interpretations.
Data may suggest interventions.
Liability:

lack of control group  multiple threats to internal validity

No pre-measure makes interpretation (e.g., of change…) difficult.
Foundations of
Research
Quasi-experiments: Existing groups
Naturally occurring events / case studies
 Single group interventions
Experiments with non-equivalent groups
Time series designs
29
30
One group pre- post-test
Foundations of
Research
Group
Observe1
Selected or
convenience
sample.
Intervention or event
Baseline Assessment
May or may not have control
over measures (e.g., surveys
v. archival measures).
Observe2
Outcome Assessment
Typically controllable,
but may be archival.
Event or intervention May or may not be controllable
by researcher, e.g., policy change.
Uses:
 Educational & social environments
 Political or health policy change
 Not feasible to have a control group
 System-wide intervention / social change (school,
public health campaign..)
Group
31
One group pre- post-test
Foundations of
Research
Observe1
Key design
Observe
1
Intervention or event
Observe2
feature:
no control group.
Observe2
Confound
Threats to internal validity (confounds):
 History
Historical / cultural events occur between
baseline & follow-up.
 Maturation
Individual maturation or growth occurs
between baseline & follow-up.
 Reactive measures
People respond to being measured or
being a measured a second time.
 Statistical regression
Extreme scores at baseline “regress” to a
more moderate level over time.
 Mortality / drop-out
People leave the experiment nonrandomly (i.e., for reasons that may affect
the results…).
Group
32
One group pre- post-test
Foundations of
Research
Observe1
Intervention or event
Observe2
Observe1
Confound
Observe2
Threats to internal validity (confounds):
 History
Historical / cultural events occur between
baseline & follow-up.
 Maturation
Individual maturation or growth occurs
between baseline & follow-up.
 Reactive measures
People respond to being measured or
being a measured a second time.
 Statistical regression
Extreme scores at baseline “regress” to a
more moderate level over time.
 Mortality / drop-out
People leave the experiment nonrandomly (i.e., for reasons that may affect
the results…).
Foundations of
Research
One group pre- post-test
EXAMPLE
Effects of HIV testing on sexual risk.
Question:
- Does HIV testing lead people to be
sexually safer?
Event:
- Self-referred HIV testing &
counseling
Sampling frame: - Participants in testing centers
Study structure: - Baseline retrospective interview at
testing session
- Follow-up interview 3 months later
Quasi-controls:
- Population characteristics as
potential predictors of group
differences in risk
Outcomes:
- Self-reports of sexual risk
33
Foundations of
Research
One group pre- post-test
Effects of HIV testing on sexual risk.
EXAMPLE
Findings:
- Significant shifts toward safety
- Few demographic predictors of change
Threats to internal validity
- Self-selection into testing group
- Mortality: non-random drop-out(?)
- History: general shift in norms & behavior during
study period
- Instrument change; people answer more
conservatively during a follow-up interviews
34
Foundations of
Research
One group pre- post-test
EXAMPLE
Effects of HIV testing on sexual risk.
 Thus a clinical follow-up design can provide
important clues about a possible
intervention…
 The constraints of this research approach
make interpretation difficult.
35
SUMMARY
Foundations of
Research
One group pre- post-test
Virtues:
 Provides more systematic data on naturally
occurring events
 Only possible design for system-wide
intervention evaluations.
 Pre-measure allows researcher to interpret
change & examine status of groups at baseline.
Liability:
 Lack of a control group
creates multiple threats
to internal validity:





History
Maturation
Statistical regression
Reactive measures
Mortality / drop-out
36
Foundations of
Research
37
Quasi-experiments: Existing groups
Naturally occurring events / case studies
Single group interventions
 Experiments with non-equivalent groups
Time series designs
Group
Observe1
Intervention or event
Observe2
Group
Observe1
Contrast group
Observe2
Group1
Group2
38
Non-equivalent experimental designs
Foundations of
Research
(No baseline)
Groups are not equivalent at
baseline, due to..
 Self-selection into the
experimental vs. control
groups
 Non-random assignment
into groups (e.g., first
people to show up in
experimental group…)
 Use of existing groups
 Participants are not blind
to which condition they
are in.
Intervention or event
Observe1
Contrast group
Observe1
Assessments may or may not
be controlled
 Survey or interviews
 Archival / existing data,
e.g., clinic records, grades
Intervention or event may or may not
be controlled by the researcher;
 Existing program
 Experimental intervention
 Naturally occurring event
39
Non-equivalent experimental designs
Foundations of
Research
Group
Observe1
Intervention or event
Observe2
Group
Observe1
Contrast group
Observe2
Non-equivalent groups
 Self-selection
 Non-random assignment
 Use of existing groups
 Participants not blind
Intervention & Assessments often
controlled by the researcher.
Observation1 used to
 Assess equivalence of groups at baseline
 Assess change: the key outcome
 Test for threats to internal validity:
Reactive measures
History, mortality effects
Regression effects
Similar to true
experimental
design, except for
non-equivalent
groups
Foundations of
Research
40
True Quasi-experiments
vs. Quasiexperiments
True
experiments
Participant
Participant Experimental
Recruitment Assignment Procedures
Experimental
Treatment or
Manipulation
Results
Group A 
Procedure A  Treatment 
Outcome
Group B 
Procedure A  Control 
Outcome
Sample
Randomly
selected
from target
population
Convenience
sample?
Probability
sample?
Randomly
assigned;
groups
identical at
baseline.
Self-selection?
Non-random
assignment?
Existing
groups?
Non-blind?
Procedures
Complete
Complete
 the
Unbiased
to groups
same assignment
control over
control over
exp. &
IV.
measures.
 for
Participant
and the
experimenter
blind
control
groups.
Might selfselected or
existing
groups require
different
procedures?
Naturally
Archival
occurring
measures?
event? (Not Existing
a true IV)
assessment?
Foundations of
Research
Example of a formal Quasi-experiment
EXAMPLE
Safer sex intervention for drug using, risky MSM





Multi-frame targeted sampling of gay/bisexual men
Intervention group: 6 90-min. group clinical sessions
Control group: 6 90-min. general group discussion sessions
Men randomized at first group meeting
Structured risk / attitude assessment at baseline, 3-, 6-, & 12month follow-ups.
41
Foundations of
Research
Quasi-experiment example
Sample selection
EXAMPLE
Safer sex intervention for drug using, risky MSM
 We do not have a sampling frame for this sub-group of MSM.
 We necessarily use a multi-frame convenience sample.
 We ask men to call for enrollment, so there is self-selection
into the sample altogether.
 However, men are randomly assigned to groups, to there is
no self-selection there.
42
Foundations of
Research
Quasi-experiment example
Sample selection
Self-selection into the sample lessens external validity; the
sample may be unlike the larger population.
EXAMPLE
Self-selection into groups would severely compromise internal
validity;
 Participants may choose a group for reasons that affect
thesex
results,
e.g., peoplefor
really
motivated
change
may
Safer
intervention
drug
using, to
risky
MSM
select the intervention group…
Since
weahad
random frame
assignment
to groups
internal
 We do not
have
sampling
for this
sub-group
of validity
MSM
was not compromised
for this reason.
 We necessarily
use a multi-frame
convenience sample
 We ask men to call for enrollment, so there is self-selection
into the sample altogether.
 However, men are randomly assigned to groups, to there is
no self-selection there.
43
Foundations of
Research
Quasi-experiment example
Group assignment
EXAMPLE
Safer sex intervention for drug using, risky MSM
 The groups are randomly assigned. (Not self-selected!)
 We try to convince participants that each arm of the
intervention (control vs. treatment) are equal, but they still
cannot be blind.
 Of course the interventionists cannot be blind.
44
Foundations of
Research
Quasi-experiment example
Procedures
EXAMPLE
Safer sex intervention for drug using, risky MSM
 The procedures are highly standardized, so they are
equivalent across group.
 All assessments are done with computer interviews, so the
measurement procedures are also equivalent.
45
Foundations of
Research
Quasi-experiment example
Treatment
EXAMPLE
Safer sex intervention for drug using, risky MSM
 We designed the experimental and control interventions, so
we have complete control.
 (Different if we were assessing a “naturally occurring”
therapy or health program.)
46
Foundations of
Research
Quasi-experiment example
Bottom line
EXAMPLE
Safer sex intervention for drug using, risky MSM
 A randomized controlled trial of a behavioral intervention has
both true- and quasi-experimental features.
 Groups cannot be perfectly equivalent
 Interpretation of findings have to take into account:
 Sampling methods
 Non-blind participants & interventionists
47
Foundations of
Research
True v. quasi-experimental designs
48
True experiments:
Quasi-experiments:
Emphasize internal validity
 Assess cause & effect (in
relatively artificial environment)
 Test clear, a priori hypotheses
Emphasize external validity
 Describe “real” / naturally
occurring events
 Clear or exploratory hypotheses
Participants randomly assigned
to exp. or control groups
 Participants & experimenter
Blind to assignment
Non-equivalent groups
 Existing groups
 Non-random assignment
 Participants not blind
 Self-selection
Full control may not be possible
Control study procedures
 Manipulate independent variable  May not be able to manipulate the
independent variable
 Control procedures & measures
 Partial control of procedures &
measures
EXAMPLE
Foundations of
Research
Example of a non-equivalent control
group design; condom distribution
Question:
Does condom ed. & distribution:
- increase safety
- increase sexual activity
Sampling frame:
- Schools in New York & Chicago
- Schools matched for SES, race, size
Intervention:
- Condom education & distribution in
High School health classes
Study structure:
- NY = intervention schools, Chicago
are contrast schools.
- Baseline  sexual health programming
 9 mo. Follow-up
Outcomes:
- Clinical measures: STIs
- Self-reports: sexual activity & safety
49
Foundations of
Research
Non-equivalent control group
EXAMPLE
Findings:

NY (intervention) students; lower STI rate, safer sex

NY and Chicago students; similar levels of sexual
activity

Thus; sexual health classes appeared to increase
safety without increasing sexual activity.
Internal validity?:

Reactive measures; Study is not blind; NY students
know they are the intervention group

Non-equivalent groups: Possible differences
between cities = unmeasured confounds
50
Foundations of
Research
Non-equivalent control group

Most common quasi-experimental approach.

Used where:


Some form of control or contrast group is possible
Groups cannot be equivalent:
SUMMARY





Virtue:



Participants cannot be blind re: group assignment
Random assignment not possible
Must use existing groups
Participants self-select into (or out of) groups.
Study natural / “real world” interventions
Contrast group lessens major threats to internal validity
Liability:

Using non-equivalent groups may introduce serious
unmeasured confounds.
51
Foundations of
Research
Click
Which of these is not a cause of nonequivalent groups?
A. Self-selection
B. Non-random assignment
C. Use of existing groups
D. Convenience sampling
E. Participants not blind
52
Foundations of
Research
53
Quasi-experiments: Existing groups
Naturally occurring events / case studies
Single group interventions
Experiments with non-equivalent groups
 Time series designs
Group
Measure1
M2
M3
M4
Intervention
or event
M5
M6 …
Foundations of
Research
54
Simple time series design
Group
Measure1
M2
M3
M4
M5
M6…
Group2
Measure1
M2
M3
M4
M5
M6…
Group3
Measure1
M2
M3
M4
M5
M6…

Examine / describe changes over time in one or more key
variables.

Describe or test hypotheses about group differences over
time.

Groups may be assigned, in a true experiment.
 … intervention groups with long-term follow-up
 Groups may be measured or naturally occurring.
 … age, gender or ethnic groups.

Longer time-frame yields more valid & interpretable data.
Foundations of
Research
Simple time series design; Are rates of
childhood obesity lessening?
2003  2012 data
 Older kids (2 - 19):
no change
 Toddlers appear to
EXAMPLE
be doing better.
 Supports
effectiveness of
recent infant
programs.
Longer time frame: 1999  2012
 Older kids no still show no change
 Toddlers only look better because of a spike in 2003.
 Looking back to 1999 shows a flat line with lots of variance.
http://www.nytimes.com/2014/04/15/health/obesity-studies-tell-two-stories-both-right.html?
Original article: http://archpedi.jamanetwork.com/article.aspx?articleid=1856480
55
EXAMPLE
Foundations of
Research
Simple time-series design; Childhood obesity.
Time series designs
 Provide much more sensitive data than simple “one
shot” measurements.
 But are still sensitive to the length and nature of their
time frame.
56
Foundations of
Research
Group
Interrupted time series design
Measure1
M2
M3
M4
M5
Intervention
or event


Test effect of intervention or event on ongoing
series of measurements.
Intervention may be experimental or observed
 Policy shift, e.g., educational policy
 Uncontrolled event; e.g., 9/11/01, Media event
 Assessments may be experimental or archival
 Successive cross-sectional surveys
 Traffic data, clinic or crime reports, test scores
M6 …
57
Foundations of
Research
Group
Measure1
M2
Multiple baseline


58
Time series designs
Demonstrate stable effect
 long-term crime rates
 disease prevalence
 economic performance…
Or steady rate of change…
M3
M4
M5
M6 …
Intervention
or event
Hypothesis, tested by:


Shift in stable rate after
intervention.
Increase / decrease in rate
of change after intervention
Foundations of
Research
Group
59
Time series designs
Measure1
M2
M3
M4
M6 …
M5
Intervention
or event
Multiple baseline
Hypothesis; tested by:
4

Shift in stable rate after
intervention
3.5
3
2.5
2
Intervention or event
1.5
1
0
1
2
3
4
5
6
7
8
9
10
Foundations of
Research
Group
60
Time series designs
Measure1
M2
M3
M4
M6 …
M5
Intervention
or event
Multiple baseline
Hypothesis; tested by:
4

Shift in rate of change after
intervention
3.5
3
2.5
2
Intervention or event
1.5
1
0
1
2
3
4
5
6
7
8
9
10
Foundations of
Research
Group
Measure1
61
Time series designs
M2
M3
M4
M5
M6 …
Intervention
or event
Multiple baseline
Hypothesis;




Shift in stable rate.
Increase / decrease rate of change
Threats to internal validity:

Sensitive to very local history

Single group possibly prey to confound
Advantage for internal validity

Eliminates carryover effects of repeated measurement

Tests maturation, history, reactive measurement, etc.
Foundations of
Example of interrupted time series:
Research
Shift in Baboon culture.
62
Question:
Do baboon troops develop and transmit
learned “culture”?
EXAMPLE
Baseline:
Click for original article in PLOS Biology
Long-term observational data on
aggressiveness in a specific baboon
troop.
Intervention:
 Tuberculosis outbreak due to infected food.
 Dominant / aggressive males fed first
Image: Shutterstock.
 are selectively infected
 are naturally culled from troop

Naturally occurring event ongoing field study.
Foundations of
Research
Baboon culture: findings
63
Outcome measures: Standardized indices of
aggression & dominance.
Core findings:
EXAMPLE
 With dominant males gone,
remaining males showed more
cooperative behavior
 As new males entered the
troop over time they were
socialized into the cooperative
culture
Robert Sapolsky
After a tuberculosis outbreak killed the
most aggressive male baboons, the
remaining members showed a greater
willingness to foster a more patriotic
spirit.
 Despite many new males over
time, the troop remained
cooperative for over 20 years.
NY Times; Click image for article
Click for an NYC Radio Lab description of
the study; Is warfare innate in primates?
Foundations of
Research
Example: Interrupted time series data
The “Magic Johnson effect” on HIV testing
Question: Does a celebrity or “role model” getting HIV
EXAMPLE
affect others…?
Examining HIV testing
rates before and after
Magic’s announcement
allows us to test the
effect of a naturally
occurring event.
Magic Johnson announces that he
is HIV-positive in 1991.
Click for YouTube video of the announcement.
64
Foundations of
Research
Example: Interrupted time series data
The “Magic Johnson effect” on HIV testing
Data: Archival records of HIV tests reported to CDC,
EXAMPLE
collected monthly

Data show stable baseline over multiple observations

Timing of intervention precise relative to data collection
Intervention: Magic reports infection on national TV.

Uncontrollable, “naturally occurring” event

Tests hypothesis re: modeling effects in health behavior
Core Finding: Initial spike in testing rates, followed by
leveling off at higher base rate.

Initial increase expected

Hypothesis tested by longer-term shift in testing rates
65
Foundations of
Research
66
Example of time-series data: “Magic” / HIV effect.
Time-series data showing shift in
HIV testing after Magic’s
announcement.
Magic’s
Announcement
EXAMPLE
Initial spike
New, higher
base rate
Low & variable
baserate of testing
Multiple (monthly)
measures.
Tesoriero, J.M., Sorin, M.D., Burrows, K.A., LaChance-McCullough, M.L. (1995). Harnessing the heightened public awareness of celebrity HIV
disclosures: “Magic” and “Cookie” Johnson and HIV testing. AIDS Education and Prevention, 232-250.
Foundations of
Research
67
Multiple time series data
Group 1
Measure1
M2
M3
M4
M5
M6 …
Group 2
Measure1
M2
M3
M4
M5
M6 …
Groups typically formed by blocking
variable measured post-hoc;
Intervention
or event
Hypothesis; tested by an
interaction of the blocking variable
by the repeated measure:
 Health claims in NYC v. other
cities post- 9/11/01
 Younger v. older voting patterns
post- Iraq invasion
 Heterosexual v. gay HIV testing
rates post- Magic Johnson media
event.

Over repeated measurements,

Does one group change while
the other remains consistent?

Do the groups change at
different rates?
Foundations of
Research
Blocking variables in the HIV testing data.
68
Core questions:
EXAMPLE


Heterosexuals and Ethnic minorities had low HIV
testing rates

Perceive HIV as a “white gay” problem?

They may lack resources or venues for testing.
Will having a prominent African-American Heterosexual
disclose HIV+ status may change those perceptions?
Hypotheses:

Heterosexuals will respond more strongly than will
gay/bisexual men.

African-American and Latino men and women will
respond most strongly.
Foundations of
Research
Testing blocking variables: Gay / IDU data.
69
EXAMPLE
Risky men &
injection drug users:
High baseline, high
variability.
Gay / bisexual men:
less variable, but
low baseline.
Risky men & IDUs:
slight increase, high
variability.
Gay & bisexual men:
no change.
Foundations of
Research
Testing blocking variables: Heterosexuals
70
EXAMPLE
In contrast to gay /
bisexual men or IDUs,
heterosexual show an
initially low baserate.
Followed by a large spike
after the
announcement
And a much higher new
baseline.
The hypothesis that heterosexuals would be more affected by the
“Magic” announcement was supported by the interaction of
Time x Sexual Orientation.
Foundations of
Research
Testing blocking variables: Ethnic differences.
71
EXAMPLE
African-Americans
and Hispanics show
low baseline and a
high spike after the
announcement
Both groups go back
toward their
baseline shortly
post-announcement.
Foundations of
Research
Testing blocking variables: Ethnic differences.
72
EXAMPLE
 HIV testing among
Whites was similar
to African-Americans
& Hispanics at
baseline,
 They showed stable,
much higher testing
rate after Magic’s HIV
announcement.
EXAMPLE
Foundations of
Research
Summary: Blocking variables in time series data
A series of measures before & after an event allows us to clearly
identify patterns of behavior, and to test group differences (via
blocking variables).
The hypothesis that ethnic groups would differ was supported by
interaction of Time x the blocking variable of ethnicity (but in a
direction that was not predicted: Whites showed more change).
73
Foundations of
Research


Time series designs: Summary
Time series is most common with archival data: existing,
standard records collected for other purposes.
Used where:
The hypothesis concerns changes in long-term trends
 Typically an experiment cannot be run

 Simple practicality or cost, e.g., health care issues
 Ethics; crime rates, rates of domestic violence, etc.
 The target events are not controllable.

Virtue:


Study natural / “real world” processes or interventions
Blocking variables – comparing time trends across
groups – allows us to test hypotheses.
74
Foundations of
Research
Click
When studying naturally occurring
events…
A. The independent variable is often assessed
after the event.
B. The researcher has control over the study
measures.
C. It is possible to derive a random sample.
D. Internal validity is typically high.
75
Foundations of
Research
Click
Which of these is not a cause of nonequivalent groups?
A. Self-selection
B. Non-random assignment
C. Use of existing groups
D. Convenience sampling
E. Participants not blind
76
Foundations of
Research

Exam issues
Key terms:



Slide 2: “true” v. quasi –experiments
Threats to internal validity
Basic forms of quasi-experiments



Single shot
Single group pre- post- test
“Non-equivalent” two group designs:






Self-selection (in or out [mortality])
Existing groups
Non-blind
Non- random assignment
Interrupted time-series / group contrasts
Virtues (external validity) and problems (internal
validity)
77