Non-Experimental
Methods I
Class 22
Why Alternatives to Controlled Experiment?
Advantage of controlled experiment?
Advantages to not using controlled experiment?
Controlled Experiment:
Great, but Not Always the Best
Ethical Concerns
Practical Concerns
Methodological Concerns
Generalizability
Internal analyses
Not all interesting questions are causal
Types of non-experimental studies
a. Correlational
b. Pseudo-experimental
c. Quasi-experimental
Correlational Studies
FIASCO STUDY: Shared meals will reduce major fights
Correlation of meals to major fights = -. 37
Interpretations:
1. ???????????
2. ???????????
3. ???????????
Correlational Studies
FIASCO STUDY: Shared meals will reduce major fights
Correlation of meals to major fights = -. 37
Interpretations:
Dinners -----------------> Fewer fights
Fewer fights ----------> More dinners
more dinners
Less overtime
at work
fewer fights
??? Panel Design
Do shared meals reduce the frequency of major
family conflicts?
Time
1
Dinners
Time
2
(3 years)
r = +.24
r = - .31
r = - .37
A
Dinners
B
Fights
r = +.40
Fights
Cross-Lagged Panel Design
Do shared meals reduce the frequency of major
family conflicts?
Time
1
Dinners
Time
2
(3 years)
r = +.24
r = - .31
r = - .37
A
Dinners
B
Fights
r = +.40
Fights
Cross-Lagged Panel Design
Do shared meals reduce the frequency of major
family conflicts?
Time
1
Dinners
(3 years)
Time
2
r = +.24
r = - .31
r = - .37
r = -.15
Dinners
r = -.39
Fights
r = +.40
(Pssst: Ask Kent about Walter Michel study)
Fights
????? Designs
1. One-shot (post-test only) case study
2. One-group pretest-posttest design
3. Static-group comparison design
Pseudo-experimental Designs
1. One-shot (post-test only) case study
2. One-group pretest-posttest design
3. Static-group comparison design
One Shot Case Study
Design: Observe population of interest
Examples:
Client case studies (client case studies, famous people)
Group case studies (focus groups)
Population case studies (anthropological studies)
FIASCO: Study families at dinner time,
observe dinner together --> less conflicts.
Problems?
One Shot Case Study
Design: Observe population of interest
Examples:
Client case studies (client case studies, famous people)
Group case studies (focus groups)
Population case studies (anthropological studies)
FIASCO: Study families at dinner time,
observe dinner together --> less conflicts.
Problems:
No comparison:
with other families displaying same behavior
with same family under different conditions.
Excellent Case Studies
De Tocqueville in America
Freud’s studies of Anna O, and the Wolfman
Cantril’s study of NJ after Mars Invasion
Wertheimer on train  Gestalt psychology
One Group Pretest-Posttest Design
Observe before and after treatment
Pretest and posttest provide basis for comparison
Example:
* Monitor families before and after dinner
* Record number/intensity of conflicts
pre vs. post dinner.
* Finding: Less tension after dinner
Problems with this design???
Threats to Internal Validity
1. History effects: ?????
2. ????: People change over time
3. Testing: ????
4. ?????: Experimenter, apparatus change.
5. Mortality: ????
6. Selection: ????
Threats to Internal Validity
1. History effects: Things co-occur with TX
2. Maturation: People change over time
3. Testing: People react to being observed
4. Instrumentation: Experimenter, apparatus change.
5. Mortality: People drop out before study ends.
6. Selection: People volunteering for study may differ
from those who did not volunteer.
Static Group Comparison Design
General idea: Take advantage of naturally occurring comparison group.
Examples:
School intervention: Look at students not in new program
FIASCO: Look at families nearby, that don't
have dinner together.
Advantage: Provides some basis for comparison.
Problem?
Selection.
Maybe non-dinner family found that not sharing dinner
helps prevent conflicts.
???-Experiments
Features
* Control administration of treatment (TX)
* Control the collection of DV measures
* BUT, no random assignment to TX or control
WITHIN GROUP QUASI-DESIGNS
Simple time series designs
Equivalent time-samples design
BETWEEN GROUP QUASI-DESIGN
Non-equivalent Control-group design
Multiple-group time-series designs
Quasi-Experiments
Features
* Control administration of treatment (TX)
* Control the collection of DV measures
* BUT, no random assignment to TX or control
WITHIN GROUP QUASI-DESIGNS
Simple time series designs
Equivalent time-samples design
BETWEEN GROUP QUASI-DESIGN
Non-equivalent Control-group design
Multiple-group time-series designs
The Arith-Matisse Project
Problem: Many students have math anxiety. Also,
they don't know how to engage in visual skills that
might improve their math achievement.
Solution: Provide students 1 hour of drawing / fine
arts instruction to help develop visual skills, and
also to relax them before math class. Art class
must immediately precede math class.
Prediction: Art class preceding math class will lead
to better math performance.
Time Series Experiment:
Simple Time Series Design
Effect of art hour on math class performance
Art Hour Offered
Day
Score on Daily
Math Quiz
No
Monday
60%
No
Tuesday
57%
No
Wednedsay
63%
Yes
Thursday
84%
No
Friday
74%
No
Monday
64%
No
Tuesday
59%
Why It’s Necessary to Monitor Outcome
Beyond Treatment Administration
Art Hour Offered
Day
Score on Daily
Math Quiz
No
Monday
60%
No
Tuesday
57%
No
Wednedsay
63%
Yes
Thursday
84%
Question: Why not stop monitoring after treatment (e.g., Thurs)?
No
Friday
81%
No
Monday
87%
No
Tuesday
90%
Reason: If stopped on Thurs., would miss continuation of change.
Problems with Simple Time Series

Hawthorne Effect:
Being observed can, in itself, be motivating
or otherwise affect change.

History Effect
Events co-occurring with treatment could
be influencing outcomes.
Problems with Simple Time Series

Hawthorne Effect: ????

History Effect: ????
Time Series Experiment:
Equivalent Time Samples Design
Effect of art hour on math class performance
Art Hour Offered
Day
Score on Daily
Math Quiz
No
Monday
60%
No
Tuesday
57%
No
Wednesday
63%
Yes
Thursday
84%
No
Friday
74%
No
Monday
64%
No
Tuesday
59%
Yes
Wednesday
85%
No
Thursday
62%
No
Friday
65%
Problems With
Equivalent Time Samples Design?
Art Hour Offered
Day
Score on Daily
Math Quiz
No
Monday
60%
No
Tuesday
57%
No
Wednesday
63%
Yes
Thursday
84%
No
Friday
74%
No
Monday
64%
No
Tuesday
59%
Yes
Wednesday
85%
No
Thursday
62%
No
Friday
65%
Problems With
Equivalent Time Samples Design?
Art Hour Offered
Day
Score on Daily
Math Quiz
No
Monday
60%
No
Tuesday
57%
No
Wednesday
63%
Yes
Thursday
84%
No
Friday
74%
No
Monday
64%
No
Tuesday
59%
Yes
Wednesday
85%
No
Thursday
62%
No
Friday
65%
No comparison condition. Example: Students do well on math
in order to retain art class, but not b/c art helps with math
Psychologists Sans Data
William James
Father of Amer. psychology
The Self
Emotions
Group Dynamics
Perception
Sigmund Freud
Unconscious Processes
Psychosomatics
Social, Personality, Developmental
Kurt Lewin
Group Processes
Social Systems
The Self
Solomon Asch
Perception
Conformity, group dynamics
Gordon Allport
Founds Personality Theory
Prejudice research
Humanists
Victor Frankl
Rolo May
Abraham Maslow
Carl Rogers
“Invasion From ???”: The ‘Big Bad Boo’ of 1938
Orson Well broadcast of invasion from Mars—as if a
breaking news story, Oct. 30, 1938
Approximately 30 million people affected, to varying degrees
Historical context
a. ??? relatively new—and trusted
b. World on brink of ???
“Invasion From Mars”: The ‘Big Bad Boo’ of 1938
Orson Wells broadcast of invasion from Mars—as if a
breaking news story, Oct. 30, 1938
Approximately 30 million people affected, to varying degrees
Historical context
a. Radio relatively new—and trusted
b. World on brink of world war
Cantril Study (1940)
Rationale: Rare event; oppty to study mass behavior
Event structures research:
* How people respond during panic/stress
* Power of real-life event, outside the lab
Method:
* Multi-method approach—provides converging evidence:
-- Interviews (n = 135)
-- Unobtrusive measures (e.g., phone company)
-- Commercial survey research (Gallop, AIPO)
-- Newspapers
Observations from Invasion from Mars Study
Recovery from panic
* Prolonged anxiety – sometimes
“I was sick in bed for 3 days after the broadcast”
* “Nevermind” – sometimes
“We listened to the rest of the play and went out dancing”
Role of religion and faith
* “I held a crucifix in my hand and prayed while looking out of
my window for falling meteors”
Observations from Invasion from Mars Study
(continued)
Role of purposeful action in coping
* “...having to concentrate on the driving held me together
somewhat”
Fear of unknown
* I didn’t (know) what I was fleeing from, and that made me all
the more afraid.”
Background fears based on reality influence immediate, ambiguous threat.
* “Don’t you know New Jersey is destroyed by the Germans...”
Final note: Open-ended interviews provide invaluable historical record.