Scientific Method and
Experiemental Design
Deduction v. Induction
All men are mortal
Socrates is a man.
Therefore Socrates is mortal.
--versus-Is Socrates a Man?
Induction
• Sun came up yesterday
• Sun came up day before yesterday
• Sun came up the day before that
• …Sun will come up tomorrow
David Hume
• Deduction concerns “relations among
ideas”
• Induction: big problems
– In the past, the future has always been like the
past…
– Is this circular logic?
Assumptions of Science
• Materialism – nature consists of matter that
is observable
• The universe is mechanistic – there are
discoverable cause-and-effect relations
• Determinism– Everything has a material
cause.
Is it wrong to be a dualist?
• Could the mind interface the body through
the pineal gland? YES.
• Is that question appropriate to science? NO.
The problem with “Creation
Science”:
• Popper: Scientific theories are
falsifiable.
Characteristics of science:
• Theories cannot be proved
• Theories can only be disproved
Why do science then?
• Intuition can lead us astray
• Example: Perception is a leap of faith; our
brain uses many shortcuts
– Availability heuristic
– Fundamental attribution error
– Confirmation bias
Example: Clinical Judgment
• Paul Meehl (1920-2003)
– “Clerks” better than psychiatrists
at predicting prisoner recidivism
• Medical treatment
recommendations
– Back surgery in Santa Barbara
vs. Bronx
• Hypothesis: a limited statement regarding
cause and effect in specific situations
– Dog barked because mailman approached
• Model: Mathematical (or algorithmic)
statement that makes predictions over a
limited range of situations.
– Bark Rate = H x 1/Dmailman
• Theory: Barking caused by the approach of
a unfamiliar organism.
Research methods
• Correlational
• Quasi-experimentation
• Experimentation
Correlational Research
“CASA research shows a statistical relationship between
use of tobacco, alcohol and marijuana and use of drugs
such as cocaine, heroin and acid. Among teens with no
other problem behaviors, those who used cigarettes,
alcohol and marijuana at least once in the past month are
almost 17 times likelier to use another drug like cocaine,
heroin and acid. Those who drank and smoked cigarettes
at least once in the past month are 30 times likelier to
smoke marijuana than those who did not.”
-The National Center on Addiction and Substance Abuse at
Columbia University
Correlation  Causation
Cocaine
use
Pot
use
Cocaine
use
Pot
use
Pot
use
?
Cocaine
use
9
mg Cocaine/wk
8
7
6
5
4
3
2
1
0
0
2
4
6
# Joints/week
8
10
12
Correlation Coefficients
high
r = -1
low
low
high
high
r = +1
low
low
high
Research methods
• Correlational
• Quasi-experimentation:
– Older adults lower “fluid intelligence” than
younger adults
• Experimentation
Quasi-experiments
• Used when it is impossible to manipulate IV
• Or when unethical to manipulate IV
• Problems:
– Do groups differ beyond IV?
Research methods
• Correlational
• Quasi-experimentation:
– Older adults lower “fluid intelligence” than
younger adults
• Experimentation
Experiment
• Experimenter manipulates independent
variable and observes effects on the
dependent variable
• Example:
– IV: Drug A v. No Drug A
– DV: Pain sensitivity
Experiment Design
• Between Subjects
– Each subject experiences one level of the IV
• Within Subjects
– Each subject experiences all levels of the IV
Between-Subjects Design
Within-Subjects Design
1/2
Drug A Test  PlaceboTest
All subjects
1/2
Placebo  TestDrug ATest
You want to see whether a GRE prep course is
effective. You randomly select 50 students from
Columbia and give them a GRE. Then you give the
students the prep course and assess their GRE scores
again at the end of the course. Their scores improved
significantly. Is there a problem with this conclusion?
Research Issues
• Internal validity: is the IV really responsible for
the effects on the DV? Are there confounds?
– Extraneous variables: Other variables, in addition to the
IV, affect the DV.
• External validity: Does our experiment generalize
to real situations?
• Construct Validity: Do our measures tap the
intended construct.
• Reliability: Can this finding be replicated?
• Expectancy: Did expectations of experimenter or
participant affect results?
Internal Validity Questions:
Did the Treatment Cause a Change in
Behavior?
Research Issues
• Internal validity: is the IV really responsible for
the effects on the DV? Are there confounds?
– Extraneous variables: Other variables, in addition to the
IV, affect the DV.
• External validity: Does our experiment generalize
to real situations?
• Construct Validity: Do our measures tap the
intended construct.
• Reliability: Can this finding be replicated?
• Expectancy: Did expectations of experimenter or
participant affect results?
External Validity Questions: Can the
Results be Generalized?
• Can the results be generalized to other
participants?
• Can the results be generalized to other
settings?
Research Issues
• Internal validity: is the IV really responsible for
the effects on the DV? Are there confounds?
– Extraneous variables: Other variables, in addition to the
IV, affect the DV.
• External validity: Does our experiment generalize
to real situations?
• Construct Validity: Do our measures tap the
intended construct.
• Reliability: Can this finding be replicated?
• Expectancy: Did expectations of experimenter or
participant affect results?
Construct Validity Questions:
Can We Make the Leap from the
Physical World to the Mental
World?
3 Follow-up Questions*
3 Key Construct Validity
Questions
1. What does the measure really measure?
2. What does the manipulation really
manipulate?
3. Is the participant’s behavior genuine or an
act?
Research Issues
• Internal validity: is the IV really responsible for
the effects on the DV? Are there confounds?
– Extraneous variables: Other variables, in addition to the
IV, affect the DV.
• External validity: Does our experiment generalize
to real situations?
• Construct Validity: Do our measures tap the
intended construct.
• Reliability: Can this finding be replicated?
• Expectancy: Did expectations of experimenter or
participant affect results?
Reliability: The statistics you obtain in your
experiment don’t always equal the population
statistics.
54322167459
87545321554
78 434591374
56421356375
34251953458
7715
327
415
Your Sample
The World
Research Issues
• Internal validity: is the IV really responsible for
the effects on the DV? Are there confounds?
– Extraneous variables: Other variables, in addition to the
IV, affect the DV.
• External validity: Does our experiment generalize
to real situations?
• Construct Validity: Do our measures tap the
intended construct.
• Reliability: Can this finding be replicated?
• Expectancy: Did expectations of experimenter or
participant affect results?
Clever Hans
How to address validity issues:
• Internal validity:
–
–
–
–
–
Matching
Random assignment
Counterbalancing
Control Groups
Single- or double blind design
• External Validity
– Random selection
– Judicious selection of dependent measures and
independent variables
Example Experiment #1
• Study at CPMC: Expectant mothers in US
randomly assigned to Groups: “Pray” or “NoPray”
• People somewhere in Asia prayed for members of
the “Pray” group. Double-blind study.
• Lower miscarriage rate in the “Pray” group.
Example Experiment #2
• Participants randomly assigned to two groups:
“Stretch+Lift” or “Lift Only”
• All participants embark on 6 mo training regimen.
Members of “S+L” group accompanied by trainer
to guide stretching. “LO” group no trainer, no
stretching.
• Bigger strength gains in the “S+L” group.
Example Experiment #3
• By J.P. Broca: Measured weight of 292
male and 140 female brains (at autopsy).
• Found that male brains averaged 1325g and
female brains 1144g.
• Conclusion: Confirmation of male
intellectual superiority.
Bias in intellectual assessment
Actual questions from the Army Alpha Test (and early intelligence test):
Crisco is a
a) patent medicine
b) disinfectant
c) toothpaste
d) food product
Christy Mathewson is famous as a(n)
a) writer
b) artist
c) baseball player
d) comedian
What’s missing?
Example Experiment #4
• Intro Psych “volunteers” randomly assigned
to 2 groups: nasty feedback, no feedback.
• Nasty group performs academic exercise,
then is given very unfavorable feedback
about work.
• All S’s debriefed.
• 1 year later, enduring effects are assessed.
• Conclusion: Psych experiments don’t have
permanent detrimental effects.
Experiment #5
• Intended to assess effects of Supportive Therapy
or Interpersonal Therapy on depression.
• Patients randomly assigned to SP or IPT. Some
therapists did SP, some did IPT (30 therapists
total). 12 weekly therapy sessions at times that
were convenient for patients.
• Bigger reduction in depressive symptoms in IPT
than SP group.