EXPERIMENTS AND EXPERIMENTAL
DESIGN
ETHICAL ISSUES IN RESEARCH
Logic of causality
If X, then Y
• and
If not X, then not Y
Or, colloquially:
If the stimulus is given, then the outcome occurs
• and
If the stimulus is not given, then the outcome does not
occur
Basic requirements of a true experiment:
- At least 2 groups: one experimental & one control group
- Random assignment of subjects to groups. Every case has:
(1) the same chance of being selected (i.e. equal probability)
(2) the chance of selection does not change (i.e. constant
probability)
- Manipulation of the IV.
- Measurement of the DV.
- Except for the experimental manipulation, the 2 groups =
treated exactly alike
Results
- controlling for pre-existing subject differences; differences
are to be neutralized/distributed approximately evenly
among the groups  random assignment of subjects to
treatment & control groups
- making sure that the events occurring within each
experimental group are exactly the same, except for the
manipulated independent variable  controlling for
extraneous factors & experiences during the experiment
Randomization for groups
Tests
• Use tests of statistical significance to determine the
likelihood that the results occurred by chance rather than
being caused by the treatment variable (i.e. significant
differences btw. the groups).
• A result is statistically significant at the 0.05 level:
it means that only 5% of the time (5 times in 100 if we were
to repeat the experiment) would differences btw. the
experimental & control groups occur by chance when the
experimental variable actually has no effect.
Tests
• Differences between the mean values for T and C
• The differences between slopes for T and C
Mean values for T and C
Slope
Staging of an experiment
1. Introduction: Instructions and Cover Story
2. Manipulation of the IV
3. Measurement of the DV (verbal reports; behavioral
observation)
4. Debriefing (explain purpose of experiment; explore subjects’
reactions; alleviate subjects’ anxiety about their
performance/ participation)
Validity
• Internal Validity: the approximate truth about inferences
regarding the tested causal relationships in the experimental
setting
• External Validity: the degree to which the conclusions in the
study would hold for other persons in other places & at
other times (i.e. how generalizable are the results of the
study)
Issues to consider
• Pre-testing of experiments
• Laboratory experiments vs. experiments in natural settings
(field experiments)
• Trade-off btw. increase in external validity & control over
extraneous variables & manipulation of the IV
• Subjects’ reaction to the social meaning of the situation:
often a problem for laboratory experiments
Types of experiments
•
•
•
•
•
Pre-test – post-test control group design
Post-test-only control group design
The Solomon four group design
Factorial design
Quasi-experimental design
– Experimental Design in Survey
Pre-test – post-test control group design
Tt1
X
Ct1
--
Tt2
R
Ct2
R = random assignment of subjects to groups
T = Treatment group, C = Control group
t1 = first measurement, before experimental treatment
(pretest), t2 = 2nd measurement, after experimental
treatment (post-test)
Post-test-only control group design
X
Tt1
--
Ct1
R
Many variations on this experimental design:
See www.socialresearchmethods.net - various experimental designs.
The Solomon four group design
Tat1
X
Tat2
Cat1
--
Cat2
X
Tbt1
--
Cbt1
R
Outcome I
Outcome II
Source: www.socialresearchmethods.net
Factorial design
The Main Effects and Interaction Effects
• A main effect is an outcome that is a consistent difference
between levels of a factor.
• Because of the way we combine levels in factorial design, it
also enables us to examine the interaction effects that exist
between factors. An interaction effect exists when
differences on one factor depend on the level of another
factor.
Explanation of effects, I
Explanation of effects, II
Explanation of effects, III
Explanation of effects, IV
Explanation of effects, V
Quasi-experimental design (QED)
QED takes an experimental approach without having full
experimental control; lacks some feature of true
experiments, generally randomization
•
- Separate sample pretest-posttest design (both groups
receive the treatment, but one is tested before the
treatment, and the other afterward)
•
- None-equivalent control group design; control group
but without random assignment;
•
- Interrupted time series-design: series of observations
before and after the treatment;
•
- Multiple time series design: extends the one above by
adding a series of observations on a non-equivalent control
group.
Experimental Design in Surveys
• Differently worded questions: As part of a larger survey:
direct a question with one wording to a randomly selected
sub-sample of respondents; and a differently worded
question to the remaining respondents.
• Vignettes: Short, detailed and concrete descriptions of
situations that contain references to the factors that are
thought to be important in decision making.
To test experimentally which factors affect the subject’s
decision making, a number of versions of the vignettes are
developed by systematically varying the details of the
situation. The different versions are then randomly assigned
to respondents.
• Constructing different sets of questionnaires
ETHICAL ISSUES IN RESEARCH
Basic question:
For what to to whoe the researchers in the social sciences are
resposible?
Human subjests
Peers
Society
Ethics of treatment of human subjects
• - Ethical considerations preclude withholding treatment
from a control group: education, correction, clinical
psychology, medicine frequently the case.
• Institutional review boards assess the ethics of the
proposed research (including on risk vs. benefits).
Overall guiding principle: potential benefits of research
should be weighed against potential costs (risks).
General rules:
• - Researcher should not expose participants to
substantial risk of physical or psychological harm, unless
the benefits of participation exceed the risks, and the
subject knowingly chooses to participate;
• - Subjects should be informed that their participation is
voluntary throughout the entire research; should be
told about any aspects of the research that may
influence their willingness to participate;
• - If deception is necessary- appropriate debriefing as
soon as possible after their participation;
• - Confidentiality and anonymity: researchers should use
all possible means to protect the confidentiality of
information participants provide (identity).
Ethics of data collection and analysis
• - Honesty and accuracy in collection & analysis of data, and
reporting of results
• - Scientific misconduct, if identified, kills researcher’s career
Ethics of responsibility to society
• Awareness of potential use and misuse of scientific results
by others