Experimental design
Experimental design
• What is an experiment?
• Controlling the experiment
• Design categories (Experimental, Quasi-
experimental and Ex Post Facto)
• Comparison
• Meta analysis
• Examples
Experimental design
Experiment
Ordered investigation that attempts to prove
or disapprove a hypothesis
Questions
Hypothesis
Experiment
Analysis
Conclusion
Scientific Method Components
Experimental design
Experimental design
• So .. The hypothesized relationship is
Independent variable -> dependent variable
Experimental design
• An independent variable is one that the
researcher studies as a possible cause of
something else.
• Dependent variable is a variable that is
potentially influenced by the independent
variable.
• Manipulation: Researcher manipulates the
independent variable ( treatment – intervention)
Experimental design
• So .. The hypothesized relationship is
Independent variable -> dependent variable
Experimental design
• So .. The hypothesized relationship is
Independent variable -> dependent variable
Certain Drug
impact on cancer
Experimental design
• A researchers can convincingly identify causeand-effect relationships by experimental
design
• Many possible factors that might cause or
influence a particular condition or
phenomena.
• The researcher attempts to control for all
influential factors except the study factor.
Controlling for confounding variables
•
•
•
•
1- keep something constant
2- Include a control group
3- Randomly assign people to group
4- Assess equivalence before the treatment with
one or more pretests .
• 5- Expose participants to all experimental
conditions
• 6- statistically control for confounding variables.
Internal Validity
• Regarding cause-effect or causal relationships, internal
Validity is the approximate truth about inferences.
• It is relevant in studies that try to establish a causal
relationship.
• The key question in internal validity is whether observed
changes can be attributed to your program or intervention
(i.e., the cause) and not to other possible causes.
• Did the treatment cause the outcome to occur? Or were there
other confounding factors that caused the outcome?
External Validity
• Are the findings unique to just participants we
studied or could they apply to other groups?
• Refers to the extent to which the results of an
experiment can be generalized across populations,
time and settings.
Single Group Threats
• Imagine that we are studying the effects of an education
program in mathematics for first grade students on a measure
of math performance such as a standardized math
achievement test.
• In the post-only design, we would give the first graders the
program and then give a math achievement posttest.
• Consider what would happen if you observe a certain level of
posttest math achievement or a change or gain from pretest
to posttest.
• You want to conclude that the outcome is due to your math
program. How could you be wrong?
Single Group Threats
• History Threat: It's not your math program that caused the
outcome, it's something else, some historical event that occurred.
• Maturation Threat: The children would have had the exact same
outcome even if they had never had your special math training
program. All you are doing is measuring normal maturation or
growth in understanding that occurs as part of growing up.
• Testing Threat: This threat only occurs in the pre-post design. What
if taking the pretest made some of the children more aware of that
kind of math problem -- it "primed" them for the program so that
when you began the math training they were ready for it in a way
that they wouldn't have been without the pretest.
Single Group Threads
• Instrumentation Threat: Like the testing threat, this one only
operates in the pretest-posttest situation. What if the change from
pretest to posttest is due not to your math program but rather to a
change in the test that was used?
• Mortality Threat: It means that people are "dying" with respect to
your study.
• Regression Threat: Also known as a "regression artifact" or
"regression to the mean" is a statistical phenomenon that occurs
whenever you have a nonrandom sample from a population and
two measures that are imperfectly correlated.
Design Categories
• Pre-experimental designs
• True Experimental designs
• Quasi-experimental designs
• Ex post facto designs
• Factorial designs
Design Categories
• Some are true experimental designs as such
they allow us to identify cause-and-effect
• Some give alternative explanations of an
observed change.
Design Categories
• All of the designs have one thing in common :
clearly identify independent and dependent
variable.
Design Categories
We illustrate the designs using tables that have
this general format "Table"
Group
Group 1
Group 2
Time ->
Pre- Experimental Design
• The cells have one of four notations:
• Tx: Indicates that a treatment is presented
• Obs: Indicates that observation is made
•
: Indicates that nothing occurs during a
particular time period.
• Exp: Indicates a previous experience
Pre- Experimental Design: One-shot
Experimental Case Study
The most primitive type of experiment that might
be termed "research“
Group 1
Tx
Obs
• The design has low internal validity
• 1- the characteristics or the behavior observed
after the treatment existed before the treatment
as well.
• 2- influenced by other factors
• 3- a single measurement or observation doesn't
guarantee that situation has change or not.
Pre- Experimental Design: One-shot
Experimental Case Study
• Example ..
• The design will be something like this :
Exposure to cold + Damp ground (TX)-> Child has a cold (Obs)
• One-shot experimental case study is simple to
carry out, its results are meaningless.
Pre- Experimental Design: One- Group
Pretest-posttest Design
• We know that a change has taken a place. But
we have not ruled out other possible
explanation for the change.
Group 1
Obs
Tx
Obs
Pre- Experimental Design: Static Group
Comparison
• Involves both an experimental group and a control
group.
Group 1
Group 2
Tx
Obs
Obs
• No attempt is made to check wither they are
similar or not before the treatment so no way to
know if the treatment actually causes any
differences between the two groups.
Types of Design
•
If random assignment is used, we call the design a randomized experiment or true
experiment.
•
•
If random assignment is not used, then we have to ask a second question: Does
the design use either multiple groups or multiple waves of measurement?
If the answer is yes, we would label it a quasi-experimental design. If no, we
would call it a non-experimental design.
True Experimental Design
• True experimental design is regarded as the most accurate
form of experimental research, in that it tries to prove or
disprove a hypothesis mathematically, with statistical analysis.
• For an experiment to be classed as a true experimental
design, it must fit all of the following criteria:
– The sample groups must be assigned randomly.
– There must be a viable control group.
– Only one variable can be manipulated and tested. The tested subjects
must be randomly assigned to either control or experimental groups.
True Experimental Design: Pretest-Posttest
Control Group Design
• The pretest-posttest equivalent groups design
provides for both a control group and a measure of
change but also adds a pretest.
• It is important that the two groups be treated in a
similar manner.
Random
Assignment
Group 1
Obs
Group 2
Obs
Tx
Obs
Obs
True Experimental Design: The Solomon
Four-Group Design
• The Solomon Four-Group Design is designed to deal with a potential
testing thread.
• This design has four groups.
• Two of the groups receive the treatment and two do not.
Random
Assignment
Group 1
Obs
Tx
Obs
Group 2
Obs
-
Obs
Group 3
-
Tx
Obs
Group 4
-
-
Obs
Quasi-Experimental Designs
• Sometimes, randomness is either impossible or impractical. In
those situations use quasi-experimental design.
• However, has features that can eliminate many threats to
internal validity.
• Used frequently in evaluation because:
– Often randomization is impossible or difficult
– Ethical/legal prohibitions against randomization
– No viable control group available
– Inadequate resources to conduct randomization
Quasi-Experimental Designs: Nonrandomized
Control Group Pretest-Posttest Design
• To show that two groups are equivalent with respect to the
dependent variable prior to the treatment, thus eliminating
initial group differences as an explanation for post-treatment
differences.
• Differs from experimental designs because test and control
groups are not totally equivalent; equivalence on the pretest
ensures equivalence only for variables that have specifically
been measured.
Group 1
Obs
Tx
Obs
Group 2
Obs
-
Obs
Quasi-Experimental Designs: Nonrandomized
Control Group Pretest-Posttest Design
• Threats to validity:
– Partly controls for history threat(external event would
affect both groups, provided groups, provided groups are
similar),
maturation,
instrument
threats
and
instrumentation threats.
– However, even if groups are statistically very similar, if the
intervention is given to volunteers they may behave
differently than control group(due to self-selection).
– Regression threat.
Quasi-Experimental Designs: Simple time-series
experiment
• To show that, for a single group change occurs during a
lengthy period only after the treatment has been
administered.
• Provides a stronger alternative to “One group pretest-posttest
design; external validity can be increased by repeating the
experiment in different places under different condition.
Group 1
Obs
Obs
Tx
Obs
Obs
Quasi-Experimental Designs: Simple time-series
experiment
• Threats to validity:
– Extension of pretest-posttest design but reduces
maturation, testing, regression threats
– No control group so no selection threat.
– History threat is controlled partially.
– if measurement changes around time of program
instrumentation threat may be present
Quasi-Experimental Designs: Control group, time-series
design
• Bolstering the internal validity of the preceding design with
the addition of a control group.
• Involves conducting parallel series of observations for
experiment and control groups.
Group 1
Group 2
Obs
Obs
Obs
Obs
Tx
-
Obs
Obs
Obs
Obs
Quasi-Experimental Designs: Reversal, time-series
design
• Showing , in a single group or individual, that a treatment
consistently leads to a particular effect.
• Is an on-again, off-again design in which the experimental
treatment is sometimes present, sometimes absent.
Group 1
Tx
Obs
-
Obs
Tx
Obs
Quasi-Experimental Designs: Alternating treatments
design
• Showing , in a single group or individual, that different
treatments have different effects.
• Involves sequentially administrating different treatments at
different times and comparing their effects against the
possible consequent of non-treatment.
Group1
Txa
Obs
-
Obs Txb Obs
Quasi-Experimental Designs: Multiple-baseline design
• Showing, the effect of a treatment by initiating at different
times for different groups or individuals, or perhaps in
different setting for a single individual.
• Involves tracking two or more groups or individuals over time,
or tracking a single individual in two or more settings, for a
lengthy period of time, as well as initiating the treatment at
different times for different groups, individuals, or settings.
Group1
-
Obs
Tx
Obs
Tx
Obs
Group2
-
Obs
-
Obs
Tx
Obs
Quasi-Experimental Designs: Alternating treatments
design
• Showing , the effect of a treatment by initiating it at different
times for
• Involves sequentially administrating different treatments at
different times and comparing their effects against the
possible consequent of non-treatment.
Group1
Txa
Obs
-
Obs Txb Obs
Ex Post Facto Designs
• Sometimes we can’t manipulate some variables
– Impossible, e.g.: Characteristics
– Unethical, e.g.: Virus
• Ex post facto = after the fact
– Already happened in the past
Ex Post Facto: Simple Design
• Possible effect of an experience/condition that
occurred in the past
• May show difference but not conclusive
Group
Time ->
Group 1
Exp
Obs
Group 2
-
Obs
Factorial Designs
• 2+ independent variables
• Simultaneously or sequential
Time ->
Factorial Design:
Two-factor experimental design
Random
Assignment
Group
Time ->
Group 1
Tx1
Tx2
Obs
Group 2
Tx1
-
Obs
Group 3
-
Tx2
Obs
Group 4
-
-
Obs
Factorial Design:
Two-factor experimental design
Random
Assignment
Group
Time ->
Group 1
Tx1
Tx2
Obs
Group 2
Tx1
-
Obs
Group 3
-
Tx2
Obs
Group 4
-
-
Obs
Factorial Design:
Two-factor experimental design
Random
Assignment
Group
Time ->
Group 1
Tx1
Tx2
Obs
Group 2
Tx1
-
Obs
Group 3
-
Tx2
Obs
Group 4
-
-
Obs
Factorial Design:
Two-factor experimental design
Random
Assignment
Group
Time ->
Group 1
Tx1
Tx2
Obs
Group 2
Tx1
-
Obs
Group 3
-
Tx2
Obs
Group 4
-
-
Obs
Factorial Design:
Two-factor experimental design
Random
Assignment
Group
Time ->
Group 1
Tx1
Tx2
Obs
Group 2
Tx1
-
Obs
Group 3
-
Tx2
Obs
Group 4
-
-
Obs
Factorial Design:
Two-factor experimental design
Random
Assignment
Group
Time ->
Group 1
Tx1
Tx2
Obs
Group 2
Tx1
-
Obs
Group 3
-
Tx2
Obs
Group 4
-
-
Obs
Factorial Design:
Two-factor experimental design
Random
Assignment
Group
Time ->
Group 1
Tx1
Tx2
Obs
Group 2
Tx1
-
Obs
Group 3
-
Tx2
Obs
Group 4
-
-
Obs
Factorial Design:
Two-factor experimental design
Random
Assignment
Group
Time ->
Group 1
Tx1
Tx2
Obs
Group 2
Tx1
-
Obs
Group 3
-
Tx2
Obs
Group 4
-
-
Obs
Factorial Design:
Two-factor experimental design
• Example: Clustering using K-means
– Tx1: A different method to choose initial centroids
– Tx2: A different formula to calculate distance
Factorial Design: Combined experimental
and ex post facto design
Group1
Group2
Time ->
Expa
Expb
Random
Random
Assignment Assignment
Group
Group 1a
Txa
Obs
Group 1b
Txb
Obs
Group 2a
Txa
Obs
Group 2b
Txb
Obs
Factorial Design: Combined experimental
and ex post facto design
• Example: Clustering using K-means
– Expa, Expa: 2 different kinds of datasets
– Txa, Txb: 2 different formulas to calculate distance
Comparison
Meta-analyses
• Replication
• Quantitative analysis of the analyses
– Primarily statistical
Examples & Discussion