PSY 395 Design Lab 1
PSY 395 Lab 10: Deciding on a Design
How do you get from your research question to a research
design?
Just guess?
Hope it comes to you in a dream?
Take what you know from 395 and make educated
guesses? YES!
You need to think about your research question and the
independent variable to begin:
Can you randomly assign on all variables (all of your
Independent Variables)?
Yes
No
The answer to this question limits the type of design you
have. Why?
PSY 395 Design Lab 2
If you answer yes, you have true experimental design. If
you answer no, you have either a quasi-experimental or an
observational design.
To decide whether you have a quasi-experiment or
observational experiment, you need to ask…
Can I manipulate any variables?
What does knowing about manipulation tell us?
Yes
No
If yes, then you have a quasi-experimental design
If no, then you have an observational design.
Non-experimental/Observational research falls into 3
categories:
 Archival research (analyzing data that someone else has
already collected)
 Survey research (collecting data on questions of interest
to you)
 Correlational research – the overarching category of
observational design – using correlations to test the
strength of relationship between two variables
PSY 395 Design Lab 3
Let’s stick with true experiments for now…
The next question you want to ask is:
Do you have a pretest?
YES
NO
You always assume that you have a post-test measure, so
you aren't going to be asking that as one of your
questions, but you do need to know if you’ll have a pretest.
If you do have a pretest, then you have what is called a
repeated measures design. Because you’ll be comparing
people’s scores on the pretest with scores on the post-test
to check for differences due to the treatment, you have
multiple measures, which is why this design is called
repeated measures. If you have multiple groups you’re
comparing pretests and post-tests on, this type of design is
called between groups repeated measures, because you
want to see if there are differences between groups on
your measures.
PSY 395 Design Lab 4
No pretest?
If you do not have a pretest, you have something called a
randomized two-group design. While you have randomly
assigned people to your conditions, all you’re looking at
are the differences between groups on your post-test,
because you have no pretest measure to compare people
on.
Continuing on with repeated measures…
The next question you want to ask is:
Are you comparing differences across more than 2
groups?
YES
NO
Why does the number of groups make a difference?
Think about a design you talked about in class that used
more than 2 groups. What was unique about that design?
PSY 395 Design Lab 5
Not more than 2 groups?
If you do not have more than 2 groups, then you have a
design called a pretest-posttest two group design. This
design is just what it says: it uses a pretest and posttest for
two groups (still random assignment and repeated
measures) and compares differences on your DV across
the two groups.
If you do have more than 2 groups, and only some of
those groups take a pretest, then…
You have what is called a Solomon Four-Group Design.
In this design, two groups take a pretest and the posttest,
while the other two groups do not take the pretest, but
only take the posttest. What advantages does this kind of
design give to the experimenter (think validity)?
And that gets us to the end of true experimental designs –
let’s go back to look at some quasi-experimental designs.
PSY 395 Design Lab 6
Remember what defines a quasi-experiment: If you
cannot randomly assign on all of your variables, but you
can manipulate some, then you have a quasiexperiment.
Let’s ask the same questions we did to determine what
type of experimental design we had:
The next question you want to ask is:
Do you have a pretest?
YES
NO
Remember, when using a quasi-experimental design, you
do not have control over things such as individual
differences because you did not randomly assign people
to your conditions. Therefore, different groups in quasiexperimental designs are often referred to as
nonequivalent because differences across the two groups
might exist (due to no random assignment).
PSY 395 Design Lab 7
If you do have a pretest…
Then you have what is called a pretest-posttest
nonequivalent control group design. This too is a
repeated measures design, because you have both a
pretest and posttest and you have one control group and
one experimental group. Your goal is to examine the
differences in outcomes between your two groups due to
your treatment. The only thing different about this type
of design from a “true experiment repeated measures” is
that your groups are not equivalent as explained just a
minute ago.
If you do not have a pretest…
Then you have what is called a static-group comparison
design. This type of design simply compares differences
in your DV across two groups, without taking a pretest
measure in either group. Remember, one group will still
receive a treatment while the other serves as the control.
You have a between groups design, however it is NOT
repeated measures. What type of threat to validity does
this design open you up to?
PSY 395 Design Lab 8
If you have multiple pretests (and multiple
posttests)…
Then you have something called an interrupted time
series design. This type of design uses multiple pretests
prior to your treatment and multiple posttests following
the treatment. What type of threat does this help the
researcher understand better?
To narrow down the type of design even further, the next
question you ask is “Do you have more than 2 groups?”
The design discussed in your book only talks about what
happens when you have 1 and only 1 group.
This type of design is called a one-group pretest-posttest
design. What kind of comparison do you think this type
of design makes? (remember you’ve only got one group,
so this would not be a between groups design).
This type of design is also called simple panel research –
this is a within-groups (repeated-measures) design, since
it looks at changes in your outcome within just one group,
using the same participants prior to the treatment as after
the treatment.
PSY 395 Design Lab 9
Non-Experimental/Observational designs are used
when you have NO random assignment and NO
manipulation!
3 General Categories of non-experimental design:
1. Archival- using data that has already been collected
by another person
2. Survey- collecting data by administering
questionnaires - Note: while you do not use
assignment, you can/will use random or non-random
sampling!
3. Correlational- using correlation to test the strength
of the relationship between two variables.
An important thing to remember with survey and archival
data is that you can have within-group designs as well
between (independent) group designs.
PSY 395 Design Lab 10
To review: The types of designs we’ve covered are:
Non-Experimental/Observational
- Archival research
- Survey research
- Correlational research
True Experiment
- Randomized two group design (posttest only)
- Pretest-posttest two group design (pre-post repeated
measures: between groups)
- Solomon four group design
Quasi-Experiment
- Pretest-posttest nonequivalent control group design
- Static group comparison design
- Interrupted time series design
- One group pretest-posttest design (within group)
PSY 395 Design Lab 11
So, now you can figure out what type of design you have
given some information about the experiment. This is
part of your homework for today. The other part is
deciding what type of statistical test you would use to test
your results.
Type of Design
Statistical Test
Non-experimental/Observational Correlation
Regression
Analysis of Covariance
ANOVA
Independent Measures t-test
Repeated Measures t-test
Randomized Two Group Design Independent Measures t-test
Pretest-Posttest Two Group
Repeated Measures t-test
Design
Solomon Four Group Design
ANOVA
Pretest-Posttest Nonequivalent
Repeated Measures t-test
Control Group Design
Static Group Comparison
Independent Measures t-test
Interrupted Time Series Design Repeated Measures ANOVA
One Group Pretest-Posttest
Repeated Measures t-test
Design
(within group)
Identifying Statistical Tests
Some of this can be figured out, but some of this also
requires memorization.
Let’s review: what do each of these tests tell us?
Correlation
Regression
Independent Measures t-Test
PSY 395 Design Lab 12
Repeated Measures t-test
ANOVA
Something new: Analysis of covariance (ANCOVA)
What’s the difference between ANOVA and ANCOVA?
ANCOVA acknowledges that in observational research,
there may be additional variables that covary or change
with the variables you’re interested in. The relationship
between these additional, or confounding variables, can
influence the relationship between the IV and DV that
you want to look at.
ANCOVA is a technique that removes the influence of
the covarying variable from the relationship between your
IV and DV so that you can look at the “pure” relationship
between your two variables. In other words, ANCOVA
allows you to look at the strength of the relationship
between your IV and DV by removing the influence of a
confounding variable.
PSY 395 Design Lab 13
Project Questions?
Methods Section: when proposing your research design,
you must go beyond saying whether you have a nonexperimental, quasi-experimental or true experimental
design. You need to describe a specific design (e.g., static
group comparison, randomized two group design) and
give enough detail so that it’s clear we understand the
design you’re using and why it is appropriate for your
research question.
Any other project related questions…
The next section of your project requires you to:
1) Assume that all went well and your results support your
research hypothesis. What are the internal and external
threats to validity that might lead you to draw the
wrong conclusion about your research results? (you
need to give specific examples of threats to validity
here, at least one of each)
2) Assume that your results failed to support your research
hypothesis. What might have gone wrong in your
research to yield this result? (this is where you would
discuss the limitations of your research and critique
what you’ve proposed – start with your methods section
if you’re having trouble)
PSY 395 Design Lab 14
Start Here: Can you randomly assign
on all variables?
NO
Can you manipulate
any IV?
YES
Non-Experimental design
- Archival research
- Survey research
- Correlational research
YES
True Experimental Design
NO
Quasi-Experimental Design
Do you have
a pretest?
YES
Repeated
Measures
Design
NO
YES
NO
Randomized
Two-Group
Design
Pretest-Posttest
Static-Group
Nonequivalent
Comparison
Control Group Design
NO
NO, I only have 1 group
Do you have
more than 2
groups?
YES
One-Group Pretest-Posttest Design
keep going
Do only some
of your groups
take a pre-test?
Pretest-Posttest
Two-Group Design
YES Solomon Four-Group Design
YES, I have multiple pretests
Interrupted Time Series
Design
PSY 395 Design Lab 15
Homework for Lab #10
Read the following 3 scenarios. For each scenario, identify: (a) the design of the
experiment and explain why, (b) the statistical test you would use to analyze the data, and
(c) one specific validity threat (external or internal) and explain why it’s a threat.
1. Can blueberries slow down aging? A study indicates that antioxidants found in blueberries
may slow down the process of aging. In this study, 19-month old rats (equivalent to 60-year
old humans) were fed either their standard diet or a diet supplemented by either blueberry,
strawberry, or spinach powder. After eight weeks, the rats were given memory and motor
tests. Although all supplemented rats showed improvement, those supplemented with blue
berry powder showed the most notable improvement.
2. [abstract from a published study] We examined the effectiveness of a cognitive
intervention to help adolescents cope with stress and other forms of negative emotional
arousal. Through this intervention procedure, youths learned how to identify and monitor
stress-promoting cognitions, restructure these cognitions into more adaptive thoughts, use
self-instructions to control stress-engendering self-statements, and practice and apply these
acquired skills. A total of 21 adolescent males, 16–17 years old, participated in the study.
The project was conducted at a college preparatory, parochial high school for boys in a
midwestern city. Admission to this school is based on competitive examinations taken by
students from over 100 elementary schools in the surrounding area. In recent years, 98% of
the graduates have enrolled in college. The nature and purpose of the project was described
to juniors and seniors by the head guidance counselor, and any students who were interested
in learning more about the stress-management project were asked to attend an orientation
meeting. Originally, 30 youths attended this orientation meeting, during which the research
was described in full by the researchers and parental and youth consent forms were
distributed. Twenty-four youths returned signed consent forms and volunteered for the
project. These youths were randomly assigned to either the experimental group that received
training or a waiting list control group. Before the training was completed, three youths in the
experimental group dropped out of the study because of schedule conflicts. As a result, the
final sample was composed of 9 experimental youths and 12 control youths. All youths were
White with the exception of one Black youth in the experimental group. Both groups were
given pre- and post-assessments in 1-hr group sessions on the outcomes of interest.
3. A developmental psychologist observed children who attended daycare at least 30 hours
per week and compared their behavior to their peers being raised at home. The psychologist
found that the daycare children scored significantly higher than the other children on a
measure of aggression.