Lab 9—Last Lab
Statistical Analyses & Threats to
Validity
Background

Recall from last lab that there are two major types
of designs.
 If
you are interested in the association between
continuous variables, you should use a correlational
design.
 If you are interested in comparing two (or more)
groups, it is likely you will use an experimental design.

The design you use will determine:
 statistical
analysis you will perform
 threats to validity you will most likely need to address
 conclusions you can make from your research.
Correlational/Observational
Design


In this design, you are not comparing groups. Rather,
you are examining the association between two
continuous variables.
You can use either correlation or regression analysis
to analyze your data.
Correlation



Used when interested in assessing the strength of
association between two variables
Used with continuous variables
Example research hypothesis: are higher levels of
self-esteem associated with lower levels of
depression (what direction correlation would be
expected?)
Regression
Simple Linear Regression is a statistical method
concerned with the amount of variance that one
variable accounts for in another variable (that is,
the amount of shared variance between
variables).
You also want to use regression when you are
predicting one variable from the other.
Used when you have continuous variables (e.g.,
motivation measured on a Likert scale, body
weight).
Regression
Shared variance
Variable X
Variable Y
http://www.math.csusb.edu/faculty/stanton/m262/regress/regress.html
Correlation vs. Regression



Both correlation and regression are concerned with the
amount of variance that variables share
The major difference is that with correlation we are only
interested in an association between variables, but with
regression we assign one variable to be the independent
variable (x; the predictor) and the other to be the
dependent variable (y; the criterion)
Note: While we typically are concerned with predictive
relationships when using regression, the certainty with
which we can say that two variables are causally related
depends upon the design (true experiments are
best!...why?) and quality of the study
Comparing Groups


If you have an experimental design (manipulation
of IV), you should use the following analyses to
compare groups (non-continuous variables).
In some cases, you will want to compare groups
even when you don’t manipulate anything.
 Comparing
male versus female on performance.
 Examining differences in opinion of various political or
religious groups.
Independent samples (between-groups)
t-test




Used to determine dependent variable mean differences
between two different groups
Used with categorical or dichotomous variables (e.g.,
gender, democrat or republican, yes/no responses)
Only one independent variable with only two
levels/groups
Example research hypothesis: do women have higher
levels of depression on average than do men?
Analysis of Variance (ANOVA)


Like t-test, used to determine dependent variable
mean differences between groups/levels, but can
use when there are more than two groups (which
is the same as more than two levels of your IV)
Remember, though, you’re still only using one IV,
just there are more than two groups/levels (e.g.,
you might have the IV of political party, with the
groups Democrat, Republican, and Independent)
Analysis of Variance (ANOVA)



Can use for your final project if you have just one IV with
more than two groups (i.e., one-way ANOVA), but can be
used for more than one IV (i.e. two-way ANOVA, where
say you have political party [3 groups] that may differ by
gender [2 groups] for a 2x3 two-way ANOVA)
Independent variable(s) have two or more levels; if you
only have one IV with two levels, though, use a t-test
Example research question: Do psychology graduate
students have higher IQs than medical students, physics
graduate students, and English graduate students?
Repeated measures (within-groups) ttest or ANOVA



Used to test mean differences in the DV for the group
measured at two or more times (e.g., the same group of
clients is exposed to every type of counseling [the IV]
and symptoms [the DV] are measured)
Remember, the t-test involves an independent variable
with two levels; ANOVA involves independent
variable(s) with more than two levels.
Population parameters and sample statistics the same as
between-group t-test and ANOVA above.
Repeated measures (within-groups) ttest or ANOVA


Example research hypothesis for repeated-measures ttest: Do people like the taste of Pepsi more than Coke
(same people would taste both types of pop)?
Example research hypothesis for repeated-measures
ANOVA: Does exposure to death metal music lead to
better cognitive performance than exposure to jazz,
classical, or country music (same people would listen to
each type of music [IV] and perform a cognitive task
[DV])?
THREATS TO VALIDITY
Threats to Internal & External Validity

Keep in mind that in science it is always important to
balance multiple concerns.
 Ethics:
Harm versus Benefit
 Sampling: Representativeness versus Practicality

When choosing a research design, it is important to
address whether:
 You
can address or eliminate alternative explanations
for your results (Internal Validity).
 You can generalize your results (External Validity).
Research Designs & Validity

In general, experiments with random assignment
are high on internal validity.
 Why?


(What does random assignment do?)
Correlational studies are often higher than
experiments on external validity but lower on
internal validity.
Ideally, you would want studies to both be high
internal and external validities.
 No
single method does this.
 Multiple methods important.
An example



You are interested in studying why individuals help
each other. You believe that sadness leads to
helping because it makes individuals feel better.
Let’s say you conduct your study on September 12,
2001. How do you think this would influence your
results?
To the extent that other explanations (outside of
sadness) can account for the associations that you
find, your internal validity is threatened.
Internal Validity Threats

Given that the IV is related to the DV, is it plausibly causal from one
operational variable to another?







History – is there something about the experience & history of one group
that could lead to differences?
Maturation – would natural development influence the associations?
Testing – does actual participation in the study change behavior or
responses?
Instrumentation – what happens if the instruments you use change during the
course of a study or if different groups of participants interpret your
questions differently?
Regression towards the mean – if you start with extreme groups, they are
likely to be less extreme at another time point.
Selection – are participants in your study somehow different than those who
are not?
Mortality – those who drop out of your study could be qualitatively different
than those who remain.
Internal Validity Threats
 Given
that the IV is related to the DV, is it plausibly
causal from one operational variable to another?
Diffusion or imitation – what would happen if participants in the
different conditions communicated?
 Interaction with Selection – what if any of the above happen
differentially for one group?
 Direction of causality – can the association happen in the
opposite direction than what you predicted?
 Compensatory equalization of treatments – what if
experimenters know what conditions participants are in?
 Compensatory rivalry – what would happen if participants in the
different conditions started competing?
 Resentful demoralization – is it possible that participants in one
condition give up?

External Validity Threats

Given that there is probably a causal relationship
from construct A to construct B, how generalizable is
this relationship across persons, settings, and times?
 Interaction
of selection and treatment (sample to
population)
 Interaction of setting and treatment
 Interaction of history and treatment (time)
For your paper



You will need to discuss the internal and external
threats to validity in the discussion of your paper.
Keep in mind that no study is perfect.
However, when you bring up threats to validity, you
MUST explain:
 How
it is a threat to validity
 How your results would change if the threat were
addressed
 What you could do to address the threat

In other words, DO NOT simply state the threat.
Homework 9


For each of the three research hypotheses described below, list what type
of statistical test you would use to test it (independent (between-groups) ttest, between-groups ANOVA, repeated measures t-test, repeated
measures ANOVA, correlation, regression), and whether external or internal
validity is most threatened and why (6 points; 2 for each hypothesis).
Research Hypotheses:
 Women who experience a great deal of stress during pregnancy are
more likely to give birth to low birth weight or premature babies than
pregnant women who do not experience a great deal of stress.

The degree to which someone believes in God relates to his/her
psychological well-being. Further, the more someone believes in God
the higher his/her well-being will be.
Homework 9


Research Hypotheses:
 A new manualized (standardized) treatment for depression will
decrease depressive symptoms in a group of people seeking help at a
mental health clinic. They will be assessed when they come into the
clinic, immediately after the treatment is completed, and 6 months after
treatment is complete.
Lastly, describe your hypothesis, what design you will use for your project
(from previous homework), what statistical analysis you plan to conduct,
and identify 2 threats to validity (any combination of external or internal)
and how they could influence the interpretation of your results. (4 points)