Chapter Thirteen.
External Validity and Critiquing
Experimental Research
External Validity: Generalizing
Your Experiment to the Outside

The second type of evaluation that you must make of your
experiment involves external validity.
External Validity: Generalizing
Your Experiment to the Outside


The second type of evaluation that you must make of your
experiment involves external validity.
When you consider external validity, you are asking a question
about generalization.
External Validity: Generalizing
Your Experiment to the Outside

External validity
External Validity: Generalizing
Your Experiment to the Outside

External validity

A type of evaluation of your experiment that asks whether your
experimental results apply to populations and situations that are
different form those of your experiment.
External Validity: Generalizing
Your Experiment to the Outside

External validity


A type of evaluation of your experiment that asks whether your
experimental results apply to populations and situations that are
different form those of your experiment.
Generalization
External Validity: Generalizing
Your Experiment to the Outside

External validity


A type of evaluation of your experiment that asks whether your
experimental results apply to populations and situations that are
different form those of your experiment.
Generalization


Applying the results from an experiment to a different situation or
population.
In essence, we would like to take our results beyond the narrow
confines of our specific experiment.
External Validity: Generalizing
Your Experiment to the Outside

External validity


A type of evaluation of your experiment that asks whether your
experimental results apply to populations and situations that are
different form those of your experiment.
Generalization



Applying the results from an experiment to a different situation or
population.
In essence, we would like to take our results beyond the narrow
confines of our specific experiment.
Generalization is an important aspect for any science.
External Validity: Generalizing
Your Experiment to the Outside

There are three customary types of generalization in which we
are interested.
External Validity: Generalizing
Your Experiment to the Outside

There are three customary types of generalization in which we
are interested.

Population generalization
External Validity: Generalizing
Your Experiment to the Outside

There are three customary types of generalization in which we
are interested.


Population generalization
Environmental generalization
External Validity: Generalizing
Your Experiment to the Outside

There are three customary types of generalization in which we
are interested.



Population generalization
Environmental generalization
Temporal generalization
External Validity: Generalizing
Your Experiment to the Outside

There are three customary types of generalization in which we
are interested.
 Population generalization



Applying the results from an experiment to a group of
participants that is different and more encompassing than
those used in the original experiment.
Environmental generalization
Temporal generalization
External Validity: Generalizing
Your Experiment to the Outside

There are three customary types of generalization in which we
are interested.



Population generalization
Environmental generalization
 Applying the results from an experiment to a situation or
environment that differs from that of the original experiment.
Temporal generalization
External Validity: Generalizing
Your Experiment to the Outside

There are three customary types of generalization in which we
are interested.



Population generalization
Environmental generalization
Temporal generalization
 Applying the results from an experiment to a time that is
different from that when the original experiment was
conducted.
Threats to External Validity
(Based on Methods)

Here is a summary of factors relating to external validity from
Campbell and Stanley (1966):
Threats to External Validity
(Based on Methods)

Here is a summary of factors relating to external validity from
Campbell and Stanley (1966):

Interaction of Testing and Treatment
Threats to External Validity
(Based on Methods)

Here is a summary of factors relating to external validity from
Campbell and Stanley (1966):

Interaction of Testing and Treatment
 A threat to external validity that occurs when a pretest
sensitizes participants to the treatment yet to come.
Threats to External Validity
(Based on Methods)

Here is a summary of factors relating to external validity from
Campbell and Stanley (1966):

Interaction of Testing and Treatment
 A threat to external validity that occurs when a pretest
sensitizes participants to the treatment yet to come.
 Occurs for the pretest-posttest control group design
Threats to External Validity
(Based on Methods)

Here is a summary of factors relating to external validity from
Campbell and Stanley (1966):

Interaction of Testing and Treatment
 A threat to external validity that occurs when a pretest
sensitizes participants to the treatment yet to come.
 Occurs for the pretest-posttest control group design
 Because of a pretest, your participants’ reaction to the
treatment will be different.
Threats to External Validity
(Based on Methods)

Here is a summary of factors relating to external validity from
Campbell and Stanley (1966):


Interaction of Testing and Treatment
Interaction of Selection and Treatment
Threats to External Validity
(Based on Methods)

Here is a summary of factors relating to external validity from
Campbell and Stanley (1966):


Interaction of Testing and Treatment
Interaction of Selection and Treatment
 A threat to external validity that can occur when a treatment
effect is found only for a specific sample of participants.
Threats to External Validity
(Based on Methods)

Here is a summary of factors relating to external validity from
Campbell and Stanley (1966):


Interaction of Testing and Treatment
Interaction of Selection and Treatment
 A threat to external validity that can occur when a treatment
effect is found only for a specific sample of participants.
 Occurs when the effects that you demonstrate hold true only
for the particular groups that you selected for your experiment.
Threats to External Validity
(Based on Methods)

Here is a summary of factors relating to external validity from
Campbell and Stanley (1966):


Interaction of Testing and Treatment
Interaction of Selection and Treatment
 A threat to external validity that can occur when a treatment
effect is found only for a specific sample of participants.
 Occurs when the effects that you demonstrate hold true only
for the particular groups that you selected for your experiment.
 Treatment interaction becomes greater as it becomes more
difficult to find participants for your experiment (Campbell &
Stanley, 1966).
Threats to External Validity
(Based on Methods)

Here is a summary of factors relating to external validity from
Campbell and Stanley (1966):



Interaction of Testing and Treatment
Interaction of Selection and Treatment
Reactive Arrangements
Threats to External Validity
(Based on Methods)

Here is a summary of factors relating to external validity from
Campbell and Stanley (1966):



Interaction of Testing and Treatment
Interaction of Selection and Treatment
Reactive Arrangements
 A threat to external validity caused by an experimental
situation that alters participants’ behavior, regardless of the IV
involved.
Threats to External Validity
(Based on Methods)

Here is a summary of factors relating to external validity from
Campbell and Stanley (1966):



Interaction of Testing and Treatment
Interaction of Selection and Treatment
Reactive Arrangements
 A threat to external validity caused by an experimental
situation that alters participants’ behavior, regardless of the IV
involved.
 We cannot be sure that the behaviors we observe in the
experiment will generalize outside that setting because the
artificial conditions of the experiment do not exist in the real
world.
Threats to External Validity
(Based on Methods)

Here is a summary of factors relating to external validity from
Campbell and Stanley (1966):




Interaction of Testing and Treatment
Interaction of Selection and Treatment
Reactive Arrangements
Demand characteristics
Threats to External Validity
(Based on Methods)

Here is a summary of factors relating to external validity from
Campbell and Stanley (1966):




Interaction of Testing and Treatment
Interaction of Selection and Treatment
Reactive Arrangements
Demand characteristics
 Features from the experiment that inadvertently lead
participants to respond in a particular manner.
Threats to External Validity
(Based on Methods)

Here is a summary of factors relating to external validity from
Campbell and Stanley (1966):




Interaction of Testing and Treatment
Interaction of Selection and Treatment
Reactive Arrangements
Demand characteristics
 Features from the experiment that inadvertently lead
participants to respond in a particular manner.
 Demand characteristics make generalizations difficult because it
is not clear from a set of research findings whether the
participants are responding to an experiment’s IV, its demand
characteristics, or both.
Threats to External Validity
(Based on Methods)

Here is a summary of factors relating to external validity from
Campbell and Stanley (1966):





Interaction of Testing and Treatment
Interaction of Selection and Treatment
Reactive Arrangements
Demand characteristics
Multiple-Treatment Interference
Threats to External Validity
(Based on Methods)

Here is a summary of factors relating to external validity from
Campbell and Stanley (1966):





Interaction of Testing and Treatment
Interaction of Selection and Treatment
Reactive Arrangements
Demand characteristics
Multiple-Treatment Interference
 A threat to external validity that occurs when a set of findings
results only when participants experience multiple treatments
in the same experiment ( repeated measures designs).
Threats to External Validity
(Based on Our Participants)
Threats to External Validity
(Based on Our Participants)

The Infamous White Rat
Threats to External Validity
(Based on Our Participants)

The Infamous White Rat

If you are interested in the behavior of subhumans, generalizing
from rats (and pigeons) to all other animals may be a stretch.
Threats to External Validity
(Based on Our Participants)

The Infamous White Rat


If you are interested in the behavior of subhumans, generalizing
from rats (and pigeons) to all other animals may be a stretch.
If you are interested in generalizing from animal to human
behavior, there are certainly closer approximations to humans (and
pigeons) in the animal kingdom.
Threats to External Validity
(Based on Our Participants)


The Infamous White Rat
The Ubiquitous College Student
Threats to External Validity
(Based on Our Participants)


The Infamous White Rat
The Ubiquitous College Student

Psychologists who want to conduct human research turn to a
ready, convenient source of human participants – students in
introductory psychology courses (a technique referred to as
convenience sampling).
Threats to External Validity
(Based on Our Participants)


The Infamous White Rat
The Ubiquitous College Student


Psychologists who want to conduct human research turn to a
ready, convenient source of human participants – students in
introductory psychology courses (a technique referred to as
convenience sampling).
Convenience Sampling
Threats to External Validity
(Based on Our Participants)


The Infamous White Rat
The Ubiquitous College Student


Psychologists who want to conduct human research turn to a
ready, convenient source of human participants – students in
introductory psychology courses (a technique referred to as
convenience sampling).
Convenience Sampling
 A researcher’s sampling of participants based on ease of
locating the participants; often does not involve true random
selection.
Threats to External Validity
(Based on Our Participants)



The Infamous White Rat
The Ubiquitous College Student
The “Opposite” or “Weaker” or “Inferior” or “Second” Sex
Threats to External Validity
(Based on Our Participants)



The Infamous White Rat
The Ubiquitous College Student
The “Opposite” or “Weaker” or “Inferior” or “Second” Sex

All four of these derogatory labels have been applied to women at
various points in time.
Threats to External Validity
(Based on Our Participants)



The Infamous White Rat
The Ubiquitous College Student
The “Opposite” or “Weaker” or “Inferior” or “Second” Sex


All four of these derogatory labels have been applied to women at
various points in time.
The supposed inferiority of women has carried over into some
psychological theories.
Threats to External Validity
(Based on Our Participants)



The Infamous White Rat
The Ubiquitous College Student
The “Opposite” or “Weaker” or “Inferior” or “Second” Sex


All four of these derogatory labels have been applied to women at
various points in time.
The supposed inferiority of women has carried over into some
psychological theories.
 Freud’s theories
Threats to External Validity
(Based on Our Participants)



The Infamous White Rat
The Ubiquitous College Student
The “Opposite” or “Weaker” or “Inferior” or “Second” Sex


All four of these derogatory labels have been applied to women at
various points in time.
The supposed inferiority of women has carried over into some
psychological theories.
 Freud’s theories
 Erikson’s theory of psychosocial crises (“Eight Stages of Man”)
Threats to External Validity
(Based on Our Participants)



The Infamous White Rat
The Ubiquitous College Student
The “Opposite” or “Weaker” or “Inferior” or “Second” Sex

Carol Tavris’s (1992) thesis is that “despite women’s gains in many
fields in the last twenty years, the fundamental belief in the
normalcy of men, and the corresponding abnormality of women,
has remained virtually untouched” (p. 17)
Threats to External Validity
(Based on Our Participants)




The Infamous White Rat
The Ubiquitous College Student
The “Opposite” or “Weaker” or “Inferior” or “Second” Sex
Even the Rats and Students Were White
Threats to External Validity
(Based on Our Participants)




The Infamous White Rat
The Ubiquitous College Student
The “Opposite” or “Weaker” or “Inferior” or “Second” Sex
Even the Rats and Students Were White

Just as history has failed to record the accomplishments of many
women throughout time, it has largely ignored the
accomplishments of African Americans and other minority groups.
Threats to External Validity
(Based on Our Participants)




The Infamous White Rat
The Ubiquitous College Student
The “Opposite” or “Weaker” or “Inferior” or “Second” Sex
Even the Rats and Students Were White


Just as history has failed to record the accomplishments of many
women throughout time, it has largely ignored the
accomplishments of African Americans and other minority groups.
When we conduct research and make generalizations, we should
be cautious that we do not exclude minority groups from our
considerations.
Threats to External Validity
(Based on Our Participants)





The Infamous White Rat
The Ubiquitous College Student
The “Opposite” or “Weaker” or “Inferior” or “Second” Sex
Even the Rats and Students Were White
Even the Rats, Students, Women, and Minorities Were American
Threats to External Validity
(Based on Our Participants)





The Infamous White Rat
The Ubiquitous College Student
The “Opposite” or “Weaker” or “Inferior” or “Second” Sex
Even the Rats and Students Were White
Even the Rats, Students, Women, and Minorities Were American

Although experimental psychology’s early roots are based in Europe, this
aspect of the discipline quickly became Americanized, largely due to the
influence of John B. Watson’s behaviorism.
Threats to External Validity
(Based on Our Participants)





The Infamous White Rat
The Ubiquitous College Student
The “Opposite” or “Weaker” or “Inferior” or “Second” Sex
Even the Rats and Students Were White
Even the Rats, Students, Women, and Minorities Were American

In the mid-1960’s, psychologists started taking culture and ethnicity
more seriously. The field of cross-cultural psychology has
evolved from those changes that began in the 1960’s.
Threats to External Validity
(Based on Our Participants)

Cross-cultural psychology


A branch of psychology whose goal is to determine the universality
of research results.
Ethnocentricity

Other cultures are viewed as an extension of one’s own.
The Devil’s Advocate: Is External
Validity Always Necessary?

Mook (1983) pointed out four alternative goals of research that
do not stress external validity:
The Devil’s Advocate: Is External
Validity Always Necessary?

Mook (1983) pointed out four alternative goals of research that
do not stress external validity:

We may merely want to find out if something can happen (not
whether it actually happens).
The Devil’s Advocate: Is External
Validity Always Necessary?

Mook (1983) pointed out four alternative goals of research that
do not stress external validity:


We may merely want to find out if something can happen (not
whether it actually happens).
We may be predicting from the real world to the lab – seeing a
phenomenon in the real world, we think it will operate in a certain
manner in the lab.
The Devil’s Advocate: Is External
Validity Always Necessary?

Mook (1983) pointed out four alternative goals of research that
do not stress external validity:



We may merely want to find out if something can happen (not
whether it actually happens).
We may be predicting from the real world to the lab – seeing a
phenomenon in the real world, we think it will operate in a certain
manner in the lab.
If we can demonstrate that a phenomenon occurs in a lab’s
unnatural setting, the validity of the phenomenon may actually be
strengthened.
The Devil’s Advocate: Is External
Validity Always Necessary?

Mook (1983) pointed out four alternative goals of research that
do not stress external validity:




We may merely want to find out if something can happen (not
whether it actually happens).
We may be predicting from the real world to the lab – seeing a
phenomenon in the real world, we think it will operate in a certain
manner in the lab.
If we can demonstrate that a phenomenon occurs in a lab’s
unnatural setting, the validity of the phenomenon may actually be
strengthened.
We may study phenomena in the lab that don’t even have a realworld analogy.
The Devil’s Advocate: Is External
Validity Always Necessary?

Replication
The Devil’s Advocate: Is External
Validity Always Necessary?

Replication

An additional scientific study that is conducted in exactly the same
manner as the original research project.
The Devil’s Advocate: Is External
Validity Always Necessary?

Replication


An additional scientific study that is conducted in exactly the same
manner as the original research project.
When we replicate an experimental finding, we are able to place
more confidence in that result.
The Devil’s Advocate: Is External
Validity Always Necessary?

Replication



An additional scientific study that is conducted in exactly the same
manner as the original research project.
When we replicate an experimental finding, we are able to place
more confidence in that result.
Replication with extension
The Devil’s Advocate: Is External
Validity Always Necessary?

Replication



An additional scientific study that is conducted in exactly the same
manner as the original research project.
When we replicate an experimental finding, we are able to place
more confidence in that result.
Replication with extension

An experiment that seeks to confirm (replicate) a previous finding
but does so in a different setting or with different participants or
under different conditions.
Guidelines for Critiquing
Psychological Research Literature

Does the literature review adequately describe the research
area? Is this material consistent with the specific research
question?
Guidelines for Critiquing
Psychological Research Literature

Does the literature review adequately describe the research
area? Is this material consistent with the specific research
question?

As a research project evolves, the literature review and the actual
experiment diverge somewhat over time.
Guidelines for Critiquing
Psychological Research Literature

Does the literature review adequately describe the research
area? Is this material consistent with the specific research
question?


As a research project evolves, the literature review and the actual
experiment diverge somewhat over time.
After you complete your project and work on the report, doublecheck to make certain that the actual project still shows a direct
link with your research literature.
Guidelines for Critiquing
Psychological Research Literature

Does the literature review adequately describe the research
area? Is this material consistent with the specific research
question?



As a research project evolves, the literature review and the actual
experiment diverge somewhat over time.
After you complete your project and work on the report, doublecheck to make certain that the actual project still shows a direct
link with your research literature.
Because most researchers carry out programmatic research,
their new research ideas are likely to build directly on their (and
others’) previous research.
Guidelines for Critiquing
Psychological Research Literature

Is the research question stated clearly? Do you have a clear
idea concerning the research to be reported?
Guidelines for Critiquing
Psychological Research Literature

Is the research question stated clearly? Do you have a clear
idea concerning the research to be reported?

The title and abstract of a research report should give you an
indication of the research’s topic, although they may not contain
the specific question per se.
Guidelines for Critiquing
Psychological Research Literature

Is the research question stated clearly? Do you have a clear
idea concerning the research to be reported?


The title and abstract of a research report should give you an
indication of the research’s topic, although they may not contain
the specific question per se.
You will find the author’s review of relevant literature in the article’s
introduction. As you read further into the introduction, the
literature should apply more specifically to the particular research
question. The research question will often be in the last paragraph
of the introduction.
Guidelines for Critiquing
Psychological Research Literature

In view of the research area and research question, are the
hypotheses appropriate, clearly stated, and able to be stated in
general implication form?
Guidelines for Critiquing
Psychological Research Literature

In view of the research area and research question, are the
hypotheses appropriate, clearly stated, and able to be stated in
general implication form?

Appropriate hypotheses are those that follow logically from the
literature review.
Guidelines for Critiquing
Psychological Research Literature

In view of the research area and research question, are the
hypotheses appropriate, clearly stated, and able to be stated in
general implication form?


Appropriate hypotheses are those that follow logically from the
literature review.
If you find a hypothesis that seems to come from nowhere and
surprises you, it may be inappropriate – reread the introduction to
make sure.
Guidelines for Critiquing
Psychological Research Literature

In view of the research area and research question, are the
hypotheses appropriate, clearly stated, and able to be stated in
general implication form?



Appropriate hypotheses are those that follow logically from the
literature review.
If you find a hypothesis that seems to come from nowhere and
surprises you, it may be inappropriate – reread the introduction to
make sure.
A clearly stated hypothesis is one that you can easily understand
without having to guess what the researcher is predicting.
Guidelines for Critiquing
Psychological Research Literature

In view of the research area and research question, are the
hypotheses appropriate, clearly stated, and able to be stated in
general implication form?




Appropriate hypotheses are those that follow logically from the
literature review.
If you find a hypothesis that seems to come from nowhere and
surprises you, it may be inappropriate – reread the introduction to
make sure.
A clearly stated hypothesis is one that you can easily understand
without having to guess what the researcher is predicting.
Remember that general implication form is the “if….then” format.
Guidelines for Critiquing
Psychological Research Literature

Are the key terms operationally defined?
Guidelines for Critiquing
Psychological Research Literature

Are the key terms operationally defined?

The reader should not have to guess what a researcher means
when he or she refers to a specific independent, dependent, or
extraneous variable.
Guidelines for Critiquing
Psychological Research Literature

Are the key terms operationally defined?


The reader should not have to guess what a researcher means
when he or she refers to a specific independent, dependent, or
extraneous variable.
Remember that operational definitions mean that you should define
your variables in terms of the operations you use to manipulate,
measure, or control them.
Guidelines for Critiquing
Psychological Research Literature

Are the IV’s and their levels appropriate?
Guidelines for Critiquing
Psychological Research Literature

Are the IV’s and their levels appropriate?

Be sure to pick a manipulation that is actually appropriate to the IV
– don’t choose something merely because it is easy or convenient
to use.
Guidelines for Critiquing
Psychological Research Literature

Are the IV’s and their levels appropriate?


Be sure to pick a manipulation that is actually appropriate to the IV
– don’t choose something merely because it is easy or convenient
to use.
Be sure to choose the levels of your IV appropriately. Choose
levels of the IV to answer your experimental question, but do so
economically (remember the principle of parsimony).
Guidelines for Critiquing
Psychological Research Literature

Is the DV appropriate for this research? Should the researcher
have included more than one DV if only one was recorded?
Guidelines for Critiquing
Psychological Research Literature

Is the DV appropriate for this research? Should the researcher
have included more than one DV if only one was recorded?

If a researcher wishes to study a particular outcome, the behavior
chosen for measuring (the dependent variable) should be a good
indicator of that outcome.
Guidelines for Critiquing
Psychological Research Literature

Is the DV appropriate for this research? Should the researcher
have included more than one DV if only one was recorded?


If a researcher wishes to study a particular outcome, the behavior
chosen for measuring (the dependent variable) should be a good
indicator of that outcome.
The operational definition of the DV should be one that other
researchers would judge to be valid.
Guidelines for Critiquing
Psychological Research Literature

Is the DV appropriate for this research? Should the researcher
have included more than one DV if only one was recorded?



If a researcher wishes to study a particular outcome, the behavior
chosen for measuring (the dependent variable) should be a good
indicator of that outcome.
The operational definition of the DV should be one that other
researchers would judge to be valid.
A researcher with broad interests should use multiple DV’s to get a
better sense of the concept he or she is measuring.
Guidelines for Critiquing
Psychological Research Literature

Are the controls sufficient and appropriate? Are there any
uncontrolled variables that could affect the results of the
experiment?
Guidelines for Critiquing
Psychological Research Literature

Are the controls sufficient and appropriate? Are there any
uncontrolled variables that could affect the results of the
experiment?

Leaving variables uncontrolled can result in a confounded
experiment which leaves the researcher unable to draw a
conclusion.
Guidelines for Critiquing
Psychological Research Literature

Are the controls sufficient and appropriate? Are there any
uncontrolled variables that could affect the results of the
experiment?


Leaving variables uncontrolled can result in a confounded
experiment which leaves the researcher unable to draw a
conclusion.
As you look for possible extraneous variables, you should
concentrate on variables that have a legitimate or reasonable
chance to actually make a difference.
Guidelines for Critiquing
Psychological Research Literature

Are the controls sufficient and appropriate? Are there any
uncontrolled variables that could affect the results of the
experiment?



Leaving variables uncontrolled can result in a confounded
experiment which leaves the researcher unable to draw a
conclusion.
As you look for possible extraneous variables, you should
concentrate on variables that have a legitimate or reasonable
chance to actually make a difference.
Look for extraneous variation, but don’t go overboard and find
variation that most researchers would consider negligible.
Guidelines for Critiquing
Psychological Research Literature

Did the author(s) use an appropriate research design to test the
specific hypotheses and answer the general research question?
Guidelines for Critiquing
Psychological Research Literature

Did the author(s) use an appropriate research design to test the
specific hypotheses and answer the general research question?

With poor planning, it is possible to gather data for which there is
no appropriate research design and, thus, no appropriate statistical
test.
Guidelines for Critiquing
Psychological Research Literature

Did the author(s) use an appropriate research design to test the
specific hypotheses and answer the general research question?


With poor planning, it is possible to gather data for which there is
no appropriate research design and, thus, no appropriate statistical
test.
Make sure that research reports use designs that match the
question(s) they sought to answer.
Guidelines for Critiquing
Psychological Research Literature

Did the author(s) use an appropriate research design to test the
specific hypotheses and answer the general research question?


With poor planning, it is possible to gather data for which there is
no appropriate research design and, thus, no appropriate statistical
test.
Make sure that research reports use designs that match the
question(s) they sought to answer.
 For example, if the researcher asked a question involving
multiple IV’s, the experiment should involve a factorial design.
Guidelines for Critiquing
Psychological Research Literature

Assuming you had access to the appropriate equipment and
materials, could you replicate the research after reading the
method section?
Guidelines for Critiquing
Psychological Research Literature

Assuming you had access to the appropriate equipment and
materials, could you replicate the research after reading the
method section?

The method section should contain enough detail about the
variables and procedures of the experiment to enable a reader to
replicate the experiment.
Guidelines for Critiquing
Psychological Research Literature

Assuming you had access to the appropriate equipment and
materials, could you replicate the research after reading the
method section?


The method section should contain enough detail about the
variables and procedures of the experiment to enable a reader to
replicate the experiment.
The reader should not have to guess about any of the
manipulations, measurements, or controls the researcher used.
Guidelines for Critiquing
Psychological Research Literature

Assuming you had access to the appropriate equipment and
materials, could you replicate the research after reading the
method section?



The method section should contain enough detail about the
variables and procedures of the experiment to enable a reader to
replicate the experiment.
The reader should not have to guess about any of the
manipulations, measurements, or controls the researcher used.
The reader must have all the vital details of the experiment in
order to evaluate the operational definitions, the variables, and the
procedures used in the experiment.
Guidelines for Critiquing
Psychological Research Literature

Did the researcher(s) use appropriate sampling procedures to
select the participants and assign them to groups?
Guidelines for Critiquing
Psychological Research Literature

Did the researcher(s) use appropriate sampling procedures to
select the participants and assign them to groups?

Random sampling and assignment in creating independent groups
or the appropriate matching or repeated measures approach for
correlated groups are important for both internal and external
validity.
Guidelines for Critiquing
Psychological Research Literature

Did the researcher(s) use appropriate sampling procedures to
select the participants and assign them to groups?


Random sampling and assignment in creating independent groups
or the appropriate matching or repeated measures approach for
correlated groups are important for both internal and external
validity.
If a researcher uses sampling techniques that result in biased
samples, the internal validity of the experiment is threatened
because the groups are likely to be different before the experiment.
Guidelines for Critiquing
Psychological Research Literature

Did the researcher(s) use appropriate sampling procedures to
select the participants and assign them to groups?



Random sampling and assignment in creating independent groups
or the appropriate matching or repeated measures approach for
correlated groups are important for both internal and external
validity.
If a researcher uses sampling techniques that result in biased
samples, the internal validity of the experiment is threatened
because the groups are likely to be different before the experiment.
Biased samples also threaten external validity.
Guidelines for Critiquing
Psychological Research Literature

What procedures were used to ensure group equivalence prior
to the experiment?
Guidelines for Critiquing
Psychological Research Literature

What procedures were used to ensure group equivalence prior
to the experiment?

Poor sampling techniques can result in biased samples.
Guidelines for Critiquing
Psychological Research Literature

What procedures were used to ensure group equivalence prior
to the experiment?


Poor sampling techniques can result in biased samples.
Biased samples are usually not equivalent before the experiment
begins, so it would be impossible to draw valid conclusions about
the effects of the IV (internal validity would be compromised).
Guidelines for Critiquing
Psychological Research Literature

What procedures were used to ensure group equivalence prior
to the experiment?



Poor sampling techniques can result in biased samples.
Biased samples are usually not equivalent before the experiment
begins, so it would be impossible to draw valid conclusions about
the effects of the IV (internal validity would be compromised).
If you have reason to doubt the equivalence of your groups
beforehand, you would be wise to use a pretest.
Guidelines for Critiquing
Psychological Research Literature

Did the research use a sufficient number of participants?
Guidelines for Critiquing
Psychological Research Literature

Did the research use a sufficient number of participants?

With small numbers of participants, statistical tests are simply less
powerful to detect differences – the differences between groups
have to be quite large for the difference to turn out significant.
Guidelines for Critiquing
Psychological Research Literature

Did the research use a sufficient number of participants?


With small numbers of participants, statistical tests are simply less
powerful to detect differences – the differences between groups
have to be quite large for the difference to turn out significant.
Don’t back yourself into a corner so that you use the age-old
student lament after your experiment: “If I had run more
participants, my differences might have been significant.”
Guidelines for Critiquing
Psychological Research Literature

Were there any history, instrumentation, statistical regression,
or mortality effects that might have influenced the results?
Guidelines for Critiquing
Psychological Research Literature

Were there any history, instrumentation, statistical regression,
or mortality effects that might have influenced the results?

For history, be alert to outside events that occur that could affect
the results.
Guidelines for Critiquing
Psychological Research Literature

Were there any history, instrumentation, statistical regression,
or mortality effects that might have influenced the results?


For history, be alert to outside events that occur that could affect
the results.
Be sure to check the operation of your equipment before each
session to avoid instrumentation effects.
Guidelines for Critiquing
Psychological Research Literature

Were there any history, instrumentation, statistical regression,
or mortality effects that might have influenced the results?



For history, be alert to outside events that occur that could affect
the results.
Be sure to check the operation of your equipment before each
session to avoid instrumentation effects.
Choosing extreme high- or low-scoring participants can result in
lower or higher scores, respectively, simply due to statistical
regression.
Guidelines for Critiquing
Psychological Research Literature

Were there any history, instrumentation, statistical regression,
or mortality effects that might have influenced the results?




For history, be alert to outside events that occur that could affect
the results.
Be sure to check the operation of your equipment before each
session to avoid instrumentation effects.
Choosing extreme high- or low-scoring participants can result in
lower or higher scores, respectively, simply due to statistical
regression.
If many participants drop out of one condition in the experiment
(i.e., mortality), the participants who are left in that condition may
differ in some important way(s) from the participants in other
conditions.
Guidelines for Critiquing
Psychological Research Literature

Were the appropriate statistical tests used, and are they
reported correctly?
Guidelines for Critiquing
Psychological Research Literature

Were the appropriate statistical tests used, and are they
reported correctly?

You may need to consult a statistic text or someone who teaches
statistics to help you answer this question.
Guidelines for Critiquing
Psychological Research Literature

Were the appropriate statistical tests used, and are they
reported correctly?


You may need to consult a statistic text or someone who teaches
statistics to help you answer this question.
On the other hand, this guideline points out the importance of
becoming statistically knowledgeable so that you can evaluate this
guideline on your own.
Guidelines for Critiquing
Psychological Research Literature

Were the appropriate statistical tests used, and are they
reported correctly?



You may need to consult a statistic text or someone who teaches
statistics to help you answer this question.
On the other hand, this guideline points out the importance of
becoming statistically knowledgeable so that you can evaluate this
guideline on your own.
Remember that statistics are merely a tool experimenters use to
decipher the results they obtained – you should be well armed with
the proper tools as you evaluate and conduct research.
Guidelines for Critiquing
Psychological Research Literature

Did the author(s) report means, standard deviations, and a
measure of effect size?
Guidelines for Critiquing
Psychological Research Literature

Did the author(s) report means, standard deviations, and a
measure of effect size?

Group means may allow the reader to compare participants’
performance against existing norms.
Guidelines for Critiquing
Psychological Research Literature

Did the author(s) report means, standard deviations, and a
measure of effect size?


Group means may allow the reader to compare participants’
performance against existing norms.
Standard deviations may allow the reader to determine that
nonsignificant findings are due to extreme variability between
groups rather than small differences between means.
Guidelines for Critiquing
Psychological Research Literature

Did the author(s) report means, standard deviations, and a
measure of effect size?



Group means may allow the reader to compare participants’
performance against existing norms.
Standard deviations may allow the reader to determine that
nonsignificant findings are due to extreme variability between
groups rather than small differences between means.
Effect sizes give standard comparison units so that readers can
compare significant differences from several different experiments.
Guidelines for Critiquing
Psychological Research Literature

Are the tables and figures clearly and appropriately labeled and
presented accurately?
Guidelines for Critiquing
Psychological Research Literature

Are the tables and figures clearly and appropriately labeled and
presented accurately?

Tables and figures should present a large amount of data than is
possible in writing.
Guidelines for Critiquing
Psychological Research Literature

Are the tables and figures clearly and appropriately labeled and
presented accurately?


Tables and figures should present a large amount of data than is
possible in writing.
Just as paragraph after paragraph of statistical results can be
confusing, a poorly constructed table or figure can confuse the
reader.
Guidelines for Critiquing
Psychological Research Literature

Does the author(s) correctly interpret the results? Does the
discussion follow logically from the results?
Guidelines for Critiquing
Psychological Research Literature

Does the author(s) correctly interpret the results? Does the
discussion follow logically from the results?

Did the researcher correctly interpret p < .05 as significant and p >
.05 as nonsignificant?
Guidelines for Critiquing
Psychological Research Literature

Does the author(s) correctly interpret the results? Does the
discussion follow logically from the results?


Did the researcher correctly interpret p < .05 as significant and p >
.05 as nonsignificant?
Did the researcher give a correct interpretation of his or her results
in light of previous research?
Guidelines for Critiquing
Psychological Research Literature

Does the author(s) correctly interpret the results? Does the
discussion follow logically from the results?



Did the researcher correctly interpret p < .05 as significant and p >
.05 as nonsignificant?
Did the researcher give a correct interpretation of his or her results
in light of previous research?
Does the discussion “make sense” given the data the researcher
just presented.
Guidelines for Critiquing
Psychological Research Literature

Does the author(s) correctly interpret the results? Does the
discussion follow logically from the results?




Did the researcher correctly interpret p < .05 as significant and p >
.05 as nonsignificant?
Did the researcher give a correct interpretation of his or her results
in light of previous research?
Does the discussion “make sense” given the data the researcher
just presented.
Authors should make it clear when conclusions follow from data
and when they are engaging in speculation.
Guidelines for Critiquing
Psychological Research Literature

Are the conclusions and generalizations valid and
justified by the data? Did the author(s) consider
other possible interpretations of the results?
Guidelines for Critiquing
Psychological Research Literature

Are the conclusions and generalizations valid and justified by
the data? Did the author(s) consider other possible
interpretations of the results?

This difficulty often comes when researchers have a favorite theory
that they espouse.
Guidelines for Critiquing
Psychological Research Literature

Are the conclusions and generalizations valid and justified by
the data? Did the author(s) consider other possible
interpretations of the results?


This difficulty often comes when researchers have a favorite theory
that they espouse.
Sometimes, this theoretical leaning is so strong that it seems to
blind them to any alternative explanations.
Guidelines for Critiquing
Psychological Research Literature

Are the conclusions and generalizations valid and justified by
the data? Did the author(s) consider other possible
interpretations of the results?



This difficulty often comes when researchers have a favorite theory
that they espouse.
Sometimes, this theoretical leaning is so strong that it seems to
blind them to any alternative explanations.
Alternative explanations for findings may provide you with the
impetus for a new experiment.
Guidelines for Critiquing
Psychological Research Literature

Do all references cited in the text appear in the reference
section, and vice versa?
Guidelines for Critiquing
Psychological Research Literature

Do all references cited in the text appear in the reference
section, and vice versa?

It is highly unlikely you would find this problem in a published
study.
Guidelines for Critiquing
Psychological Research Literature

Do all references cited in the text appear in the reference
section, and vice versa?


It is highly unlikely you would find this problem in a published
study.
There should be a one-to-one correspondence of the citations in
the text and the references at the end of the study.
Guidelines for Critiquing
Psychological Research Literature

Do all references cited in the text appear in the reference
section, and vice versa?



It is highly unlikely you would find this problem in a published
study.
There should be a one-to-one correspondence of the citations in
the text and the references at the end of the study.
The reference section of an APA-format report consists only of
material that you have read and included in the report.
Guidelines for Critiquing
Psychological Research Literature

Did the experimenter follow appropriate ethical procedures
during all phases of the experiment?
Guidelines for Critiquing
Psychological Research Literature

Did the experimenter follow appropriate ethical procedures
during all phases of the experiment?

To evaluate this guideline, you man need to refresh your memory
of the ethical principles that psychologists follow in conducting
research.
Guidelines for Critiquing
Psychological Research Literature

Did the experimenter follow appropriate ethical procedures
during all phases of the experiment?


To evaluate this guideline, you man need to refresh your memory
of the ethical principles that psychologists follow in conducting
research.
Some older research involves some procedures that have been
hotly debated as far as their ethical nature is concerned.
Guidelines for Critiquing
Psychological Research Literature

Did the experimenter follow appropriate ethical procedures
during all phases of the experiment?



To evaluate this guideline, you man need to refresh your memory
of the ethical principles that psychologists follow in conducting
research.
Some older research involves some procedures that have been
hotly debated as far as their ethical nature is concerned.
It is doubtful that any ethically questionable study would receive
approval from an institutional review board.