Comprehensive
Exam Review
Click the LEFT mouse key ONCE to continue
Research and
Program
Evaluation
Part 2
Click the LEFT mouse key ONCE to continue
Threats to Internal and External
Experimental Design Validity
The “threats” to experimental
design validity are potential
sources of invalidity in the results
of an experimental study.
Internal threats to design validity have to do
with procedures or methodology.
External threats to design validity have to do
with generalizability of the results.
Internal
Threats
Contemporary History is invalidation
resulting from events that occur during the
course of the experiment (e.g., exposure to
media events, life situation changes, or
random personal activities).
Maturation is invalidation resulting from
changes in the subjects during the course (i.e.,
time period) of the experiment (e.g., developmental change, fatigue, or hunger).
Testing (sometimes called pre-test sensitivity)
is invalidation resulting from changes in
subjects’ scores on a subsequent test due to
having taken the test on a previous occasion
(i.e., changes due to test content familiarity or
recall).
Statistical Regression is invalidation resulting
from the “natural” regression to the mean
among persons initially at an extreme (e.g.,
inclination to homeostasis).
Selection is invalidation resulting from
initial differences between subjects in the
various groups (e.g., selection bias).
Attrition (sometimes called Experimental
Mortality) is invalidation resulting from
subjects dropping out of participation in a
study (e.g., differential attrition across groups).
Instrumentation is invalidation resulting from
changes in the measurement instruments or
procedures (e.g., calibration, accuracy,
interpretation, or experience).
Interactions Among Factors is invalidation
resulting from either confounding or
compounding combinations of the preceding
seven threats.
Compensatory Equalization of Treatments is
invalidation resulting from “compensation” (e.g.,
special treatment) being provided to a control
group presumed not to be receiving the
treatment.
Resentful Demoralizing of Subjects in Less
Desirable Treatment Conditions is invalidation
resulting from subjects in one (e.g., alternative
treatment or control) group “giving up” on
performing appropriately.
Compensatory Rivalry by Subjects Receiving
Less Desirable Treatments is invalidation
resulting from members of the control group
attempting to “outperform” the experimental
group (e.g., to demonstrate equivalent
competence).
Diffusion or Imitation of Treatments is
invalidation resulting from intra- or intergroup communication among subjects (e.g.,
“helping” one another during the course of
an experiment).
External Threats
Reactive Effects of Testing is a
generalizability limitation resulting from the
difference between the experimental and
real life situations (e.g., pre-test sensitization
is not present in real life situations).
Interaction of Selection and Treatment is a
generalizability limitation resulting from
experimental subjects being particularly
susceptible to the treatment (e.g., volunteers
may be particularly motivated to change).
Reactive Effects of Experimental Arrangements is a generalizability limitation resulting
from lack of comparability between the
experimental (e.g., laboratory) and real world
setting.
Multiple-Treatment Interference is a generalizability limitation resulting from either
the sequential or concurrent application of
various treatments.
Single-Subject Designs
Single-subject experimental designs are
sometimes known as N = 1 or as intrasubject designs.
Single-subject experimental designs are usually
used to assess effectiveness in changing a
specific behavior of an individual.
Common features of single-subject designs
include:
Clear specification of the treatment goal,
usually presented as a behavioral goal (i.e.,
change in a targeted behavior).
A focus on changing only one variable at a time
during the treatment phase(s).
Repeated measurement of the target variable
throughout the experiment.
Consistency and stability of baseline and
treatment conditions.
Diligent description of all the conditions
under which the target behavior is observed
(i.e., measured).
Common notations for single-subject
designs:
A designates the baseline condition.
B designates the treatment condition
The A-B design is the simplest, but least
interpretable, of the single-subject designs.
The premise of the A-B design is that after
the baseline is established, the condition
(i.e., behavior) would have continued in
the same pattern had the treatment not
been introduced.
The A-B design
A
B
Baseline
Treatment
X X X X X
O O O O O
O O O O O
Time
The A-B-A design is sometimes known as
the “reversal” or “withdrawal” design.
The A-B-A design allows a strong causal
inference if the target behavior returns
to the baseline level after the treatment
is terminated.
The A-B-A design
A
B
Baseline
Treatment
A
Baseline
X X X X X
O O O O O
O O O O O
Time
O O O O O
The A-B-A-B design affords the strongest
causal inference about the effect of the
treatment.
The A-B-A-B design has the advantage of
ending the experiment with the treatment
in effect (which may be desirable for
ethical reasons).
The A-B-A-B design
A
B
A
B
Baseline
Treatment
Baseline
Treatment
X X X X X
O O O O O
O O O O O
X X X X X
O O O O O
Time
O O O O O
The Randomized A-B design involves a
randomized presentation of the A and B (i.e.,
baseline and treatment) conditions over
time.
Due to the random presentation of conditions,
parametric statistical analyses can be applied
to the resultant data.
The Randomized A-B design also can be
used to analyze “carryover” effects of the
treatment.
Multiple-Baseline designs employ the A-B
design logic, but allow collection of data on
two or more actions, subjects, situations, or
combinations thereof.
In Multiple-Baselines Across Behavior designs,
baselines are established for two or more
independent behaviors for one subject.
The (same) treatment is first applied to one
behavior, then to the next, and so on in
sequence using the same time interval for
each application of the treatment.
In Multiple-Baselines Across Situations
designs, the same behavior is targeted across
situations (e.g., settings).
The A-B design is replicated (i.e., same intervention and time periods) across situations
and then the results are compared.
In Multiple-Baselines Across Individuals
design, the behavior and the situation are held
constant.
The establishment of a baseline and subsequent application of the treatment proceed sequentially across individuals.
Computers and Research
Computers serve three major functions in
regard to research: data management,
document production, and internet
transversal.
The data management functions include uses
for data gathering, storage, and
manipulation.
The data gathering functions may include
use of optical scanners and computer-based
assessments.
The data storage functions include dataentry and retrieval as well as storage.
The data storage functions are facilitated
through use of spreadsheet and data-base
programs.
Spreadsheet programs are designed to make
data entry and understanding as easy as
possible by using a row-column format.
Database programs are designed to make
data-entry and reporting functions as easy as
possible through “free format” procedures.
The data manipulation functions include data
analyses and other procedures to clarify the
nature of the data.
The document production functions
primarily include using word processors to
develop manuscript and/or electronic forms
of communications of research.
The most important word processing
functions include formatting, spellchecking, and grammar-checking.
A current trend in dissemination of research
documents is “publication” on the internet.
The internet serves four primary functions in
regard to research: aiding collaboration
among researchers, simplifying information
gathering, facilitating data collection, and
fostering professional discourse.
A major, current trend is increased postpublication electronic communication about
research results and implications.
Legal Aspects of Research
General legal principles for research include
that:
Subjects have the right to freedom from
invasion of privacy.
Subjects’ participation in research is always
voluntary.
Any type of research is permitted if
appropriate procedures are followed.
Researchers are responsible for whatever
happens to subjects during or as a result of
participation in the research.
The Family Educational Rights and Privacy Act
(FERPA) of 1974, also known as the Buckley
Amendment, stipulates that:
written parental permission is required for
access to students’ records.
person-data linkages should be avoided (i.e.,
data should remain anonymous) whenever
possible.
A written request for data about a student must
include:
exact specification of the data requested,
explanation of how the data are to be used, and
explanation of to whom the data will be
disclosed and how it will be disseminated.
The data obtained can be used only for the
purpose for which the request was made!
The National Research Act (NRA) of 1974
applies to all persons associated with any
institution and/or agency that receives any
federal funds.
This act established the requirements for
“informed consent” procedures in research.
Institutional Review Boards (IRBs), also
known as “human subjects committees,” were
established to ensure that the provisions of the
NRA are implemented effectively and legally.
There is no single informed consent form. An
informed consent form is created by the
researcher(s) for each research project.
The informed consent form must communicate
clearly:
that participation in a research activity is being
requested.
the duration (and/or frequency of occurrence)
of the participation requested.
an explanation of the nature of the participation requested in language that the
person receiving the letter can be reasonably expected to understand.
a (brief) description of the research
procedures.
specific identification of any procedures that
are experimental.
a description of any potential risks and / or
discomfort resulting from participation.
a description of any benefits of
participation, specifically including
whether there will or will not be
monetary compensation.
a statement to the extent of confidentiality
and/or anonymity to be accorded to
participants.
sources of assistance should harm or
discomfort occur during or as a result of
participation.
the name and contact information to
which to direct questions or inquiries
about the research.
a statement that participation is completely
voluntary.
a statement that discontinuing
participation at any time will NOT result
in any form of harm, recourse, or loss of
benefits to the participant.
Each subject must be given a copy of the informed consent form (separate from the one
retained by the researcher).
This concludes Part 2 of the
presentation on
RESEARCH AND
PROGRAM
DEVELOPMENT