Comprehensive
Exam Review
Click the LEFT mouse key ONCE to continue
Research and
Program
Evaluation
Part 1
Click the LEFT mouse key ONCE to continue
Types of Research
Theoretically, basic research is conducted solely
for the purpose of developing and/or refining
theory.
Theoretically, applied research is conducted solely
for the purpose of evaluating the application of
theory for the solution of problems.
This distinction is not, however, a particularly
helpful way to consider research methodologies,
because basic and applied are actually extremes
on a continuum of application and generalizability to other situations.
The two most commonly discussed types of
research are qualitative and quantitative.
Technically, qualitative and quantitative are
different “approaches to inquiry” rather than
distinctly different research methodology
categories.
However, the qualitative / quantitative differentiation facilitates explanation of different
research methodologies.
Qualitative Research
Qualitative research involves collection of
extensive data on many variables over an
extended period of time in a naturalistic setting.
Qualitative research methodologies are
grounded in the belief that behavior is
significantly influenced by the environment in
which it occurs.
Qualitative researchers seek to find holistic, indepth understanding of the phenomenon
observed.
Qualitative research is inductive, and its primary
goal is to promote greater understanding by
explaining how and why people behave the way
they do.
Measurement; data collection, analysis, and
interpretation techniques; and research design
and method in qualitative research are flexible
and evolve as the research process proceeds.
Qualitative research may involve interactive
strategies, such as participant observation, or
noninteractive strategies, such as review of
documents.
Participant or nonparticipant observation are
the most frequently used research strategies in
qualitative research, particularly in the social
sciences.
The primary distinguishing characteristic of
qualitative research is that the actual
methodology used cannot be determined a priori;
the methodology emerges quite literally during
the process and often involves both interactive
and noninteractive strategies.
Qualitative research usually involves purposeful sampling, as opposed to probabilistic
sampling, to ensure that the “right” person(s)
and/or situation(s) are examined.
Generalizability of the results is not a major
concern in qualitative research (in accord with
nonprobabilistic sampling), and usually
generalizability is very restricted.
A qualitative research case study design
involves focus on one phenomenon, such as a
person, a concept, a process, a group, or a
program.
Qualitative case study designs are often used to:
describe and analyze a situation, event, or
process,
develop a concept or model,
evaluate a program,
investigate social and cultural beliefs, or
serve as a prelude to quantitative research.
Ethnographic research is sometimes considered synonymous with qualitative research
and sometimes as a subtype of qualitative
research.
In the latter perspective, ethnographic research
is usually interpreted to mean the study of a
“culture.”
In this context, a “culture” is any group of
people who regularly associate with one
another and who develop characteristic ways
of behaving and thinking.
Other key concepts in qualitative research
include:
Participant observation, in which the researcher literally becomes a participant in a
situation and makes research observations
while participating.
Observer effect, in which the presence of the
participant observer alters the nature of the
situation.
Observer bias, in which the participant observer
makes inaccurate and/or invalid interpretations
of the phenomenon being observed.
Fieldnotes, which are the actual “data” resulting
from the participant observation process.
Grounded theory, which is theory (or components of theory) developed from the data
collected in real world settings.
Peer debriefer, which is a colleague who works
with the primary researcher to generate
meaning from the data collected.
Audibility, which is the process of maintaining a
record of data management techniques that
“document the decision trail” used.
Key informant, which is a person other than the
researcher who provides information (data)
about the phenomenon being studied.
Low inference descriptors, which are concrete,
precise, and almost literal descriptions of
phenomena.
Comprehensive sampling, in which all members
of an entire group are selected as the units of
study.
Maximum variation sampling, in which objects
of observation are selected because they
represent disparate examples of the phenomenon
being studied.
Critical case sampling, in which a unit is selected
for study because it represents a dramatic
example of the phenomenon being studied.
Network (sometimes known as snowball)
sampling, in which future successive
participants are identified by prior participants.
Negative case / discrepant data, in which a unit
is not behaving within the parameters of the
emerging pattern.
Quantitative Research
Quantitative research involves deductive logic,
focused and specific questions, collection and
analysis of empirical data, and generalization
of results.
To facilitate discussion, quantitative research
methods may be divided into descriptive and
experimental research.
Descriptive research does not involve
manipulation of variables, whereas experimental
research does involve manipulation of one or more
variables.
Descriptive research is conducted to provide a
characterization of a situation or circumstance,
or a complete account of a process.
Descriptive research does not alter whatever is
being studied.
Descriptive research methods include
historical, case study, field study, survey,
developmental, causal comparative, and
correlational research.
Historical research is the systematic search for
facts relating to questions about the past and
the interpretation of those facts.
Historical research involves specific definition
of the time period addressed within the
research.
It encompasses use of primary and secondary
sources.
Primary sources are first-hand accounts or
original source documents, whereas secondary
sources involve interpretations of primary
source data or information.
Case study research is the study of a single
individual for a specified period of time.
Case study research involves assessment, but
not manipulation, of a variety of variables that
potentially contribute to the person’s current
situation.
Case study research also may be conducted
through use of a qualitative research paradigm.
Field study research is the simultaneous study
of a small number of people.
Field study research is sometimes conceived
of as a “multiple case study,” but includes
consideration of the members’ interactions.
Field study research also may be conducted
through use of a qualitative research
paradigm.
Survey research is the collection of data from
members of a population to determine the
current status of the population with respect
to one or more variables.
Survey research is often a precursor to other
types of research, most commonly correlational or
experimental research.
Survey research typically includes selfreported data.
Survey research methods often include use of
questionnaires, surveys, observations,
interviews, or sociometry.
Developmental research is the study of one or
more variables in a group of people over a
relatively long period of time (i.e., long
enough for potential developmental changes
to occur).
Developmental research is usually considered
synonymous with the term longitudinal
research.
Developmental research may be either crosssectional or longitudinal in nature.
Cross-sectional research involves studying
“cross sections” (i.e., subgroups) of a
population presumed to be at different
developmental levels to determine if developmental patterns or trends exist as predicted.
Longitudinal research involves studying the
same group of people over a relatively long
period of time to determine if developmental
patterns or trends exist as predicted.
Other types of developmental research
include:
trend studies, in which a given general
population is sampled at each data-collection
point in time.
cohort studies, in which a specifically-defined
population is followed over time.
panel studies, in which the same, presumably
representative, panel (i.e., relatively small
group) is assessed periodically.
Causal comparative research is usually
considered synonymous with the term ex post
facto research.
Causal comparative research is an attempt to
attribute causation without experimental
manipulation of a variable.
Causal comparative research is based on the
premise that both the effect and the alleged
cause exist at the time the research is
conducted.
Correlational research includes studies having
as the purpose to determine the relationships
between or among variables through the use of
correlational statistics.
Correlational research has the advantage of
allowing study of many variables simultaneously.
Correlational research is probably the most
common type of research in the counseling and
development professions.
Experimental research is conducted to
determine if differences result from the
interjection of some phenomenon into peoples’
lives.
Experimental research always involves
intentional manipulation of one or more
variables.
Experimental research also involves:
comparing conditions under various stages
of the treatment (e.g., pre-post).
systematic manipulation of experimental
conditions in which extraneous influences
are controlled or eliminated.
application of Kerlinger’s MAXMINCON
principle.
Kerlinger’s MAXMINCON principle is that
experimental research should:
maximize the independent variable’s effects
on the dependent variable (i.e., maximize
treatment variance).
minimize error factors and/or variance.
control extraneous variance.
True experimental research means that the
researcher (theoretically) has control over all
the relevant variables.
True experimental research is usually conducted
in a laboratory or other highly controlled
settings.
True experimental research is relatively rare
in the counseling and development professions due to the extensive control required.
Quasi-experimental research approximates
true experimental research except that
complete control of all relevant variables is
not possible.
Quasi-experimental research is usually
differentiated from true experimental
research by the inability to assign subjects to
groups randomly (e.g., intact groups are
used) or lack of a control group.
Single-subject experimental research involves
studying an individual in both treatment and
nontreatment conditions and evaluating
performance on the dependent variable in
both conditions.
Single-subject experimental research should
not be confused with a case study; in singlesubject research, considerable effort is given
to “controlling” extraneous variables.
Action research is an attempt to solve a
specific, immediate, and concrete problem in a
local setting.
Action research is not concerned with
generalization to any significant degree.
Action research often is used to test the
effectiveness of new skills or methods.
Action research often lacks general credibility due
to vague definitions and controls.
Experimental Research Designs
The following are commonly used abbreviations in
the depiction of experimental research designs:
R = random assignment to groups
NR = nonrandom assignment to groups
E = experimental (or treatment or tx ) group
C = control group
O = observation (i.e., measurement)
X = treatment (i.e., intervention)
Experimental research designs can be divided
into two types, depending upon the nature of
the comparison or type of effect to be
evaluated: between groups and within groups
designs.
Between Groups designs involve comparison
of the variable(s) across (i.e., between) two or
more tightly-controlled conditions (e.g.,
experimental or control).
Within Groups designs involve each subject
being exposed to each treatment condition,
but under a randomly assigned sequence of
treatment presentations (i.e., each subject is
his/her own control).
Because “within subjects” designs, such as
the crossover, counter-balanced, and Latin
Squares designs, are used relatively rarely,
the focus here will be on the “between
groups” designs.
In general, experimental research designs may
be divided into three categories based on the
degree of control over extraneous variance (i.e.,
the degree to which a difference can be
attributed solely to the treatment):
Pre-Experimental Designs
Quasi-Experimental Designs
True-Experimental Designs
Pre-Experimental Designs
One-Group Posttest-Only
Design
Assignment
NR
Group
E
Pre
Treatment
X
Post
O
Pre-Experimental Designs
Nonequivalent
Groups Posttest
Only Design
Assignment
Group
NR
NR
E
C
Pre
Treatment
X
Post
O
O
Pre-Experimental Designs
Nonequivalent Groups
Alternate Treatment
Posttest-Only Design
Assignment
Group
NR
NR
E1
E2
Pre
Treatment
X1
X2
Post
O
O
Quasi-Experimental Designs
Nonequivalent
Groups PretestPosttest Design
Assignment
Group
NR
NR
E
C
Pre
O
O
Treatment
X
Post
O
O
Quasi-Experimental Designs
Single-Group
Interrupted Time
Series Design
Assignment Group
NR
E
Pre
Treatment
O1 O2 O 3 O4
X
Post
O 5 O6 O7 O8
Quasi-Experimental Designs
Control-Group
Interrupted Time
Series Design
Assignment Group
NR
NR
E
C
Pre
Treatment
O1 O2 O3 O4 X
O1 O2 O3 O4
Post
O 5 O6 O7 O8
O5 O6 O7 O8
True-Experimental Designs
Pretest-Posttest
Control Group
Design
Assignment
R
R
Group
E
C
Pre
O
O
Treatment
X
Post
O
O
True-Experimental Designs
Posttest-Only
Control Group
Design
Assignment
R
R
Group
E
C
Pre
Treatment
X
Post
O
O
True-Experimental Designs
Solomon Four-Group
Design
Assignment
R
R
R
R
Group
E1
C1
E2
C2
Pre
O
O
Treatment
X
X
Post
O1
O2
O3
O4
True-Experimental Designs continued
Factorial experimental designs contain more
than two independent variables.
In the language of factorial designs (which
should not be confused with the statistical
process of factor analysis), a factor is an
independent variable.
Each factor in a factorial design has at least
two “levels.”
“Levels” in this context is synonymous with
“categories.”
“Levels” in this context is really a poor choice
of words, because no hierarchal order of
levels/categories in the variables is implied or
necessary.
For example, the variable “gender” in a
factorial design has two “levels” (i.e., male and
female), but clearly no hierarchy exists in these
categories.
The notation for a factorial design is a set of
numbers that identifies how many independent
variables are involved and the number of levels
in each of the variables.
For example, an experiment in which the independent variables are group (experimental or
control), gender (male or female), and secondary
school grade level (9, 10, 11, or 12) would be
denoted as a:
2 x 2 x 4 factorial design
Note: read x as “by”
The number of numbers is the number of
factors, and the numbers themselves are the
number of levels in each of the respective
factors.
In the 2 x 2 x 4 factorial design example
presented, there are three factors (independent
variables) and the first factor has two levels
(categories), the second factor has two levels,
and the third factor has four levels.
The order of the numbers in a factorial
design notation is not fixed by any rule.
The example presented could have been
denoted as a 2 x 4 x 2 factorial design, in
which the first factor was gender, the second
was grade level, and the third was group.
The (factorial experimental design) diagram
for the example presented would be:
Group
Gender
Grade
Experimental
Male
Female
Control
Male
Female
9 10 11 12 9 10 11 12 9 10 11 12 9 10 11 12
This concludes Part 1 of the
presentation on
RESEARCH AND
PROGRAM
DEVELOPMENT