Research Methods in Crime and Justice
Chapter 9
Experimental Design Research
Methods
Experimental Design Research
Methods
• Experimental design research methods are
one of the ‘purest’ forms of social science
inquiry.
• Results provide insight into the actual causes
of social phenomena.
• A useful method in in explanatory research.
Experimental Research Design Basics
• An experiment is a research method that
measures how much, if any, an independent
variable causes a change to a dependent
variable.
• There are various types of experimental
design models.
• All experimental design models have three
essential characteristics
Experimental Research Design Basics
• All experimental design models have;
– An experimental group of subjects that are
exposed to the independent variable (i.e.
treatment) that the researcher believes will cause
change in the dependent variable.
– Treatment (the independent variable) that the
researcher alleges will cause change to the
dependent variable, and
– A posttest, during which the dependent variable is
measured after the treatment has been applied.
Experimental Research Design Basics
• More sophisticated experimental design
models may have;
– A pretest during which the dependent variable is
measured before the treatment is applied.
– A control group of research subjects that are
similar to the experimental group in every way
except that they are not exposed to the
treatment.
Experimental Research Design Basics
(Basic Steps)
• Select a sample of research subjects.
• Randomly divide the sample into two equivalent
groups.
• Measure the dependent variable in both groups
to confirm they are equivalent.
• Expose only the experimental group to the
treatment.
• Measure the dependent variable in both groups
to see of the treatment (independent variable)
had its intended effect on the dependent
variable.
Types of Experimental Research
Designs
• The most basic experimental design model is
often called a one group no pretest
experimental design model.
• This type of design only has the basic
elements of the experimental design model;
– an experimental group,
– a treatment, and
– a posttest
Types of Experimental Research
Designs
• The one group, no pretest experimental
design does not include a pretest or a control
group.
• Because of this;
– the effect of the treatment cannot be accurately
measured, and
– the influence of other factors on the dependent
variable cannot be identified.
Types of Experimental Research
Designs
• The one group pretest/posttest experimental
design model also includes the basic elements
of an experimental design model;
– an experimental group,
– a treatment, and
– a posttest.
• This model also include a pretest that
measures the dependent variable prior to
administering the treatment.
Types of Experimental Research
Designs
• Because the one group pretest/posttest
experimental design model includes a pretest
the researcher can measure the actual effect of
the treatment (independent variable) on the
dependent variable.
• This design does not include a control group so
there is really no way for the researcher to know
whether something other than the treatment
caused a change to the dependent variable.
Types of Experimental Research
Designs
• The two group no pretest experimental
design model also includes the basic elements
of the experimental design model;
– an experimental group,
– a treatment, and
– a posttest.
• This design includes a control group.
Types of Experimental Research
Designs
• Because the two group no pretest experimental
design model includes a control group the
researcher would be able to determine that the
independent variable (by itself) had some effect
on the dependent variable.
• Because there is no pretest the researcher cannot
measure how much effect the independent
variable had on the dependent variable.
Types of Experimental Research
Designs
• The two group pretest/posttest experimental
design model also includes the basic elements
of the experimental design model;
– an experimental group,
– a treatment, and
– a posttest.
• In addition this model has both a pretest and
a control group.
Types of Experimental Research
Designs
• Because the two group pretest/posttest
experimental design model contains both a
pretest and a control group, the researcher
can
– Measure the actual effect of the treatment on the
dependent variable, and
– Remove all other factors that might also have an
effect on the dependent variable.
Types of Experimental Research
Designs
• The Solomon Four Group experimental
design model includes the basic elements of
the experimental design model;
– an experimental group,
– a treatment, and
– a posttest.
• This model also contains;
– a pretest, and
– a control group.
Types of Experimental Research
Designs
• More significantly, this model contains
– an extra experimental group, and
– an extra control group.
• This results in an experimental model with
four groups of research subjects.
Types of Experimental Research
Designs
• Because the Solomon Four Group
experimental design model contains an extra
experimental and control group the
researcher is able to determine the potential
for a pretest effect.
Validity
• Previously, validity was defined as the accuracy of
a measure.
• In experimental research the term, validity, has
different meanings.
– Internal validity refers to the ability of an
experimental design to document the causal
relationship between an independent and dependent
variable.
– External validity refers to the capacity of an
experimental research finding to be applicable to
other settings.
Threats to Internal Validity
• History - Major events happen during an
experiment that affect the research subjects
and thus the dependent variable.
• Maturation - Natural developmental changes
in the research subjects affect the outcome of
the experiment.
• Mortality - A loss of research subjects can
occur over the course of an experiment and
affect the outcome of the experiment.
Threats to Internal Validity
• Testing - Exposing research subjects to a
pretest prior to the treatment can change the
outcome of the posttest.
• Instrumentation - Differences between the
pretest and posttest instruments cause a
change in the dependent variable.
Threats to Internal Validity
• Regression - Although there may be an initial
treatment effect, the effect diminishes over
time, indicating that the independent variable
has no long-term effect.
• Selection bias – When assigning members to
the experimental and control groups the
researcher erred so that so that the groups
are not equivalent.
Threats to External Validity
• Reactivity - An awareness that they are being
measured causes a change in the behavior of
research subjects.
• Interaction between selection bias and the
dependent variable - There is a failure to
ensure that the subjects assigned to the
experimental and control groups are more or
less equivalent with respect to the variables
that might influence the dependent variable.
The Benefits of Experimental Research
• Experimental research is useful for;
– Isolating the effect of an independent variable on
a dependent variable.
– Measuring how much of an effect a treatment has
on an outcome.
– Demonstrating causality, or cause and effect
relationships.
The Limitations of Experimental
Research
• Experimental research is not useful when;
– Resources (time and money) are limited.
– The researcher cannot control the behavior of the
research subjects.
– There are serious ethical concerns associated with
the control of the research subjects or the
treatment.
The Experimental Research Process
• A Case Study in Experimental Research (The
Minneapolis Domestic Violence Experiment)
– Lawrence W. Sherman and Richard A. Berk (1984)
– Applied an experimental model to determine the
most effective way to reduce domestic violence
recidivism.
– Research continues to have a profound effect on
contemporary policing practice.
The Experimental Research Process
• Asking a Research Question in Experimental
Research
– Because of their ability to isolate the effect of a
single variable, experimental designs are most
often used for explanatory research.
– “If the purpose of police responses to domestic
violence calls is to reduce the likelihood of that
violence recurring, the question is which of these
approaches (no action, active mediation, or arrest)
is more effective than the others?” (1984b: 1).
The Experimental Research Process
• Conducting a Literature Review in Experimental
Research
– A review of the prior experimental research may
reveal useful testing instruments or innovative ways of
measuring variables.
– This literature review focused on;
• classic research on police decision-making,
• research on mediation and arbitration in domestic violence
prevention, and
• more contemporary research on police responses to
domestic violence (Sherman and Berk 1994a and 1984b).
The Experimental Research Process
• Refining the Research Question in
Experimental Research
– Because experimental models are best suited to
explanatory research, it is important for the
researcher to create solid hypotheses.
– This research involved one nominal independent
variable with three attributes and two interval
dependent variables.
– The result was six sets of alternative and null
hypotheses.
The Experimental Research Process
• Conceptualization and Measurement in
Experimental Research
– Experimental researchers should be concerned
about;
• The measures used to determine equivalency between
the experimental and control groups.
• The conceptualization and measurement of the
independent variable, and
• The conceptualization and operational measurement of
the dependent variable.
The Experimental Research Process
• Conceptualization and Measurement in
Experimental Research – cont’d
– Sherman and Berk defined ‘domestic abuse’
broadly to include numerous forms of abusive
behaviors (Sherman and Berk, 1984a: 263).
– Sherman and Berk conceptualized the frequency
of reoffending as the number of repeat offenses
that occurred within a six-month period following
the first domestic violence incident.
The Experimental Research Process
• Designing Experimental Research
– The choice of experimental design depends on a
number of factors, including;
• how confident you want to be in your results,
• what is feasible and ethical in a given research setting,
and
• what you want to know about the research subjects.
– Sherman and Berk used three one group
pretest/posttest experimental designs, one for
each strategy (arrest, separate, mediate)
The Experimental Research Process
• Designing Experimental Research – cont’d
– It was critical to insure that the three groups (arrest,
separate, mediate) were equivalent with respect to all
factors (e.g. severity of injury) that could affect the
outcome.
– Because domestic violence incidents occurred over
time it was not possible assign cases at one time.
– Sherman and Berk developed a lottery system to help
police officers randomly assign cases to the three
groups (Sherman and Berk, 1984b: 3).
The Experimental Research Process
• Collecting Data in Experimental Research
– Most data collection during an experiment occurs
during the pretest and the posttest.
– Sherman and Berk spend considerable time training
the police officers that participated in the experiment.
– They pretested their design and made appropriate
changes.
– They identified a need to train data entry assistants to
be sure they adhered to the experiment’s
measurement strategy (1984b: 5).
The Experimental Research Process
• Analyzing and Interpreting Data from
Experimental Research
– Quantitative results are the most common type of
data gathered through an experimental design.
– In most cases the analysis requires comparing preand posttest results.
– Sherman and Berk used three statistical models.
– Overall they found that arrest had the highest
deterrent effect (Sherman and Berk, 1984a: 268).
The Experimental Research Process
• Analyzing and Interpreting Data from
Experimental Research – cont’d
– Because their research had potentially profound
effects on public policy, Sherman and Berk spent
considerable time discussing its weaknesses.
• Officers that did not adhere to the experiment’s
procedures for assigning cases.
• The incapacitation effect of arrests.
• Small sample sizes.
The Experimental Research Process
• Communicating the Findings from Experimental
Research
– How, when and where the results of an experiment
are reported typically depends on who is interested in
the results of the experiment.
– Because this research would be of interest to criminal
justice scholars and policing professionals it was
published in;
• A scholarly journal
• A trade publication that is distributed to policing leaders.
The Experimental Research Process
• Asking another Research Question in
Experimental Research
– Good research tends to produce as many
questions as it answers.
– One would think that with results this convincing,
the controversy over whether a mandatory arrest
policy will reduce domestic assaults would have
ended. But this is not exactly what happened.
– Additional questions continue to this day on the
external validity of these results.
Getting to the Point
• An experiment is a research method that
measures the effect of an independent variable
on a dependent variable.
• All experimental design models feature;
– an experimental group,
– a treatment, and
– a posttest.
• More sophisticated experimental design models
also include a pretest and a control group.
Getting to the Point
• The one group, no pretest experimental design
model only includes the basic elements of an
experimental design model – the experimental
group, a treatment and a posttest.
• This design does not include a pretest or a control
group.
• Because of this the effect of the treatment
cannot be accurately measured and the influence
of other factors on the dependent variable
cannot be identified.
Getting to the Point
• The one group pretest/posttest experimental design
model includes the basic elements of an experimental
design model – an experimental group, a treatment
and a posttest.
• In addition, this model includes a pretest that allows
the researcher to measure the actual effect of the
treatment (independent variable) on the dependent
variable.
• This design does not include a control group so there is
really no way for the researcher to know whether
something other than the treatment caused a change
to the dependent variable.
Getting to the Point
• The two group no pretest experimental design
model includes the basic elements of the
experimental design model – an experimental
group, a treatment and a posttest.
• This design includes a control group so the
researcher would be able to determine that the
independent variable, by itself, had some effect
on the dependent variable.
• Because there is no pretest the researcher cannot
measure how much effect the independent
variable had on the dependent variable.
Getting to the Point
• The two group pretest/posttest experimental
design model includes the basic elements of the
experimental design model – an experimental
group, a treatment and a posttest.
• In addition this model has both a pretest and a
control group.
• These two features enable the researcher to
measure the actual effect of the treatment on the
dependent variable and to determine whether or
not other factors might have caused a change in
the dependent variable.
Getting to the Point
• The Solomon Four Group experimental design model
includes the basic elements of the experimental design
model – an experimental group, a treatment and a
posttest.
• This model also contains a pretest and a control group.
• More significantly, this model contains an extra
experimental group and an extra control group.
• These additional features enable the researcher to
determine how much, if any, the research subjects’
exposure to the pretest affected their performance on
the posttest.
Getting to the Point
• Internal validity refers to the ability of an
experimental design to document the causal
relationship between an independent variable
and a dependent variable.
• There are seven common threats to the
internal validity of an experiment: history,
maturation, mortality, testing,
instrumentation, regression and regression
Getting to the Point
• External validity refers to the generalizability
of an experiment’s results to other settings
and situations.
• There are two common threats to external
validity: Reactivity and interaction between
selection bias and the dependent variable.
Getting to the Point
• Experimental research is effective at isolating
and measuring the effect of a single
independent variable on a dependent
variable.
• Experimental research is also effective at
demonstrating a causal relationship between
two variables.
Getting to the Point
• Experimental research requires considerable
resources (e.g. time and money).
• Often experimental research is not feasible
because of the amount of control the
researcher must exert over the research
subjects.
• In experiments involving human subjects
there is a potential for ethical violations.
Getting to the Point
• Because of their ability to isolate the effect of
a single variable on an outcome, experimental
designs are most often used in explanatory
research. Experiments are appropriate for
both pure and applied research purposes.
Getting to the Point
• In preparing to conduct an experiment,
researchers should review the previous
literature, paying particular attention to how
past researchers measured the dependent
variable and what independent variables have
been found to affect the dependent variable.
Getting to the Point
• Because experimental models are used in
explanatory research, creating specific
hypotheses is an essential step in the
experimental research process.
Getting to the Point
• Experimental researchers should be
concerned about conceptualization and
measurement of the independent and
dependent variables, as well as the variables
used to determine equivalency between the
experimental and control groups.
Getting to the Point
• Experimental researchers should determine
which experimental design model they want to
use early in the research process.
• The choice of experimental design depends on a
number of factors, including;
– how confident the researchers want to be in their
results,
– what is feasible and ethical in a given research setting,
and
– what researchers want to know about the research
subjects.
Getting to the Point
• The number and nature of data collection
strategies used by experimental researchers
depend on the experimental design used by
the researcher.
Getting to the Point
• Experimental designs tend to include variables
that are measured at the interval or ratio levels.
• As such, t-tests or analyses of variance are the
most common statistical techniques used to
analyze data in experimental research.
• When interpreting the findings from these and
other analyses, experimental researchers should
be up front about the possible threats to internal
and external validity within the experimental
design.
Getting to the Point
• How, when and where the results of an
experiment are reported typically depends on
who is interested in the results of the
experiment.
Getting to the Point
• Good research tends to produce as many
questions as it answers. These new questions
are opportunities to continue the research
process.
Research Methods in Crime and Justice
Chapter 9
Experimental Design Research
Methods