Research Methods in Crime and Justice
Chapter 13
Evaluation Research Methods
Evaluation Research Methods
• Evaluation research has become a critical part
of criminal justice administration.
• The analysis of a program’s efficiency and
effectiveness is an important part of a criminal
justice practitioner’s job.
• Evaluation research considers the
effectiveness, efficiency, and the unintended
consequences of a program, process or policy.
Evaluation Research Basics
• Like the qualitative and non-reactive research
traditions, evaluation research is not a single
methodology.
• Instead, evaluation research is any applied
research method that considers the
effectiveness, efficiency, and the unintended
consequences of a program, process or policy.
Types of Evaluation Research
• There are two commonly used methods in
evaluation research.
– A cost/benefit analysis refers to a technique that
compares the costs and benefits of an existing or
future program, process or policy.
– The multiple methods approach involves the use
of multiple data gathering techniques to evaluate
a program, process or policy.
The Benefits of Evaluation Research
• Evaluation research is effective at;
– Demonstrating the effectiveness or efficiency of
an existing program, process or policy.
– Monitoring the effectiveness or efficiency of new
programs, processes or policies,
– Evaluating the unintended consequences of a
program, process or policy, and/or
– Informing the debate on controversial programs,
processes and policies.
The Limitations of Evaluation Research
• Evaluation research;
– Can evaluate but cannot determine the underlying
outcome or objective of a program, process or
policy.
– Is pointless when a program, process or policy is
mandatory or legally required.
The Evaluation Research Process
• A Case Study in Evaluation Research (The Police
Protective Custody Process)
– In most states, there is a process by which the police
can remove abused and/or neglected children from
their homes and place them in temporary housing.
– This process is expensive and in most cases results in
considerable physical and psychological disruption.
– Withrow (2001) and Withrow and Bolin (2003) were
asked to conduct an evaluation of the Police
Protective Custody process in Kansas.
The Evaluation Research Process
• Asking a Research Question in Evaluation
Research
– The research question in evaluation research is
whether the program or policy is performing as it
was intended to.
• Is it effective?
• Is it efficient?
• Are their unintended consequences or opportunities?
The Evaluation Research Process
• Asking a Research Question in Evaluation
Research – cont’d
– The PCC evaluation had the following research
objectives (questions).
• To document the PPC process.
• To determine what factors affect a child’s length of stay.
• To make recommendations for reducing the number of
PPC referrals.
• To make recommendations for reducing a child’s length
of stay (Withrow, 2001 p. i).
The Evaluation Research Process
• Conducting a Literature Review in Evaluation
Research
– The primary literature in evaluation research is
other evaluation studies on the same topic.
– Many of these studies are in the possession of
criminal justice and allied agencies.
The Evaluation Research Process
• Conducting a Literature Review in Evaluation
Research – cont’d
– There were actually two versions of the PPC
report.
• The scholarly journal included literature from scholarly
sources, most notably research on the effect of
separation during the PPC process.
• The version written for the Wichita Children’s Home
included only a nominal amount of scholarly literature
because it was intended to be consumed by
practitioners.
The Evaluation Research Process
• Refining the Research Question in Evaluation
Research
– Most evaluation research projects do not include
formal hypotheses.
– Instead they ask;
• Did the policy, process or program have its intended
effect?
• Was the policy, process or program implemented
efficiently?
• Did the policy, process or program result in unintended
consequences?
The Evaluation Research Process
• Refining the Research Question in Evaluation
Research – cont’d
– Explicit hypotheses were not developed for the
PPC study.
– Instead Withrow and Bolin made several
predictions about the relationships between the
variables used in the study.
– These predictions enabled then to at least begin
the analysis.
The Evaluation Research Process
• Defining Concepts and Creating Measures in
Evaluation Research
– In evaluation research, conceptualization and
operationalization involve developing measures
for policy or program success, efficiency and/or
effectiveness.
– Evaluation researchers must be mindful that
stakeholders may have a different view on how to
measure a program’s success.
The Evaluation Research Process
• Defining Concepts and Creating Measures in
Evaluation Research – cont’d
– Many of the concepts in the PPC study (e.g. foster
care, respite, etc.) were defined by law.
– Most of the other variables (e.g. parents drug use)
were dichotomous (i.e. yes or no).
– This is common in criminal justice research.
The Evaluation Research Process
• Defining Concepts and Creating Measures in
Evaluation Research – cont’d
– There were competing perspectives on what
constitutes the success of the PPC process.
• Police officers associated success with the ability to refer a
child to the WCH without restriction.
• Social workers defined success in terms of whether they
were able to find a suitable long term living situation for the
child.
• Prosecutors defined success in terms of their ability to
convince a judge to approve a child in need of care petition.
• Representatives from social services agencies who were
concerned about expenditures were defined success as the
ability to limit services to children who did not need them.
The Evaluation Research Process
• Defining Concepts and Creating Measures in
Evaluation Research – cont’d
– To overcome the inevitable conflict associated
with alternative views of ‘success’ the researchers
chose to focus on producing accurate descriptions
of how the system worked.
– This, they believed, would enable policy makers to
make more informed decisions
The Evaluation Research Process
• Designing a Method in Evaluation Research
– The actual method or methods used by evaluation
researchers depends on the type of research
question asked.
– The use of multiple methods has become
particularly popular in evaluation research.
The Evaluation Research Process
• Designing a Method in Evaluation Research –
cont’d
– Because the PPC had four objectives, no single
research method would have sufficed.
– The researchers used a multiple methods in our
evaluation.
•
•
•
•
focus group
Secondary analysis
Archival analysis
Interviews
The Evaluation Research Process
• Designing a Method in Evaluation Research –
cont’d
– Focus groups with representatives from the
affected agencies allowed us to;
• Document the PPC process, and
• Identify redundancies in the process.
– Secondary analysis of data on admitted children
allowed us to identify the factors that affected;
• A child’s length of stay, and
• Whether a child would be admitted to state custody.
The Evaluation Research Process
• Collecting Data in Evaluation Research
– The process by which evaluation researchers
collect data depends on the specific methods they
use.
– There are a two important considerations specific
to evaluation research.
• Gaining full access to data during an evaluation is
sometimes difficult.
• Criminal justice data is notoriously incomplete,
inaccurate, and inconsistent.
The Evaluation Research Process
• Collecting Data in Evaluation Research – cont’d
– In the PPC study the focus group required
considerable coordination.
• The practitioners involved in the PPC process are busy.
• So finding a single day when they all could meet was a rather
daunting task.
– The data set for the secondary analysis included
information on 6,600 children.
• This data set had to be ‘cleaned’ to facilitate the analysis.
• In order to avoid the possibility of violating a child’s privacy,
the data set did not contain (at the researcher’s request)
names and other personal identifiers.
The Evaluation Research Process
• Analyzing and Interpreting Data in Evaluation
Research
– The analyzes and interprets of the evaluation’s
results depends on;
• the nature of the research methods used,
• the data produced by the research methods, and
• the overall objectives of the study.
The Evaluation Research Process
• Analyzing and Interpreting Data in Evaluation
Research – cont’d
– The PPC study produced (for the first time) a flow
chart of the PPC process that enabled policy makers to
‘see’ system redundancies.
– The secondary analysis found that children were more
likely to be taken from their parents if;
•
•
•
•
they had previously received state social or welfare services,
they had parents who used illegal drugs,
they had previously been abused, and/or
they had limited parental supervision.
The Evaluation Research Process
• Analyzing and Interpreting Data in Evaluation
Research – cont’d
– The PPC study made two primary recommendations.
• The PPC process should be managed by a team of
professionals.
• Police officers should only use the PPC process for children
who;
–
–
–
–
are being or have been abused or neglected,
have received any type of social or welfare services,
Have limited parental supervision, and
Are younger than 16 years of age.
The Evaluation Research Process
• Communicating the Findings from Evaluation
Research
– Most evaluation studies are published ‘in-house’.
– Prior to the release of the report it is prudent to allow
affected individuals to review the report for accuracy.
– Often evaluation researchers are asked to present
their findings in multiple formats to different
audiences.
– On occasion the results of an evaluation report will
cause a public controversy.
The Evaluation Research Process
• Communicating the Findings from Evaluation
Research - cont’d
– The initial report (for agency consumption) was
brief to accommodate busy practitioners.
– Another version of the report was published in a
scholarly journal.
– In both situations the reports clearly identified the
limitations of the study.
Getting to the Point
• Evaluation research is designed to determine
whether a program, process or policy is
achieving its intended outcome and/or
resulting in any unintended consequences.
• Evaluation research is becoming increasingly
critical to criminal justice practice.
Getting to the Point
• Evaluation research typically involves either;
– a cost/benefit analysis, in which the costs
associated with a policy or program are weighed
against the benefits, or
– a multiple methods approach, in which various
data collection methods are used to evaluate a
policy or program.
Getting to the Point
• Evaluation research is effective at
demonstrating and monitoring the
effectiveness or efficiency of new and existing
programs, processes or policies.
• It can also identify the unintended
consequences of and provide objective
information about controversial programs,
processes and policies.
Getting to the Point
• Evaluation research is not effective;
– without agreed upon objectives, or
– when a program is mandated by law, policy or
higher authority.
Getting to the Point
• Evaluation research can be exploratory,
descriptive or explanatory depending on the
researcher’s objective.
• It is almost always applied because the
researcher usually intends for the results to
influence policies and programs.
Getting to the Point
• Although evaluation research reports can be
found in scholarly journals, the overwhelming
majority of them are in the possession of;
– various government agencies,
– privately sponsored research organizations, and
– individual criminal justice agencies.
• Such reports are typically available through the
internet or can be made available through a
direct request to the individual agency in
possession of the report.
Getting to the Point
• Evaluation research projects may begin with
general research questions or explicit
hypotheses depending on the nature of the
project.
• In either case, the primary purpose is to
investigate whether a program, process or
policy is having its intended results.
Getting to the Point
• In evaluation research, researchers typically
start by defining and operationalizing policy or
program effectiveness.
• Because criminal justice agencies are
publically funded, the researchers who
evaluate criminal justice programs should be
mindful of the various political objectives
when developing measures of program
effectiveness.
Getting to the Point
• The actual method or methods used by
evaluation researchers to collect data depends
on the type of research question(s) asked.
• The use of multiple methods has become
particularly popular in evaluation research.
Getting to the Point
• Gaining access to data during evaluation
research is sometimes difficult.
• In many cases, individuals who make the
policy or work in the program are less than
willing to give information, lest the policy or
program be found to be unsuccessful or
inefficient.
Getting to the Point
• Criminal justice data is also notoriously
incomplete, inaccurate and inconsistent.
• As a result, evaluation researchers should
expect that they will need to spend
considerable time preparing the data for
analysis.
Getting to the Point
• The interpretation process in evaluation
research is not much different than that
required in any research project.
• But evaluation researchers should bear in
mind that criminal justice data are notoriously
incomplete and inconsistent.
• This limitation should always been
acknowledged in the interpretation of the
results.
Getting to the Point
• Evaluation researchers are often asked to
present their findings in multiple formats and
to multiple audiences.
• When the media get involved with
disseminating the results of the research,
evaluation researchers should be mindful of
the complicated dynamics of public opinion
and messaging.
Research Methods in Crime and Justice
Chapter 13
Evaluation Research Methods