EVAL 6970: Cost
Analysis for Evaluation
Dr. Chris L. S. Coryn
Nick Saxton
Fall 2014
Agenda
• Establishing an analytic framework
– Identifying the evaluation problem
– Establishing the alternatives
– Determining the audience for the
evaluation
– Selecting the appropriate type of cost
analysis
– Deciding whether a cost analysis is
worth doing
• Activity
Note…
• Beginning next week, we will begin
studying some of the more technical
aspects of cost-analysis
Identifying the
evaluation problem
• A ‘statement of the problem’ or
‘problem identification’ is perhaps
the most critical part of any form of
inquiry
• It helps in defining the specific
questions to be investigated and
often in formulating hypotheses to
be tested (if relevant)
• It also helps in defining appropriate
methods to be used in addressing
the problem, answering the research
questions, or testing formulated
hypotheses
• The ‘statement of the problem’ should
include, at a minimum
– Background of the problem (e.g., history,
relevance, importance)
– Statement of the problem situation (e.g.,
basic difficulty, need)
– Purpose (goal-oriented) with an emphasis
on practical outcomes or products
– Questions to be answered or objectives to
be investigated
– Delineation of the research problem
(explication of relationships among
variables or comparisons to be considered)
– Statement of hypotheses (if relevant)
– Scope and delimitations (narrowing of
focus)
Establishing the
alternatives
• Determine all relevant, realistic, and
warranted alternatives (i.e., solutions
to the problem) that could be
reasonably and realistically
considered
• The selection of comparisons should
be ‘exhaustive’ to the extent possible
– Potential alternatives should not be
excluded unless absolutely necessary
Determining the
audience for the
evaluation
• Identify and enumerate all ‘primary’
and ‘secondary’ audiences
– In evaluation, these are sometimes
referred to as ‘upstream stakeholders’ (i.e.,
those making decisions) and ‘downstream
stakeholders’ (i.e., those impacted by
decisions)
• Primary audiences generally should
receive priority as they will likely be
responsible for decision making
• Even so, sensitivity to secondary
audiences is crucial (failure to consider
this audience can result in negative,
undesirable consequences)
Selecting the
appropriate type of cost
analysis
• Basic options include, but are not
limited to
– Cost-effectiveness
– Cost-benefit
– Cost-utility
– Cost-feasibility
Nature of the analytic task
• Determine which method(s) of costanalysis is(are) appropriate or
inappropriate given the problem to
be investigated
• Consider the types of questions
addressed, requirements, strengths,
and weaknesses of each costanalysis method discussed last week
Type of
analysis
Analytical
questions
Cost•
effectiveness
Measure of
costs
Which
Monetary
alternativ value of
e yields a resources
given
level of
effectiven
ess for
the lowest
cost?
Measure of
outcomes
Units of
effectivene
ss
Strengths
•
•
Weaknesses
Easy to
•
incorpor
ate
standard
evaluati
ons of
effective
ness
Useful
for
alternati •
ves with
a single
or small
number
of
objectiv
es
Difficult to
interpret
results
when there
are
multiple
measures
of
effectivene
ss
Cannot
judge
overall
worth of a
single
alternative;
useful only
for
comparing
two or
more
alternatives
Type of
analysis
Analytical
questions
Cost-benefit
•
•
Measure of
costs
Which
Monetary
alternativ value of
e yields a resources
given
level of
benefits
for the
lowest
cost?
Are the
benefits
of a single
alternativ
e larger
than its
costs?
Measure of
outcomes
Monetary
value of
benefits
Strengths
•
•
Can be
used to
judge
absolute
worth
Can
compare
results
across a
wide
variety
of
alternati
ves
Weaknesses
•
Often
difficult to
place
monetary
value on all
relevant
benefits
Type of
analysis
Analytical
questions
Costutility
•
Measure of
costs
Which
Monetary
alternati value of
ve yields resources
a given
level of
utility at
the
lowest
cost?
Measure of
outcomes
Units of
utility
Strengths
•
•
•
Incorporate
s individual
preferences
for units of
effectivenes
s
Can
incorporate
multiple
measures of
effectivenes
s into single
units of
utility
Promotes
stakeholder
participation
in decision
making
Weaknesses
•
•
Sometimes
difficult to
arrive at
consistent
and
accurate
measures
of
individual
preferences
Cannot
judge
overall
worth of a
single
alternative;
only useful
for
comparing
two or
more
alternatives
Type of
analysis
Analytical
questions
Costfeasibility
•
Can a
single
alternativ
e be
carried
out within
an
existing
budget?
Measure of
costs
Measure of
outcomes
Strengths
Monetary
value of
resources
None
•
Permits
alternati
ves that
are not
feasible
to be
immedia
tely
ruled
out,
before
evaluati
ng
outcome
s
Weaknesses
•
Cannot
judge
overall
worth
because it
does not
incorporate
outcome
measures
Receptivity of the audience
• Not all audiences will willingly accept
certain types of cost-analysis (e.g.,
expressing benefits in monetary
units in a cost-benefit analysis)
• Even so, always select the method(s)
most appropriate for addressing the
problem and questions
– This is more a matter of professional
integrity than ‘doing what the client
wants’
Time constraints
• Most, but not all, cost-analyses
require several years from inception
to completion
• Certain methods (e.g., cost-utility),
however, can often be completed in
much shorter spans of time
Available expertise
• Each method of cost-analysis (i.e.,
cost-effectiveness, cost-benefit, costutility, cost-feasibility) often requires
specialized technical skills
• Not all ‘evaluators’ possess these
skills and, therefore, it may be
necessary to engage ‘experts’ in any
given cost-analysis (e.g., an
economist)
Deciding whether a cost
analysis is worth doing
• Under certain conditions, costanalysis may not be the ‘best’
method of evaluation
• Consider Scriven’s (1974) ‘cost-free’
evaluation argument
– If alternatives would result in only
insignificant, trivial improvements then
other types of evaluation may be more
relevant and beneficial
Activities
• With your group, discuss your semester
project in terms of the content covered
in today’s lecture
• In doing so, address each of the
following to the extent possible and in
as much detail as possible
–
–
–
–
–
–
The evaluation problem
Establishment of the alternatives
The audience for the evaluation
The appropriate type of cost analysis
The need for expertise and other resources
Whether a cost analysis is worth doing
• Discuss your responses to the above
with the class