“A Problem Well Put
Is Half Solved”:
Evaluating DecisionMaking
Andrea D. Beesley, IMPAQ International
Sheila A. Arens, McREL International
Evaluation 2014
Denver, Colorado
Why Care about Decision Making?
• Evaluators have access to guidance on
• Theories of change/logic modeling
• Evaluation methodologies
• Reporting and communicating evaluation
findings
• Evaluations usually start after program
selection or implementation
• But before there is an
evaluand, decisions
are made about
programs and
strategies to solve
problems
• Program decisions
are not (usually)
made by evaluators
Evaluations
often begin
Decision
making
Awareness of
problem to
solve
Choice of
approach to
solving problem
Implementation
of approach
Outcomes
Should evaluators care about program decisions
(beyond whether evaluation was used in decision making)?
• Not just about
program
implementation, or
evaluation use
• Evaluating decisionmaking helps to
contextualize program
success
• If there is a mismatch
between the problem
to be solved and
chosen program,
desired outcomes will
be attenuated
• Decisions about distribution of public goods
• Revolve around deep-seated, largely uncontested
social, political & cultural values
• Not often amenable to quick fixes,
predetermined answers, or technological
solutions
• Understanding decision-making
undergirding program selection may be
helpful
• Where the alignment between program and
needs is of interest
• Because this context influences program
engagement and evaluation
Insights from Decision-Making
Research
Given decision-making
is understood as
choosing between n
alternatives to a
problem, how do
people arrive at an
answer? (Etzioni, 1988)
• Logical-empirical
(rational)
• Normative-affective
• Combination
Models of Decision Making
Type
Logicalempirical
Questions
asked
What’s the
most
efficient
(effective)
means to an
end?
Normative What is the
-affective right
choice?
Type(s) of
Information
rationality
Reason
Complete
Analytical
rationality
Instrumental
rationality
Process
Practicalvalue
rationality
(phronesis;
Flyvbjerg,
2002)
Use experience to
recognize patterns: “feel
for the game” or habitus
(Bourdieu); determine
course of action they
think will work
(intuitive)
Limited; choices
draw on emotional
involvements and
value
commitments,
heuristics
(1) Identify the problem;
(2) generate alt
solutions; (3) select
solution that is most
logical that has desired
effect; (4) implement
and evaluate solution
• Most theories of
decision-making assume
decisions made alone
(Charness & Sutter,
2012)
• Decisions can also be
made by groups
(reasoning as social
practice; Resnick, et al.,
1993)
• Knowledge distributed
across actors
• Interactions determine
decisions and
solutions
• Decision-making
reasoning is affected by
• Goals & plans
• Evidence
• Past experiences &
expertise
• Cognitive biases
• Individual characteristics
(social status, age,
cognitive abilities, stress)
• Resources (human,
financial); unrecoverable
costs
• External context (politics)
Evaluating Decision Making
• Elicit, describe, and
document how
stakeholders made
decisions about
reaching goals
• Examine outcomes to
determine whether goals
were achieved
How did they make decisions about strategy selection?
What were the outcomes of those decisions?
Example: National Evaluation of
Comprehensive Centers
• 22 Centers supported by the U.S. Department of Education (15
regional, 7 content)
• 7 federal priority areas
• Support state education agencies (SEAs) in building capacity to
help district and schools
• Centers decide on
objectives and projects
to help states build
capacity and achieve
goals
• A state goal may lead
to a variety of Center
approaches
Federal
Priority
State
Goal
Center
Objective
Projects
Services
&
Products
ShortTerm
Outcomes
How did Centers make their decisions
about what strategies would help?
Mid-Term
Outcomes
LongTerm
Outcomes
Eliciting Decision Making Information
• Theory of action for planning and doing work with SEAs
• As enacted (not just as planned)
• Current and retrospective
• Ask about each step in theory of action, each with
several probes
•
•
•
•
•
•
•
SEA goals
Center objectives
Specific projects or activities
Customizing strategies
Contextual influences (politics, resources, etc.)
Links between activities and outcomes
Progress monitoring
• Ask in general as well as within the context of specific
projects
Once SEAs have determined their goals, how do you choose your Center
objectives—the overall results your Center hopes to achieve to support
the SEA’s goals?
PROBES:
• Who is involved in conversations/meetings about Center objectives?
• To what extent are the Center objectives determined by the SEA’s
requests of you, and to what extent do you suggest the Center’s
objectives to the SEA?
How do you choose which specific strategies or projects to pursue to
achieve your objectives? What is your theory about how these
strategies, activities, or projects will contribute to achieving your
objectives?
PROBES:
• Who is involved in conversations/meetings about strategies?
• What do you take into consideration when deciding upon strategies
(feasibility, evidence, availability of expertise, experience in other
projects, expectations of SEA, likelihood of building capacity, etc.)
• To what extent are your projects/strategies determined by the SEAs’
requests of you, and to what extent do you suggest the projects to
the SEAs?
How do you customize strategies to the SEA’s existing capacity
level and needs?
Can you describe how the local context—political, economic,
etc.—affects your choice of strategies/projects, if at all?
PROBES:
• How do you become aware of these contextual factors?
• How influential do you think context is in your decision making?
How does the availability of Center resources (financial, human,
etc.) enter into your choice of strategies? [If resources do affect
strategy choice: If you’re making decisions about projects based
on resources, do you involve stakeholders in the decision
process? If so, how?]
Lessons Learned
• Program staff may naturally think first about work
they’re doing, not how it was decided on
• Detailed level of questions and probes
is needed
• Interviews were opportunities for
staff to reflect on their work
• Variation in decision making processes
• Revisit links between decision making
and outcomes as work progresses
Questions
(get funding statement)