Review: Alternative Approaches II

Review: Alternative Approaches II
What three approaches did we last cover?
Describe one benefit of each approach
Which approach focuses on the marginalized?
What were the five cautions the authors shared
about the alternative approaches to evaluation?
Guidelines for Planning Evaluations:
Clarifying the Evaluation Request
and Responsibilities
Dr. Suzan Ayers
Western Michigan University
(courtesy of Dr. Mary Schutten)
Individuals who Affect or are Affected
by an Evaluation Study
Sponsor: authorizes the evaluation, provides
resources for its conduct
Client: requests the evaluation
Stakeholders: those who have a stake in the
program or in the evaluation’s results
Audiences: individuals, groups, agencies who
have an interest in the evaluation and receive
its results
Understanding Reasons for
Initiating Evaluation
Understanding the purpose of the evaluation is
an important first step
– Did a problem prompt the evaluation?
– Did some stakeholder demand it?
– Who has the need to know?
– What does s/he want to know? Why?
– How will s/he use the results?
It is not uncommon for the clients to be
uninformed about evaluation procedures and to
have not given deep thought about the
Frequently, the purpose is not clear until the
evaluator has carefully read the relevant
materials, observed the evaluation object, and
interviewed stakeholders
Practical Application to YOUR Plan:
Questions to Begin
1) Why is this evaluation being requested? What
questions will it answer?
2) To what use will the evaluation findings be
put? By whom? What others should receive the
3) What is to be evaluated? What does it include?
Exclude? During what time period? In what
settings? Who will participate?
4) What are the essential program activities?
How do they link with the goals and objectives?
What is the program theory?
5) How much time and money are available for
the evaluation? Who can help with it? Is any
information needed immediately?
6) What is the political climate and context
surrounding the evaluation? Will any political
factors and forces interfere in gaining
meaningful and fair information?
Informational Uses of Evaluation
Needs Assessment
– Determine whether sufficient need exists to initiate a
program and describe the target audience
– Assist in program planning by identifying potential program
Monitoring/Process Study
– Describe program implementation and whether changes
from the initial model have occurred
Outcomes Study
– Examine whether certain goals are being achieved at desired
– from the initial model have occurred
Cost Effectiveness Study
– Judge overall program value & its relative cost:value ratio
compared to competing programs
Noninformational Uses
Postponement of a decision
 Ducking responsibility [know decision already
but need to make it look good]
 Public Relations [justify the program]
 Fulfilling grant requirements
Covert, nefarious, political uses of information:
– Typically more common in federal/national
Conditions under which evaluation
studies are inappropriate
Evaluation would produce trivial information
– Low impact program, one-time effort
Evaluation results will not be used
– Regardless of outcome, political appeal/public support…
Cannot yield useful, valid information
(bad worse than none)
– Well-intentioned efforts, “mission impossible” evals
Evaluation is premature for the stage of the program
– Fitness program evaluation in first 6 weeks will not yield
meaningful information
– Premature summative evals most insidious misuse of
Motives of the evaluation are improper
– Ethical considerations, “hatchet jobs”
(propriety: eval respects rights
& dignity of data sources; help organizations address all clients’ needs)
Determining Appropriateness
Use a tool called evaluability assessment
– Clarify the intended program model or theory
– Examine the program implementation to
determine whether it matches the program
model and could achieve the program goals
– Explore different evaluation approaches to
match needs of stakeholders
– Agree on evaluation priorities and intended
uses of the study
Create working group to clarify program model
or theory, define information needs, evaluation
– Personal interviews with stakeholders
– Reviews of existing program documentation
– Site visits
Figure 10.1 (p. 186): checklist to determine
when to conduct an evaluation
Who will Evaluate?
– impartial, credible, expertise, fresh look
– participants may be more willing to reveal sensitive
information to outsiders
– more comfort presenting unpopular
information/advocating changes, etc.
– Knowledge of program, history, context, etc.
– familiarity with stakeholders
– Serve as advocates to use findings
– quick start up
– Known quantity
– Internal collect contextual information
– Internal collect data
– External directs data collection, organizes
– Internal is there to advocate and support
after external is gone
Evaluator Qualifications/Skills
Does evaluator have the ability to use
methodologies and techniques needed in the
 ….have the ability to help articulate the
appropriate focus for the study?
 ….have the management skills to carry out the
 …maintain proper ethical standards?
 …communicate results to audiences so that
they will be used?