wk 2 evaluation overview

Johns Hopkins University School of Education
Johns Hopkins University
Evaluation Overview
Evaluation Overview
Evaluation Leaders…
Michael Scriven: Claremont Graduate University
David Fetterman: Stanford University Medical School
The identification, clarification, and application of defensible
criteria to determine an evaluation object’s value (worth or merit)
in relation to those criteria.
Examining and judging to determine value…
Evaluation Overview
Evaluation uses inquiry and judgment methods, including:
1. Determining standards for judging quality and deciding
whether those standards should be relative or absolute,
2. Collecting relevant information, and
3. Applying the standards to determine value, quality, utility,
effectiveness, or significance.
Evaluation leads to recommendations intended to optimize
the evaluation object in relation to its intended
purpose(s) or to help stakeholders determine whether
the evaluation object is worthy of adoption,
continuation, or expansion.
Research vs. Evaluation
Add knowledge to the
field: grow theory
Help stakeholders make a
judgment or decision
Who sets the agenda
Researcher defines
Stakeholders in
collaboration with
Generalizability of
Designed to be
Designed to report on a
specific context
Internal validity: causality
External validity:
Accuracy: accurate
reflection of reality
Utility: practical
Feasibility: evaluation is
realistic, prudent, frugal
Propriety: legal, ethical
Trained in depth in a single
Interdisciplinary training
Informal vs. Formal Evaluation
We evaluate everyday!
Informal evaluations occur all the time… however,
they often lack:
 breadth and depth
 systematic procedures
 formally collected evidence
Informal evaluations can be influenced by experience,
instinct, generalization, and reasoning.
Purposes of Evaluation
According to Mark, Henry, and Julnes
(1999), evaluation has 4 main purposes:
1. Assessment of merit and worth
2. Oversight and compliance
3. Program and organizational
4. Knowledge development
Formative vs. Summative Evaluation
Provides information for program
Provides information to serve decisions
or assist
Formative vs. Summative
Evaluation Activity
Internal vs. External Evaluation
 Internal (employees/members):
 More familiar with organization and program history
 Knows decision-making style of organization
 Is present to remind others of results now and in the
 Can communicate technical results more frequently
and clearly
 External (outside organization)
 Can bring greater credibility, perceived objectivity
 Typically brings more breadth and depth of technical
 Has knowledge of how other similar organizations
and programs work
 Describe a situation in which an internal
evaluator would be more appropriate than an
external evaluator. What is the rationale for
your choice?
 Now Describe a situation in which an external
evaluator might be more appropriate.
Trends in Program Evaluation
Increasing priority and legitimacy of internal evaluation
Expanded use of qualitative methods
Strong shift toward multiple and diverse methods in program
evaluation (mixed methods approach)
Increased use of program evaluation by foundations and
other agencies in the not for profit sector
Increased education and involvement of stakeholders in
evaluation to empower them, increase buy in
Advances in technology make evaluation more accessible
Performance measurement in the federal government and
not for profit organizations
Increase in standards-based assessment in education as a
means of tracking performance
Growth of evaluation internationally
Empowerment Evaluation
Critical Friend or Coach—be on their side but politely
pose questions to help refine and improve
Cycles of Reflection and Action—provide data to
inform decision making, then act on it
Culture of Evidence—develop a pattern of data
collection and documentation to support positions
Community of Learners—encourage peers to learn
Reflective Practitioner—thoughtfully consider data to
guide practical day-to-day activities
Setting Goals:
Creating a Road Map
 Mission
 Vision and Value Statements
 Project Goals, Objectives, and Indicators
of Success
 Theory of Change
 Logic Model
S.M.A.R.T Goals
See you all Next Week!