Program evaluation
Sebastian Uijtdehaage
bas@mednet.ucla.edu
310.794.9009
Why do you evaluate?
Five reasons of evaluating
1. FORMATIVE
informs program development
2. SUMMATIVE
demonstrate program’s success and impact
3. UNDERSTANDING
how and why did program work?
4. DISSEMINATION
sharing best practices
5. RECOGNITION
scholarship, academic credit, leadership
Stufflebeam 2003
What type of evaluation
should I conduct?
Outcome &
Impact
Did it work?
Dan Stufflebeam:
CIPP Model of Program Evaluation
CONTEXT
Needs
Assessment
PRODUCT
What should we
do?
INPUT
Program
Planning
Outcome &
Impact
How shall we do
it?
Does it work?
PROCESS
Implementation
Are we doing as
planned?
What type of evaluation
should I conduct?
Formative or Summative?
formative & summative evaluation
Formative Evaluation
Context
Input
What needs to be
done?
How should it be
done?
Process
Is it being done?
Product
Is it working?
Stufflebeam 2003
formative & summative evaluation
Formative Evaluation
Summative Evaluation
What needs to be
done?
How should it be
done?
Were important
needs addressed?
Was program plan
defensible?
Process
Is it being done?
Product
Is it working?
Was the plan well
executed?
Did the effort make
a difference?
Context
Input
Stufflebeam 2003
What type of evaluation
should I conduct?
Quantitative or Qualitative?
Quantitative or qualitative measures?
Quantitative measures
HOW MUCH…?
Qualitative measures
WHY and HOW?
• How much did the
participant change?
• How much did the
community change?
• Why did the program
work?
• How did participants
change as result?
What makes good evaluation?
Evaluation must be planned ahead
• Evaluation should be part of program planning
• Planning and resource allocation makes
evaluation more successful
– Example: long-term tracking of participants
Confidence being admitted in a
medical school
4.8
4.6
4.4
Start of program
4.2
End of program
4
3.8
3.6
Category 1
What else makes good evaluation?
• Scientific rigor
• Evaluation produces useful information that
helps make decisions
– Assessment should be done with a particular
purpose in mind
• Evaluation is tailored to your program
objectives
• Evaluation is done with conceptual framework
in mind
LOGIC MODEL
CONDITIONS and DRIVERS
Resources
Activities
The Planned Work
Output
Outcomes
The Intended Results
Impact
LOGIC MODEL
CONDITIONS and DRIVERS
Current situation that precipitates the need for
the program
LOGIC MODEL
CONDITIONS and DRIVERS
Resources
Resources
operate the program:
Activities and inputs
Outputneeded toOutcomes
Impact
personnel, facilities, materials, equipment
LOGIC MODEL
CONDITIONS and DRIVERS
Resources
Activities
Everything
does if it had Impact
access to
Output the program
Outcomes
the resources
LOGIC MODEL
CONDITIONS and DRIVERS
Resources
Activities
Output
The “deliverables”—the events,
products and services
Outcomes
Impact
Number of participants, classes,
brochures, advisors, etc
LOGIC MODEL
CONDITIONS and DRIVERS
Resources
Activities
Output
Outcomes
If the program
delivers, the
participants
will benefit
Impact in
certain ways:
Knowledge,
skills,
behavior,
attitudes
Immediate,
mid-term and
long-term
outcomes
LOGIC MODEL
CONDITIONS and DRIVERS
Resources
Activities
Output
Outcomes
Impact
If these benefits to participants are achieved,
then certain changes in organizations and
communities might be expected to occur
LOGIC MODEL
CONDITIONS and DRIVERS
Resources
Activities
The Planned Work
Output
Outcomes
The Intended Results
Impact
LOGIC MODEL
• Links processes and outcomes:
– Funders love this!
• Great for program planning
– Work backwards
• Evaluation is built in!
• Ain’t perfect
Your turn…