Extension
Program Evaluation
Dale A. Moore
VMTRC
School of Veterinary Medicine
UC Davis
What’s the purpose of evaluation?






Improve the program
Improve teaching
Make a difference
Scholarly pursuit
Get the higher-ups off your back
Meet promotion requirements
Goals of This Presentation

Discuss “traditional” program evaluation





Help you focus your evaluations
Learn the different levels of evaluation
Help you construct evaluation questions
Provide you a tool for data analysis
Discuss educational program evaluation
using Learning Stage Theory
Pre-Test



Take out the Pre-Test
Take 5 minutes to
answer the questions
Turn them in to Pat
Who will use the results?





The public
Your extension division
Colleagues
You and program planning staff …
All your stakeholders
Who will use the results?
Your Audience
Do I have the resources to conduct this
evaluation project?





Time
Money
People
Expertise
Or – are the data already available?
Levels of Evaluation




Perceptions of the program or course
Competence with new skills, knowledge,
abilities or new attitudes
Individual performance – a change in
behavior within their environment
Industry outcomes
What are your outcomes?
Mechanisms for Evaluation








Direct observation
Audience response systems
Surveys
Telephone or face-to-face interviews
Focus groups
Expert or peer-review (formative evaluations)
Testimonials
Industry data
Audience Response System
Surveys





Who is the audience?
How are you going to deliver it?
Are you going to randomly choose possible
responders?
What percent response do you need to make it
meaningful?
Pre- and Post-program delivery to address
immediate changes in knowledge/attitudes?
Pre-Test --–-- Post-Test
Pre-Test --–-- Post-Test





Advantages:
Get attention at beginning of the program
Provide some level of anxiety so that they
“look” for the answers during the presentation
Gives instructors some measure of learning
needs / areas upon which to focus
Identify short-term changes in knowledge /
attitudes
Pre-Test – Post-Test






Disadvantages:
Takes time away from the program
Need to be able to quickly “grade” and
analyze your data
Need to be able to make adjustments to the
program if needed
Only addresses short-term learning and not
behavior change
Adults may not enjoy being graded
What kind of questions?




Reaction
Learning
Behavior
Results
?
What kind of questions/data if
the outcome is behavior change?
Questions about…






Satisfaction with different aspects of the
program
Speaker effectiveness
New knowledge or skills
A change in attitude
A “commitment to change”
New program ideas
Question Types


Open- ended questions
Closed





Set of responses
Likert scale
Numeric responses
Categorical responses
What are the advantages and disadvantages of
each kind of question?
Data Analysis






How do you go about analyzing responses to
different kinds of questions?
Comparison of means – Student’s T-test
Comparison of proportions
Categorical data analysis – Chi-Square
Non-parametric like the Sign test
Multivariate analyses
* EpiCalc2000 – free program from Web; URL in handout
Documenting the Impacts of
Continuing Education
Dale Moore and Hank Slotnick
Do you sometimes feel
that…
despite…
•
•


Results of your needs assessments, and
Your best intentions to create an appealing
program
Those who need to attend the program, don’t?
The minds of some who come aren’t open?
Objectives
Provide the theoretical framework for
documenting change
 Demonstrate applications of the theory
 Demonstrate a new method of
representing change data

Background on the Problem

Key meta-analytic findings of CME programs:
 Findings consistent with adult learning:
 Simple provision of information is often
insufficient.
 Active learning appears to be more effective.
 Multiple exposures predispose toward success.
 Most traditional CME does not change physician
behavior
Why not?
Behavioral Change Theories
Prochaska & Mezirow & Slotnick



Suggest that change occurs in stages
In each stage the learner has different needs
To become interested in the idea of change
•
•
the learner’s mind must be opened
the learner must feel a need to attend to
something, something has happened to
unsettle them, or they have an immediate
problem to solve
STAGES OF CHANGE
INTERESTED, BUT….
(contemplation)
UNAWARE OR
UNIMPRESSED
(precontemplation)
CHOICES:DECISIONS
(preparation)
ACTION
FINE TUNING &
MAINTAINING CHANGE
(maintenance)
Adapted from: Prochaska et al, 1992
Mezirow: Theory of
Perspective Transformation
1)
Disorienting dilemma
2)
Self-examination with feelings dissonance or discomfort
3)
Critical re-assessment of assumptions
4)
Recognition of discontent and that others have changed
5)
Exploration of options for new ways of seeing/behaving
6)
Planning a course of action, gathering resources
7)
Acquisition of knowledge and skills
8)
Provisional trying of new perspective/behavior
9)
Building of competence and self-confidence in new activity
10)
Integration of new perspective/behavior into one’s life
Traditional CE Efforts
Potential for mismatch between the intervention
and learner readiness

Over-emphasis on encouraging action
 Ask

them to take the leap because we say so!
Presumes learners minds are open
Documenting the Outcomes

A taxonomy of outcomes



Awareness of problem
Knowledge/skill changes and Practice changes
Improved herd outcomes
Aware
Improve
Slotnick 2001
Change
Taxonomy of Outcomes

Taxonomy based on stages in
learning episodes
 Evaluation: Learner decides
whether to learn the solution
to the problem at hand
 Learning: Learner gains
required skills and knowledge
 Gaining experience: Learner
familiarized with the skills,
knowledge & problems.
Aware
Evaluation
Improve
Learn
Change
Gain Experience
Documenting Change

Hypothesis: It should be possible to
document changes in prevalence of
learners at different stages in addressing
an educational program.
How to Document Outcomes



Present clinical/practical vignettes.
Ask respondents to indicate their ‘learning status’ for each
vignette/problem.
Vignette example: Mary was a 58-year-old woman who had
been in good health generally. She did not exercise regularly
and was about 20 pounds over her ideal weight, but her diet
was relatively low in fat, she did not smoke, and she had no
history of diabetes. She had been taking estrogen and
progesterone hormone replacement therapy for 6 years. Her
recent blood lipid panel results were as follows: (blah blah
blah). She was taking hydrochlorothiazide to treat
hypertension, and her mother had experienced an MI at 60
years of age. Mary was seen at the emergency department for a
prolonged episode of left shoulder and upper back pain which
had begun after dinner that evening.
Responses



Evaluation: I’m comfortable that I can handle the
problem in the vignette; all I might be interested in is
information so I can decide when next to update.”
Learning: I need to update myself in the skills and
knowledge needed to address this problem.
Gaining experience: I’ve recently updated skills and
knowledge. All I might be interested in is hearing
about others’ experiences with similar problems.
How to Document Outcomes

The AMEE (Slotnick) study



Goal: Introduce the idea of estimating prevalence
for people at different stages of learning in a
lecture setting (90 minutes)
Measurement: Practice vignettes were examined
pre- and post, some based on issues addressed in
the lecture and some not.
Analysis: Expected to see stage-to-stage
movement for the first set, but not the second.
The Triangular Graph
Each side represents
a response – 3 possible
responses.
% Response
50
40
30
20
10
0
Evaluation
Eval
Learning
Gain Exp
Response
Eval Dominated
50%
50%
Learning Domin.
Learning
Gain Exp.
50%
Gaining
Experience
% Response
50
40
30
20
10
0
Eval
Learning
Gain Exp
Evaluation
Response
50%
Learning
50%
50%
Gaining
Experience
% Response
50
40
30
20
10
0
Eval
Learning
Gain Exp
Evaluation
Response
50%
Learning
50%
50%
Gaining
Experience
How to Document Outcomes

The AMEE study.

3 Vignettes where no changes were expected.
Evaluation
“The task force chair needs a list
of physicians who might be called
on in an emergency. She asks
you to locate such a list.”
Learning
Gaining
Experience
How to Document Outcomes

The AMEE study.

4 Vignettes where changes were expected.
Evaluation
Learning
“You and two others are to put
together a pencil-and-paper survey
questionnaire to determine what
doctors know about preventing,
diagnosing, and managing anthrax
and small pox.”
Gaining
Experience
Dairy Veterinarian Study
Dairy Informatics Vignettes
Developed 8 practice vignettes: An example -
“A 500-cow dairy client is going to expand
his operation to add 500 more cows. To help
support his application for a loan, the bank
has asked him for information on the
productivity of the dairy herd, including milk
production and herd replacements. The bank
would like information for the last 3 years of
the dairy’s operation.”
Moore’s Study
Evaluation
Learning
Gaining Experience
Moore’s Study
Evaluation
Learn
Gain Experience
Conclusions
Evaluate group changes dependent on
stage of learning
 Documents the changes
 Provides for parsimonious graphical
display of change
 Currently working on statistical methods

Post-Test



Take out the PostTest
Take 5 minutes to
complete
Turn in to Pat
What did we learn?




There are different levels of program
evaluation
Clearly-defined program objectives lead to
easier evaluations
There are a number of mechanisms of
program evaluation
Learning stage theory helps us evaluate
WHERE the learner is with regard to making
a change
Thank
you for
your
attention
References




Dr. Jack Mezirow: Transformative Dimensions of Adult
Learning; Fostering Critical Reflection in Adulthood: A
Guide to Transformative and Emancipatory Learning.
Prochaska, J. O. and C. C. DiClemente (1982).
"Transtheoretical therapy: Toward a more integrative
model of change." Psychotherapy: Theory, Research and
Practice 19(3): 276-288.
Slotnick HB. How Doctors Learn: Physicians' Selfdirected Learning Episodes. Acad Med 1999;74:1106-17.
Moore DA. Evaluating the stages of veterinary
practitioner learning for continuing education needs
assessment and program evaluation. J Vet Med Education
2003;30(1):50-56.