ATE Evaluation: Measuring Reaction, Learning, Behavior and Results
Behavior, and Results
October 24, 2012 | ATE PI Conference
This material is based upon work supported by the National Science Foundation under grant numbers 0802245 and 1204683. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the presenters and do not necessarily reflect the views of NSF. Overview
Evaluation fundamentals
Evaluation fundamentals
Kirkpatrick Model for evaluation
Data collection
Applying Kirkpatrick
Data quality
Data quality
Data Interpretation
Evaluation
l i
fundamentals
Guiding principles
Systematic Inquiry
Competence
Integrity/Honesty
Respect for People
f
l
Responsibilities for p
General and Public Welfare
Evaluation standards
Utility
F ibilit
Feasibility
Propriety
Accuracy
Accountability
Evaluation is systematic Evaluation
is systematic
investigation of the
investigation of the
merit, worth, and/or significance of something
advanced technological technological
education
Kirkpatrick
Kirkpatrick Model for d lf
evaluation
Kirkpatrick Model
p
0. Reach
1. Reaction
2. Learning
3. Behavior
4. Results
Level 0: Reach
Level 0: Reach
The extent to which the intended audience was reached and involved di l d
Level 0: Reach
Level 0: Reach
“How well does the proposed activity broaden the participation of underrepresented groups (e.g., gender, ethnicity, disability, geographic, etc.)?” NSF’s BROADER
IMPACTS
review i
criterion
Level 1: Reaction
The extent to The
extent to
which participants react favorably
react favorably to activities and products
Level 1: Reaction
Satisfaction
Engagement
Relevance
Level 2: Learningg
The extent to The
extent to
which participants acquire the intended intended
knowledge, skills, or attitudes
Level 2: Learningg
The extent to The
extent to
which participants acquire the intended intended
knowledge, skills, attitudes, confidence or
confidence or commitment
Level 2: Learningg
Knowledge
“II know it.
know it ”
Level 2: Learningg
Skill
“II can do it can do it
now.
now.”
Level 2: Learningg
Attitude
“II believe
believe
this is
this is worthwhile ”
worthwhile.
Level 2: Learningg
Confidence
“II think I think I
can do it.”
can do it.
Level 2: Learningg
Commitment
“II intend
intend
to do it.”
to do it.
Level 3: Behavior
The extent to The
extent to
which participants apply what they
apply what they learned Level 3: Behavior
The extent to The
extent to
which participants adopt intended
adopt intended behaviors
Level 3: Behavior
Critical
Critical behaviors
key behaviors that need to be performed to bring b
f
d bi
about desired outcomes
Level 4: Results
The extent to The
extent to
which intended outcomes occur
Level 4: Results
Targeted
Targeted outcomes
related to high‐level goals/project purpose
Level 4: Results
Targeted
Targeted outcomes
What is going to be different in advanced technological education—
or the workforce
or the workforce—
because of your grant?
Kirkpatrick Modelluation
p
0. Reach
1. Reaction
2. Learning
3. Behavior
4. Results
Kirkpatrick Model
p
0. Reach
1. Reaction
2. Learning
3. Behavior
4. Results
Kirkpatrick Model
p
0. Reach
1. Reaction
2. Learning
3. Behavior
4. Results
Di
Drivers/Enablers
/E bl
- Reinforcement
- Encouragement
- Rewarding
- Monitoring
Reinforcement
Checklists
Follow‐up modules
Online support
Online support
Refreshers
Reminders
Check it out
Encouragement
g
Coaching
Mentoring
Rewards
Recognition
Awards
Monitoringg
Logs
g
Surveys
Periodic check‐ins
Periodic check‐ins
Kirkpatrick Model for evaluation
p
0. Reach
1. Reaction
2. Learning
3. Behavior
4. Results
Kirkpatrick Model for project design
pat c
ode o p oject des g
0. Reach
1. Reaction
2. Learning
3. Behavior
4. Results
Begin with the end in mind
end in mind
Data collection
Measuring Reaction
Observation
Surveys
Interviews
Focus groups
Measuring Learning
g
g
Survey
Knowledge test
Demonstration
Simulation
Role play
l l
Measuring Behavior
Interviews
Observation
Surveys
Third‐party feedback
Measuring Results
g
Institutional data
Institutional
data
Employment data
Employer feedback
Kirkpatrick Model p
0. Reach
1. Reaction
2. Learning
3. Behavior
4. Results
Collect data to build a chain of evidence
id
Applying
Ki k t i k
Kirkpatrick
Landspeeder Technology Institute
p
gy
Data Quality
Validity & reliability
y
y
Neither valid Neither
valid
nor reliable
Reliable, Reliable
but not valid
Valid and Valid
and
reliable
Ask good questions
g
q
TTo what extent do h t t td
II buy healthy food b h lth f d In a typical 7‐day week, I t i l7 d
k
you follow the for my children.
how many days do your MyPyramid Always
children consume the Guidelines when Usually
following types of foods?
making nutritional Sometimes
___ fresh fruit
decisions for your
decisions for your Never
___ fresh vegetables
fresh vegetables
family?
___ whole grain breads or cereals
___ milk, yogurt, or cheese
ilk
t
h
___ fish
___ sugar‐sweetened beverages
Establish buy‐in early
y Follow up
y
Use existing data
Use existing data
Use registration forms to collect baseline data
baseline data
Collect data from captive audiences
Follow up p
Follow up
Data Interpretation
50
2012
50
37
2011 2012
Project starts
50
35
34
37
2009 2010 2011 2012
Project starts
50
35
34
37
2009 2010 2011 2012
Project starts
65
73
82
50
35
34
37
2009 2010 2011 2012 2013 2014 2015
Project starts
50
30
37
15
2009 2010 2011 2012 2013 2014 2015
Project starts
50
30
37
15
2009 2010 2011 2012 2013 2014 2015
Project starts
65
73
82
50
30
37
15
2009 2010 2011 2012 2013 2014 2015
Project starts
65
73
82
50
30
37
15
2009 2010 2011 2012 2013 2014 2015
Project starts
65
73
82
50
35
34
37
2009 2010 2011 2012 2013 2014 2015
Project starts
Comparisons are critical!
p
‐ over time
‐ with another group ‐ with what would with what would
have happened in the absence of the
the absence of the intervention
‐ with expectations/ ith
t ti /
targets
Discussion
You’re invited!
EvaluATE webinar on N
November 28
b 28
www.evalu‐ate.org/events