ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
10/24/2012
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
October 24, 2012
This material is based upon work supported by the National Science Foundation under grant numbers 0802245 and 1204683. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the presenters and do not necessarily reflect the views of NSF. Overview
Program Evaluation Standards
Kirkpatrick Model for evaluation
Data collection
Applying Kirkpatrick
Data quality
Data Interpretation
©2012
www.evalu‐ate.org
1
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
10/24/2012
Evaluation standards
Utility
Feasibility
Propriety
Accuracy
Accountability
Kirkpatrick Model for evaluation
©2012
www.evalu‐ate.org
2
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
10/24/2012
Kirkpatrick Model
0. Reach
1. Reaction
2. Learning
3. Behavior
4. Results
Level 0: Reach
The extent to which the intended audience was reached and involved ©2012
www.evalu‐ate.org
3
10/24/2012
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
Level 0: Reach
“How well does the proposed activity broaden the participation of underrepresented groups (e.g., gender, ethnicity, disability, geographic, etc.)?” NSF’s BROADER
IMPACTS
review criterion
Level 1: Reaction
The extent to which participants react favorably to activities and products
©2012
www.evalu‐ate.org
4
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
10/24/2012
Level 1: Reaction
Satisfaction
Engagement
Relevance
Level 2: Learning
The extent to which participants acquire the intended knowledge, skills, attitudes
©2012
www.evalu‐ate.org
5
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
10/24/2012
Level 2: Learning
The extent to which participants acquire the intended knowledge, skills, attitudes, confidence, or commitment
Level 2: Learning
Knowledge
“I know it.”
©2012
www.evalu‐ate.org
6
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
10/24/2012
Level 2: Learning
Skill
“I can do it now.”
Level 2: Learning
Attitude
“I believe
this is worthwhile.”
©2012
www.evalu‐ate.org
7
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
10/24/2012
Level 2: Learning
Confidence
“I think I can do it.”
Level 2: Learning
Commitment
“I intend
to do it.”
©2012
www.evalu‐ate.org
8
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
10/24/2012
Level 3: Behavior
The extent to which participants apply what they learned Level 3: Behavior
The extent to which participants adopt intended behaviors
©2012
www.evalu‐ate.org
9
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
10/24/2012
Level 3: Behavior
Critical behaviors
key behaviors that need to be performed to bring about desired outcomes
Level 4: Results
The extent to which intended outcomes occur
©2012
www.evalu‐ate.org
10
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
10/24/2012
Level 4: Results
Targeted outcomes
related to high‐level goals/project purpose
Level 4: Results
Targeted outcomes
What is going to be different in advanced technological education—
or the workforce—
because of your grant?
©2012
www.evalu‐ate.org
11
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
10/24/2012
Kirkpatrick Model
0. Reach
1. Reaction
2. Learning
3. Behavior
4. Results
Kirkpatrick Model
0. Reach
1. Reaction
2. Learning
3. Behavior
4. Results
©2012
www.evalu‐ate.org
12
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
10/24/2012
Kirkpatrick Model
0. Reach
1. Reaction
2. Learning
3. Behavior
4. Results
Drivers/Enablers
- Reinforcement
- Encouragement
- Rewarding
- Monitoring
Reinforcement
Checklists
Follow‐up modules
Online support
Refreshers
Reminders
©2012
www.evalu‐ate.org
13
10/24/2012
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
Check it out
Encouragement
Coaching
Mentoring
©2012
www.evalu‐ate.org
14
10/24/2012
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
Rewards
Recognition
Awards
Monitoring
Logs
Surveys
Periodic check‐ins
©2012
www.evalu‐ate.org
15
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
10/24/2012
Kirkpatrick Model for evaluation
0. Reach
1. Reaction
2. Learning
3. Behavior
4. Results
Kirkpatrick Model for project design
0. Reach
1. Reaction
2. Learning
3. Behavior
4. Results
©2012
Begin with the end in mind
www.evalu‐ate.org
16
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
10/24/2012
Data collection
Measuring Reaction
Observation
Surveys
Interviews
Focus groups
©2012
www.evalu‐ate.org
17
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
10/24/2012
Measuring Learning
Survey
Knowledge test
Demonstration
Simulation
Role play
Measuring Behavior
Interviews
Observation
Surveys
Third‐party feedback
©2012
www.evalu‐ate.org
18
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
10/24/2012
Measuring Results
Institutional data
Employment data
Employer feedback
Kirkpatrick Model 0. Reach
1. Reaction
2. Learning
Collect data to build a chain of evidence
3. Behavior
4. Results
©2012
www.evalu‐ate.org
19
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
10/24/2012
Applying
Kirkpatrick
Landspeeder Technology Institute
©2012
www.evalu‐ate.org
20
10/24/2012
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
Data Quality
Validity & reliability
Neither valid nor reliable
©2012
Reliable, but not valid
www.evalu‐ate.org
Valid and reliable
21
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
10/24/2012
Ask good questions
To what extent do I buy healthy food In a typical 7‐day week, you follow the for my children.
how many days do your MyPyramid Always
children consume the Guidelines when Usually
following types of foods?
making nutritional Sometimes
___ fresh fruit
decisions for your Never
___ fresh vegetables
family?
___ whole grain breads or cereals
___ milk, yogurt, or cheese
___ fish
___ sugar‐sweetened beverages
Follow up
Establish buy‐in early
©2012
www.evalu‐ate.org
22
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
10/24/2012
Use existing data
Use registration forms to collect baseline data
©2012
www.evalu‐ate.org
23
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
10/24/2012
Collect data from captive audiences
Follow up
Follow up
©2012
www.evalu‐ate.org
24
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
10/24/2012
Data Interpretation
50
2012
©2012
www.evalu‐ate.org
25
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
10/24/2012
50
37
2011 2012
Project starts
50
35
34
37
2009 2010 2011 2012
Project starts
©2012
www.evalu‐ate.org
26
10/24/2012
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
50
35
34
37
2009 2010 2011 2012
Project starts
65
73
82
50
35
34
37
2009 2010 2011 2012 2013 2014 2015
Project starts
©2012
www.evalu‐ate.org
27
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
10/24/2012
50
30
37
15
2009 2010 2011 2012 2013 2014 2015
Project starts
50
30
37
15
2009 2010 2011 2012 2013 2014 2015
Project starts
©2012
www.evalu‐ate.org
28
10/24/2012
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
65
73
82
50
30
37
15
2009 2010 2011 2012 2013 2014 2015
Project starts
65
73
82
50
30
37
15
2009 2010 2011 2012 2013 2014 2015
Project starts
©2012
www.evalu‐ate.org
29
10/24/2012
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
65
73
82
50
35
34
37
2009 2010 2011 2012 2013 2014 2015
Project starts
Comparisons are critical!
‐ over time
‐ with another group ‐ with what would have happened in the absence of the intervention
‐ with expectations/ targets
©2012
www.evalu‐ate.org
30
10/24/2012
ATE Evaluation: Measuring Reaction, Learning, Behavior, and Results
Discussion
You’re invited!
EvaluATE webinar on November 28
www.evalu‐ate.org/events
©2012
www.evalu‐ate.org
31