Proving Public Education Works
Evaluation for the Fire and Life Safety
Educator
•Chief Michael Cassidy, Holliston (MA) Fire
•Prof. Cynthia Ouellette, M.Ed., Rhode Island College
•Jennifer Mieth, Office of the State Fire Marshal
Introduction
• Funny accents- do we need an interpreter?
• MA Public Fire & Life Safety Education Task
Force
• Student Awareness of Fire Education Program
(S.A.F.E.)
Why Evaluate?
• To know:
1) Did students learn what you thought you taught
that day;
2) At the end of the year was there an increase in
knowledge; and
3) Have you changed behavior.
4) Justifies your program – proof that what you are
doing works.
What Do Evaluations Measure?
• Knowledge – what we want the students to
know
• Behavior – what we want the students to be
able to do
4 Levels of Evaluation
•
•
•
•
Reactions and feelings
Learning
Changes in skills
Effectiveness
Types of Evaluations
• Short term
– 4th step of a lesson plan where students demonstrate
learning and application of new knowledge.
• Medium term
– Proof that students learned what you thought you taught
at a milestone
– Measurable changes in behavior
• Long term
– Proof students remember over longer-term
– Justifies your program – proof that what you are doing
works.
– Measurable changes in behavior
Evaluation Design
Key Considerations
• What is the purpose?
– Short, medium, and long-term
• Who wants the information?
• What kind of information do you need to support your
program?
• Where will you get the data?
• How can the info be reasonably collected?
• When is the information needed?
• What resources are available to collect the info?
Outcomes Based Evaluations
• Identify major outcomes that you want to
examine or verify for the program under
evaluation.
• Choose the outcomes to examine and
prioritize.
• For each outcome, specify observable
measures that will suggest achievement-what
changes in behavior?
Outcomes Based Evaluations
• Specify target goal
– What percent of students will = success?
• Was the program delivered similarly to all
students?
• Decide how to realistically gather your data.
• Analyze and report data.
Lesson plans
• Preparation
• Presentation
• Application
• Evaluation
Bloom's Taxonomy Verbs
•Uses verbs
•Complete the sentence: The Student will be able to (verb) (action that
demonstrates learning)
Knowledge
Count, Define, Describe, Draw, Find, Identify, Label, List,
Match, Name, Quote, Recall, Recite, Sequence, Tell, Write
Comprehension
Conclude, Demonstrate, Discuss, Explain, Generalize,
Identify, Illustrate, Interpret, Paraphrase, Predict, Report,
Restate, Review, Summarize, Tell
Application
Analysis
Synthesis
Evaluation
Apply, Change, Choose, Compute, Dramatize, Interview,
Prepare, Produce, Role-play, Select, Show, Transfer, Use
Analyze, Characterize, Classify, Compare, Contrast, Debate,
Deduce, Diagram, Differentiate, Discriminate, Distinguish,
Examine, Outline, Relate, Research, Separate,
Compose, Construct, Create, Design, Develop, Integrate,
Invent, Make, Organize, Perform, Plan, Produce, Propose,
Rewrite
Appraise, Argue, Assess, Choose, Conclude, Critic, Decide,
Evaluate, Judge, Justify, Predict, Prioritize, Prove, Rank,
Rate, Select,
Powered by: The Online Teacher Resource (www.teach-nology.com)
© Teachnology, Inc. All rights reserved.
Ways to Evaluate
Home Escape Planning
Draw an escape plan on a grid
Draw a filmstrip, cartoon or poster
Demonstration
Skits
YouTube Video
PowerPoint Presentations
Essay, poem, rap, song, psa
Home Escape Plan Checklist
Shows two ways out
Identifies meeting place
Makes a clear route out
Identifies doors, windows, stairs
Rooms clearly labelled
Check Lists
• A different type of scoring guide
• Lists what the students need to have done to
do a good job
• Identifies what is important
• Easy to follow but not easy to grade
YouTube Video
• How would evaluate a video?
• Watch video
17
Rubrics
• A scoring guide that measures what student
knows or is able to do.
• On a point scale –develop a grade.
• Students know what is expected of them.
• Instructor is able to identify what is important.
• Use for essays.
• Go to www.Rubistar.com for examples.
YouTube Video Contest Rubric
• YouTube video
20
Pre and Post-test Design
If you want to:
• Document that learning took place, and
• Link learning directly to your teaching, and
• Track changes and improvements over time
Advantages of Pre & Post-Test
Design
•
•
•
•
•
Identifies areas of strengths in the lesson
Identifies areas of weaknesses in the lesson
Measures knowledge not behavior
Can use test yearly to do a longitudinal study
Apply statistical tests to document
effectiveness
Steps
1.
2.
3.
4.
5.
6.
Identify essential teaching points
Focus your test on the teaching points you think
students will not learn elsewhere
Design a test focused on those teaching points
Administer the test before you teach
Teach
Administer the same test after you teach
LNTB
Kindergarten
Pre/Post Test
LNTB 3rd Grade
Pre/Post Test
S.A.F.E. Program Evaluation
• Developing tools to measure at milestones
– 3rd grade
– 6th grade
– 10th grade
• Maintain local program flexibility
• Measure knowledge of core concepts and
behaviors in age appropriate manner
• Use Dept. of Education standards
• Use Dept. of Fire Services standards
29
What is the S.A.F.E. Evaluation:
First: Important parts of an evaluation:
• Know WHAT you want to measure: The S.A.F.E. Program
• Fire safety education so students can recognize dangers of
fire
• So, we measure the EDUCATION, i.e. what students have
learned
• Know HOW to measure it: Test, reach several intelligences
• Know that the TEST is accurate and understandable
Design
1.
2.
Define Essentials: Based on Curriculum
Planning Guide, National Standards,
Research
Review by Fire & Life Safety Education
Task Force and Elementary School
Educators
Sources
• Curriculum Planning Guidebook
• Department of Education Health and Safety
Curriculum frameworks
• Learn Not to Burn
• Massachusetts Fire Incident Reporting System
• US Fire Services, Topic Fire Research Series
• Fire and Life Safety Educators Task Force
• Professional Elementary School Educators
• MCAS Question Format: multiple choice
32
10 Core Concepts
•
•
•
•
•
•
•
•
•
•
Match & lighter safety
Stop-drop-and-roll
Home escape plans
Smoke alarms
Crawl low under smoke
Reporting fires and other emergencies
Dangers of tobacco and smoking materials
Firefighters are helpers who save lives
First aid for burns
Two ways out of public places
What Is Measured?
Concepts: General statements of learning goals, formed by
combining specific details.
Knowing and practicing stop-drop-and-roll: test questions 20 and 21
Knowledge:
• Stop-drop-and-roll is a way to put out the fire
• For when fire gets on clothes
• Important to act immediately
• Stop-drop-and-roll is very different from crawl-low-undersmoke
Behaviors: Correct sequence is critical
• Stop immediately
• Drop to ground
• Cover eyes and mouth
• Roll over and over until flames are out
34
Approach
• Stories that seem ‘real’ to third grade students
• Choices based on what third grade students
think
• Uses a booklet
• Tried to use a separate answer sheet but did
not work for 3rd graders
• Is Test Accurate and Do Kids Understand it?
Tested with Six Third-Grade Classrooms
What we learned:
• Students understand the pictures and questions in the
same way, but we needed to make some changes to
achieve that: Examples:
o Smoke alarm ‘chirp’ not ‘chirp – chirp – chirp’ (p. 9)
o Placement of fire on shirt (p. 11)
• Children understood the stories and questions
• Answers are reliable, i.e. most children answer in the
same way
36
What We Learned About the Test
• Beginning third graders struggle with the test;
third graders later in the school year can
complete the test.
• Test originally was too long: needed to set
time limit to 15 minutes
• Answer sheet too difficult for third grade
students
37
Summary so far:
To evaluate something we need to:
1. Know WHAT we want to measure
2. Have a tool that measures it
3. Third grade evaluation tests the S.A.F.E. program: curricula,
teaching, training
4. Child’s application of understanding to a situation
(knowledge and behaviors)
5. Test works with third grade students, at the end of the school
year
38
10 Core Concepts
•
•
•
•
•
•
•
•
•
•
Match & lighter safety
Stop-drop-and-roll
Home escape plans
Smoke alarms
Crawl low under smoke
Exit signs and two ways out of public places
Reporting fires and other emergencies
Dangers of tobacco and smoking materials
Firefighters are helpers who save lives
First aid for burns
The test:
•30 Questions
•10 Key Concepts
•Stories & Pictures
40
1. Match and lighter
safety
41
2. Stop – drop – and - roll
42
3. Escape Plans
43
4. Smoke Alarms
44
5. Crawl low under smoke
45
6. Exit signs and two-ways out
46
7. Reporting fires and other emergencies
47
8. Tobacco and smoking materials
are dangerous in three ways.
48
9. Firefighters are
helpers who save lives
49
10. First aid for
burns
p. 11
50
Summary:
•Know what is in the test
•Understand link between concepts and test
51
Scoring:
Correct answer = 1
Incorrect answer = 0
Missing answer = 0
52
Database
p. 9 in Guide
Excel sample.xls
DFS_SAFE 3rd Grade Test Database MASTER.xls
DFS_SAFE 3rd Grade Test Database SAMPLE.xls
53
Now what do you do??
•
•
•
•
You’ve got your information
Put it together
Look at the numbers
This is what we call…..STATISTICS
Record Keeping By Student
School:
Durham CC
1
2
3
4
5
6
7
8
9
10
Grade: 1
Teacher:
Student Name
Performs All Elements
Bobby Smith
Jane Stuart
Mary Hope
John Doe
Bill Murrary
Tad Martin
Tiger Woods
Anna Donner
Laura Spencer
Tina Louise
Subtotal
Percent of Class
x
Cindy O
Needs
Coaching
Date: 2/8/2011
Re-teach
Elements
Did Not
Demonstrate
x
x
x
x
x
x
x
x
x
9
90%
2
20%
2
20%
2
20%
Record Keeping
By School
Summary
Schools
Year 1
Key Behavior: Stop, Drop and Roll
Total
Performs all
Needs
Students Elements
% Coaching %
Reteach
Elements
Did not
%
Do
%
Abby Road
Bear Creek
Cedar Junction
Danvers State
Everett
23
24
25
22
23
19
20
18
21
19
83%
83%
72%
95%
83%
2
1
3
1
1
9%
4%
12%
5%
4%
1
1
4
0
1
4%
4%
16%
0%
4%
1
2
0
0
2
4%
8%
0%
0%
9%
Total
117
97
83%
8
7%
7
6%
5
4%
Longitudinal Studies
Studies Over Time
• Is the program successful in the long term?
• Are you getting consistent results year after
year?
• Proving your program works.
– Documenting Your Outcomes Were Met.
• Amherst example
• Child Fire Deaths
Changes in Pre-Test Scores Over Time
Changes in Pre-Test Scores
90
80
70
Score
60
67
65
74
72
70
68
82
80
83
78
Year 1
Year 2
50
40
30
20
10
0
Grade 1
Grade 2
Grade 3
Grade 4
Grade 5
Pre vs. Post Test
Pre and Post Test Scores
Summary Year 1
Pre
Post
Grade 1
65
85
Grade 2
68
87
Grade 3
70
90
Grade 4
80
95
Grade 5
78
95
Averge
72.2 90.4
Year 2
Diff Pre Post
20 67
87
19 72
92
20 74
87
15 82
97
17 83
95
18 75.6 91.6
Diff
20
20
13
15
12
16
7
7
Department
Boston
Belmont
Halifax
Blackstone
Southboro
Town W
Average
15
15
What
does it
Smoke
mean
alarm
when
goes off?
alarm
chirps?
60
49
68
70
53
53
52
97
69
79
46
37
58
64
16
16
What
should
Marcus
do?
52
66
59
96
78
37
65
17
17
18
18
30
30
What
How Where to
smoke
often
put
alarm replace smoke
does? batteries? alarms
48
61
59
81
60
43
59
51
56
45
88
22
22
47
71
51
63
82
71
34
62
Using Results
•
•
•
•
Planning teaching
Reporting
Setting Goals
PubEd Program Training & Technical
Assistance
• Funding
• Measuring Changes in Behavior
DFS_SAFE 3rd Grade Test Database SAMPLE.xls
Planning Teaching
Stop, Drop & Roll
20
21
Department
Boston
Belmont
Halifax
Blackstone
Southboro
Town W
Average
Fire on
shirt:
Correct
what
SDR
should he
do?
66
81
98
100
96
67
85
57
20
88
98
34
26
54
•What does this
say about who well
we are teaching
Stop, Drop and
Roll?
•How will this
feedback affect the
lesson plan?
Reporting Effectiveness
• 82% of students know they should tell an
adult if they see cigarettes or lighters.
• Nearly 9 out of 10 third graders know that
matches and lighters are for grown-ups and
children should not play with matches.
Setting Program Goals
• Freetown Fire Department plans to
continue the match and lighter safety
module we have been using, since
evaluation data show this has been
effective. However, we would like to
improve student’s grasp of the concept,
introduced in third grade, that children
should not use matches and lighters only
adults should from 82% to 87% of
students.
Sustaining Funding
Training and Technical Assistance: Sharing Good Ideas
Questions about smoke alarms: Percent Correct
TOWN
7: Smoke alarm
goes off
Westown
74%
67%
72%
67%
62%
59%
Eastown
72%
68%
74%
66%
60%
61%
Nortown
70%
65%
78%
62%
59%
68%
Newtown
76%
80%
77%
82%
82%
70%
AVERAGE
73%
70%
75.25%
69.25%
65.75%
64.5%
15: Alarm
chirps
16: What should
he do
17: What smoke
alarm does
18: How often
replace batteries?
30: where to put
smoke alarms
67
Measuring Outcomes
• If 82% of students know matches and lighters
are for adults only, is that success?
• Is 4 out of 5 student correctly doing SDR
success?
• Is 26% of students not knowing what to do
when the smoke alarm sounds success?
• Have you met your original outcome goal?
Proving Public Education Works
MA Child Fire Deaths by Year
30
25
20
15 14
10
5
25
24
23
S.A.F.E. Program started in Fall
1995
18 18
14
9
10
12
9
11
10
8
6
7
7
4
4
4
5
5
5
1
19
86
19
87
19
88
19
89
19
90
19
91
19
92
19
93
19
94
19
95
19
96
19
97
19
98
19
99
20
00
20
01
20
02
20
03
20
04
20
05
20
06
20
07
20
08
20
09
20
10
0
1
THANK YOU!
70