“What's measured
improves”
Peter Drucker
Yvonne Pelham, Manager, Educational
Outreach, EAD
IEEE Symposium on Pre-University Teacher
Training
Hilton Tampa Airport Westshore
19 – 20 May 2012
Topics of Discussion
Outcome, measures of success, and
impact
Metrics on workshops and in-service
presentations
Impact assessment
TISP champion opportunities
2
4/13/2015
Evaluating Outcomes, Measuring
Success and Determining Impact
To evaluate outcome, define the
vision/aspiration, and determine if the
program activities fulfill that vision
To measure success, define the
goals/objectives and measure efforts
using a common set of metrics
To determine impact, define the value of
the program and assess the effort using
key performance indicators
3
4/13/2015
Vision
Through a sustained program lead by IEEE
volunteers in every section, IEEE will be
recognized by the pre-university community
as a leading provider of professional
development for science, technology,
engineering and mathematics teachers.
4
How to Get There:
Long-Term Goals
IEEE Teacher In-Service Champions in all sections
– Empower IEEE “champions” to develop collaborations
with local pre-university education community
– Increase the level of understanding of the needs of
educators
– Identify ways that engineers can assist schools and
school systems (i.e. teacher training, curriculum
enhancements, etc.)
Technological Literacy
– Enhance the level of technological literacy of preuniversity educators and students
5
Measures of Success
Empower IEEE
“champions”
Technological
literacy of preuniversity educators
Technological
literacy of preuniversity students
Understanding of
the needs of
educators
School systems
assisted
by IEEE
6
2,203 trained attendees
(81% volunteers)
93% agreed that
program enhanced
technological literacy
94% believe that
student’s technological
literacy would increase
(anecdotal)
Sustained programs in
several sections
168 presentations
reported
Current Metrics
7
Workshop Evaluations to measure effectiveness
of the training experience
– Did the training inspire the attendees to get
involved in outreach activities with teachers
and schools?
– Did the training increase understanding and
provide resources to enable collaboration?
Teacher Questionnaires to measure usefulness of
the in-service presentation
– Will the teachers use the resources provided
in the classroom?
– Did the in-service affect level of technological
literacy?
4/13/2015
Training Workshops –
Empowering Champions
8
4/13/2015
Training Workshops –
Empowering Champions
9
4/13/2015
Effectiveness of Training Workshops
How likely were you to work with
38% prior
teachers/students/schools
68% after
Positive Responses
Brazil, 59
South Africa, 55
Malaysia, 43
India, 56
Texas, US, 40
Brazil, 33 India, 31
Peru, 29
South Africa, 28
Malaysia, 23
Texas, US, 23
Montreal, 23
Peru, 15Montreal, 13
prior to training
10
after training
More stats on TISP Workshop
Feedback
72% said the workshop definitely provided
valuable information/resources to participate
in pre-university outreach
69% said the workshop contributed to their
understanding of the impact engineers can
make through pre-university outreach
11
Teacher Feedback – Questionnaire
Cumulative Results
93% of attendees who attended a TISP Presentation
and completed a feedback form are teachers.
68% of educators teach secondary level and 25%
teach primary level.
The subject areas most educators teach are science,
technology and math.
93% felt that the program added to their technical
knowledge base.
92% stated that they will use the concepts presented
in their classroom instruction in the next 6 months.
94% agreed that the “hands-on” portions of the
presentation were helpful in understanding the
concepts discussed.
12
Impact on Technological Literacy
13
Impact on teacher
Perceived impact on
student
91% agreed that
the presentations
increased their
level of
technological
literacy.
94% agreed that
the hands-on
activities will
increase their
student’s level
technological
literacy
4/13/2015
Program Evaluation
Impact Assessment to analyze the
Teacher In-Service Program
– Ability to impact science, technology,
engineering and mathematics education
– Impact on teacher and student attitudes
about engineering and technology
– Impact on student achievement levels
14
4/13/2015
Key Research Questions
How does TISP affect teacher’s pedagogy for science,
technology and mathematics?
How do the in-service sessions affect teachers attitude
towards teaching technology subjects in their classroom?
How do IEEE hands-on activities affect what students
know about engineering, technology, and related science
and mathematic topics?
How do the activities affect student’s attitudes towards
science, technology, engineering, mathematics activities,
and careers?
How do the activities impact understanding and attitudes
of female students and under-represented group in
engineering and technology fields?
15
4/13/2015
2009 Pilot Teacher Assessment:
Survey Questions
Number of IEEE lesson plans or activities that have
been used, in full or in part, in the classroom
– Reasons(s) that influenced your inability to use
the lesson plans
The lesson plan(s) supports classroom instruction
for the subject areas that I teach
The students expressed greater interest in science,
technology, engineering or math after completing
these hands-on activities (anecdotal)
16
4/13/2015
2009 Pilot Teacher Assessment:
Results
Of the respondents,
– 48% taught in the secondary school
– 57% were science teachers
– 48% used 2 to 5 lesson plans in the classroom
– Of the 26% that did not use the lesson plans, 6
out of 11 stated that they ran out of time or
was unable to fit into schedule
– 74% agreed that the lesson plans supported
their classroom instruction
– 88% agreed that the students expressed more
interest after completing the activities
17
4/13/2015
Assessment Effort
Engage a research firm to examine TISP
impact on teacher attitudes, classroom
practice and knowledge
Identify section(s) to participate in the
study
18
4/13/2015
Opportunities to contribute to
the enhancement of TISP
Make recommendations on
enhancements to the train-the-trainer
workshops
Remind teams to report TISP activities
and distribute feedback questionnaires
to update global statistics
Make recommendations on follow-on
activities with teachers
Participate in assessment research
19
4/13/2015
Let’s have
dialogue
20
4/13/2015