STEM Programs: Evaluation Evidence
and a Design Framework
Jeni Corn, Malinda Faber
The Friday Institute, North Carolina State University
Collaborative Conference
Greensboro, NC- March & April 2015
http://fti.neep.wisc.edu/neep602/LEC24/IMAGES/GlobalPopulation.GIF
Demand for Workers with STEM Competencies
Source: Georgetown University Center on Education and the Workforce forecast of
occupational growth, 2018.
• Results from Golden LEAF STEM Initiative
• Strategies and Outcomes Activity
• Asking Evaluation Questions
• Collecting and Analyzing Data
•Q&A
The Golden LEAF STEM Initiative
Source: Georgetown University Center on Education and the Workforce forecast of
occupational growth, 2018.
Characteristics of STEM Programs – In Classrooms
Inquiry-based
Hands-on
Teachers
collaborate
Integrated
content
Teachers &
STEM industry
Students & STEM
industry
administrative
data
teacher focus
groups
grant
coordinator
interviews
Instruments
student
attitudes
surveys
STEM
Schools/Programs
classroom
observations
teacher
efficacy &
beliefs surveys
STEM program
implementation
rubric
principal
leadership for
STEM survey
Changes in Teachers?
http://www.edweek.org/ew/articles/
2011/05/18/31stem.h30.html
Teachers Increased Inquiry-Based Instruction
“I’m just more open. For example, this year I’ve
allowed the kids to create their own
experiment. I would have never done that
before my grant’s training. So I’ve become more
open in those inquiry-based values.” [teacher]
Teachers Awareness of STEM Careers Increased
I know...
Where to direct students or parents to
find information about STEM
careers.
Where to find resources for teaching
students about STEM careers.
Where to go to learn more about
STEM careers.
About current STEM careers.
0%
20%
40%
60%
80%
Y3 High
Y2 High
Y3 Middle
Y2 Middle
Y3 Elementary
Y2 Elementary
“For me the visit was a very important, because
we actually talk about bio-engineering in 8th
grade science. The tour allowed me to see why
it’s important to learn about bioengineering,
what’s out there in the students’ future, the
different kinds of jobs that are available for
bioengineering, and why the whole idea of STEM
is very, very important.” [teacher]
Resources for Hands-On STEM Education are
Very Limited
“We’re trying to teach our students in a 21st
century way with 19th century technology … It’s
not engineering if you can’t apply it.” [teacher]
“I average spending about $200 a month in
materials out of my pocket for opportunities for the
kids to do things better than what we have here … If
you don’t have the materials, it’s all theoretical
again.” [teacher]
More Teacher Survey Results
Teachers confident in their instructional
knowledge
Teachers divided on whether classroom efforts
of teachers impact student learning
Teachers have positive attitudes toward 21st
century learning
Changes in Students?
http://news.vanderbilt.edu/2012/03/art2stemlab-visit/
Student Attitudes toward STEM Improved Slightly
4
3.8
Female Math
Female Science
3.6
Female Engineering and
Technology
Male Math
3.4
Male Science
Male Engineering and Technology
3.2
3
Y1
Y2
Y3
Student Interest in STEM Careers Increased
Career Area
Proportion
“Interested/Very Interested”
Y1
Y3
n
9,412
10,972
Veterinary Work
51%
Engineering
50%
53%
Biology & Zoology
48%
Medicine
50%
Computer Science
37%
Medical Sciences
42%
Chemistry
38%
Earth Science
35%
Mathematics
37%
Environmental Work
38%
Energy
33%
Physics
30%
53%
50%
49%
45%
42%
43%
41%
41%
40%
39%
33%
Student Career Interest Varies by Gender
Student Career Interest Varies by Gender
Students Perceive STEM Careers in Clusters
“CORE STEM”
CAREERS
Physics
Mathematics
Chemistry
Engineering
Earth Science
Computer Sci.
Environmental
Energy
“BIOLOGICAL”
CAREERS
Biology & zoology
Veterinary work
“MEDICAL”
CAREERS
Medicine
Medical Science
Student Engagement Increased
“I can tell a huge difference when we’re doing
an inquiry-based [lesson] … It’s a totally
different class. They’re all discussing. They’re
all working together. They’re trying to solve
the problems … You don’t see the type of offtask behaviors in class, because they are
engaged.” [teacher]
Including for Student who Struggle in School
“It doesn’t discriminate against reading ability.
We have some of the lowest readers in Project
Lead the Way and it’s amazing what they can
do when they get their hands on stuff. Their
reading ability hinders them in math and
science, but it doesn’t hinder them in that
class. It's great to see a kid get to shine.”
[teacher]
Industry-Related Activities were Beneficial
“After the community college tour, I had about
10 kids say to me, ‘Yeah, I really want to do
this.’ I had three kids who said, ‘I know what I
want to be now.’ “[teacher]
“I want more professionals to come in. I’d also
like to go out and experience it, maybe watch it
happening, because how a how a book explains
it may not be what it’s really like.” [student]
More Impacts on Students
Problem-solving skills increased
Collaboration skills increased; students
enjoyed group-work
Students want more advanced STEM courses
School Leaders’ Reflections on Grant
Successes
• Industry connections
• Emergent STEM
cultures
• Catalysts for new
STEM work
Challenges
• Grant management
• Staff turnover
• Lack of resources for
STEM education
School Leaders Reflections – Lessons Learned
Provide effective professional development
Build buy-in early
Emphasize integration
Build community partnerships
Involve the right people
Plan for the long-term
What Can STEM Industries Do?
http://www.e3bradford.co.uk/wp-content/uploads/2014/01/appren1.jpg
Education to Career Continuum for All
Awareness
Connections with
professionals of all types
Exploration
Projects in various economic Variety of careers
themes and real-world
Comparison of careerchallenges
specific information: skills
Visits to variety of worksites, needed, salary,
certification/education
employers
requirements, expected
growth in jobs, location of
jobs
Experiences: workplace
tour, guest speakers, career
fairs, visit adult
parents/friends at work
27
Skills Development
Deeper learning through
Career Training
research and experiences in
targeted career areas
Formal studies through K12, 2-year, 4-year higher
education
Skills in interviewing,
presentation, resume
Experiences: Relationship
with a mentor, project
development with industry
partner, field study, job
shadow
Mastery of career-specific
skills
Completion of certification
of career-specific
requirements
Internship/apprenticeship
Clinical experience
On-the-job training
For Teachers
Externships
Industry
Immersion
Day
Workshop
Co-Teaching
http://ncnewschools.org/community-business-leaders/what-we-do/partnerships/
For Students
Job Shadowing
Field Studies
Classroom
Coaching
Internships
http://ncnewschools.org/community-business-leaders/what-we-do/partnerships/
For the School
Schools Business
Advisory Council
School-Industry
Liaison
http://ncnewschools.org/community-business-leaders/what-we-do/partnerships/
Activity
Write answers to questions 1-3 on post-its. You will likely
have multiple answers - write one idea per post-it (paper or
virtual-http://popplet.com/ ).
1. Ultimately, what good is your program going to do (for
students, teachers, etc.)?
2. What are the major activities of your program? (How are
you going to spend your time, money, and other
resources?)
3. If you are successful implementing these activities, what
will your program accomplish this year?
Arrange your post-its on a piece(s) of paper from left to right,
from major activities (2), to immediate results (3), to the
ultimate good the program is going to do (1).
2  3 1.
Logic Models
What is a logic model?
A logic model is a
graphic representation
of the relationships
among the key
elements of a project:
• inputs,
• strategies,
• objectives,
• long-term goals.
A logic model …
• Helps to articulate the key
elements of the project.
• Enables evaluation efficiency
and effectiveness.
• Promotes stakeholder buy-in
by helping clarify how the
project works.
• Provides a great opportunity
to involve stakeholders in
planning.
www.ScienceCartoonsPlus.com
Inputs
What is
invested in the
process? Time,
money, human
resources,
partners,
equipment,
etc.
Activities
What are the
major
activities the
program will
entail? What
will be done
with the time,
money, etc.?
Outputs
How many
participants
are involved?
How many
teachers
and/or
students are
reached?
What is
produced?
Short-term
Outcomes
Mediumterm
Outcomes
Within life of
grant:
Within 1-3
Years:
What changes
in knowledge,
skills,
attitudes,
motivations,
or awareness
resulted?
What changes
in behaviors,
practices,
policies, or
procedures
resulted?
Long-term
Outcomes
After 3 or
more Years:
What are the
ultimate goals
and did
changes
results?
Student
achievement,
teacher
retention,
school culture,
etc.
Terms aren’t consistently
used … meaning is what
matters.
Whenever possible, it is helpful to try to use
the “SMART Goals” strategy for creating your target
outcomes:
Specific
Measurable
Attainable
Results-oriented
Time-bound
E.g.: Increase number of students taking advanced science
courses by 25% by fall 2015.
STEM Connect Logic Model
Strategies
Short Term Goals
Long Term Goals
Regional career work-based
experiences supported by
business community
Increase enrollment in Algebra 1
for middle school students and
upper level math and science
Raise STEM interest and
awareness
Partnerships to provide
enrichment activities for
students and differentiated PD
for teachers
Curriculum alignment for vertical
and cross curriculum through
PLCs
Increased access to content
specific technology for science
and math
Increase percentage proficiency
in Math and Science
Increase use of inquiry-based
learning in lessons
Increase student engagement
and achievement in the STEM
courses
Promote STEM educator
effectiveness through
professional development
Increase student interest and
teacher awareness in STEM
careers
Development of Leadership for
supporting STEM instruction
PD in Inquiry-Based learning
Early diagnosis and intervention
for college readiness assessment
40
Evaluation Questions
Logic
Model
Act
Analyze
Eval ?s
Collect
Data
The process for identifying evaluation
questions is critical.
Results
Data Analysis
Data Collection
EVALUATION
QUESTIONS
Two Main Types of Evaluation Questions
Questions
about
STRATEGIES
Questions
about
OBJECTIVES
Ask about how well the
strategies were
implemented.
Ask about impacts.
Quick Tips
 Try to avoid simple “yes or no” questions
 Consider QUANTITY questions, e.g:
“How many”
“How much”
“How often”
 Consider QUALITY questions, e.g.:
“How well”
“How effectively”
“In what ways”
 Be able to be tuned-in to unexpected results.
Activity
Look at your logic model and write down
one question you would like to answer
about your STEM education program.
Collecting Data
A basic chart is a great way to
organize the work of data collection
and to share easily the plan with
others.
Evaluation
Questions
Data
Sources
What information/DATA are you going to gather to
answer each question?
•
•
What data do you already have that you could use (e.g. grant-level
data from the Friday Institute)
What data do you need?
• Consider a variety of types of data and methods for collection.
• Consider the time and money required to collect the data.
• Choose carefully – what will be meaningful?
Quantitative Data
Quantitative data are
measurements that
are recorded on a
naturally occurring
numerical scale.
Look for absolute
numbers,
percentages,
averages,
differences
between groups.
•
•
•
•
•
•
•
•
•
•
Counts of participants
Counts of activities
Demographics
Grade-level information
Years of teaching experience
Standardized assessment scores
Scaled questions on surveys
Scale-scored classroom
observations
Graduation rates
Rates of course-taking or course
completion
Qualitative Data
Qualitative data are
measurements that
are not on a numerical
scale, instead they are
classified by groups or
themes.
Look for patterns
or themes in
qualitative data.
•
•
•
•
•
•
•
Interviews and focus groups with
teachers
Interviews and focus group with
students
Open-ended questions on
surveys or questionnaires
Open-ended assessments
Portfolios of student work or
other performance artifacts
Open-ended classroom
observation notes
Journals, logs or other artifacts
of teaching practice
Tips for Data Collection
• Strategize so you’ll be able to compare groups to
determine impact, e.g. before to after, one group to
another
• Include “identifier data” needed for analysis, e.g.
semester, gender, program
• Use formats that collect data easily, cleanly, e.g.
Google Forms, Survey Monkey
• Use calendars so people know what you’ll need, when
• Trust matters – trusting instruments, trusting results,
providing honest feedback
Beware of Common Data Traps!
• Biting off more than you can chew
• Not collecting data needed to answer
important questions
• Collecting data that is not really useful
• Neglecting hard-to-quantify data
• Not formalizing “informal data” (e.g.,
anecdotes, unrecorded observations)
• Not using valuable data after it has been
collected
Measurement Challenges are Real
• Education context is messy and dynamic
• Standardized test scores are blunt
• Surveys are narrow and expensive to design
• Qualitative data can be expensive to analyze
• Experiments are expensive to run
• It takes time for conclusions to be reached in
education research community
So…
Until Then, Rely On:
Multiple
measures
Sampling
strategies
Lessons from
others
Your own
thoughtful
judgments
Thank You.
Contact us:
Malinda Faber: malinda_faber@ncsu.edu
Jeni Corn: jocorn@ncsu.edu
Request the S-STEM, T-STEM Surveys:
miso.ncsu.edu/articles/s-stem-survey
miso.ncsu.edu/articles/t-stem-survey