Outcome Based Innovation and
Engagement in STEM Courses: A
Model for Success
1
UNT Student Engagement Index
1. I spend my own time to learn more about
things taught in my class.
2. I believe that I am an active participant in my
learning.
3. I become immersed when working on
challenging activities.
4. In my class I notice when a new idea is
introduced.
5. I am willing to help other students with their
learning.
2
Presenters and Participants
• Rob Petros, Ph.D. | Assistant Professor, UNT
Department of Chemistry
• Mike Simmons, Ph.D. | Senior Associate
Director, CLEAR
3
Participant Outcomes
• Identify relevant student and faculty data related to STEM course
redesign
• Describe effective, learner focused instructional practices
• Describe the connection between outcome based education and
improvement of instruction
• Describe outcome based assessment strategies
• Identify basic milestones for success
• Discuss how various assessment and instructional best practices
can be applied to a variety of settings and institutions
• Develop strategies for implementing instructional and
assessment best practices at course, department and
institutional level.
• Develop a basic outline of ideas for incorporating outcome
based, learner centered instruction and assessment into their
own courses, programs, or institutions.
4
Roadmap of Presentation
•
•
•
•
The state of STEM/recent results
Course data and information
NextGen concepts
Chemistry Redesign
– Engagement
– Learner Centered and Strategies
• Questions and Feedback
5
Dr. Rob Petros: Chemistry
The Problem
• Only 40% of entering college students that
declare STEM majors complete degrees in STEM
disciplines
• Economic projections indicate 1 million more
STEM professionals will be needed in the US
than will be produced over the next decade at
current rates
Why are we losing these students?
1. Lack of inspiration in low level science classes
2. Frustration with the ‘weed out’ courses
3. STEM graduate programs prepare researchers
not educators
6
Problem Solved!?
• Redesign has increased enrollment from 145 to 192
(32%), while reducing actual class size from 145 to 96
($120K+ in annual tuition).
• Retention rate has increased dramatically ($250K?)
# of students
# of students retention
receiving D, F, or W
registered
rate (%)
2013/2014*
39
377
90
2012/2013
83
285
71
2011/2012
83
291
71
2010/2011
87
248
65
2009/2010
77
218
65
* first year incorporating engaged learning activities (peer group learning)
• Percent effort for teaching course has decreased
dramatically
7
Student Performance During Redesign
60
% of class
50
40
Results from ACS Standardized Final Exam
podcasts
introduced
NextGen
launched
lecture
eliminated
30
20
10
0
May-11 May-12 May-13 May-14† May-14‡ May-14*
Above Natl. Average
Bottom quartile
† All students taking second semester with Petros
‡ Students that were in Petros' section for both OChem I and II
* Students that were in Petros' section for only OChem II
8
Brainstorming: What do you know about your
students?
Identify three types of data/information that
is available about your students now.
Identify three types of data/information about
your students that you wish you had!
9
Importance of Course Data
• We hope to show:
– What data can come from courses?
– Why is course data important?
– How can course data be used?
– What are the issues and challenges?
– What’s happening at UNT and other
institutions?
– Implications and opportunities
10
NextGen Course Redesign
Course
Objectives
Instructional
Strategies
Assessment*
*Carriveau, R. S., Connecting the dots : developing student learning
outcomes & outcome based assessments. Fancy Fox Publications:
Denton, TX, 2010.
11
Student Learning Outcomes
• What the student is expected to know and be
able to do
– Useful in the classroom
• To students
• To Faculty
– Connects to “larger” goals
• Program
• Institution
• Accreditation
12
Example of Three Level SLO Structure
1. Goal: The student will understand literature (by)
1.1. General Learning Outcome (GLO): The student will demonstrate
accurate, critical, analytic reading of literature (by)
specific learning outcome statement (sLO):
The student will (be able to)
1.1.1 Identify important and supporting details
1.1.2 Recognize assumptions and inferences
1.1.3 Identify sequence of events
1.1.4 Determine the main idea/theme of a passage or
piece of literature
Source: Carriveau, R.S. (2011), Connecting the Dots: Developing Student Learning Outcomes and Outcome-Based Assessments
13
Code each item to a specific
learning outcome
1 Goal: The student will understand literature (particular period or genre)
1.1 General Learning Outcome (GLO): The student will demonstrate
accurate, critical, analytic reading of literature.
Specific learning outcome statements (sLO):
1.1.1 Determine the main idea/theme of a passage or piece of literature.
Test Item or Rubric Dimension
1.1.1 What was this passage mostly about? (Could also be CR)
A. Bias and prejudice can affect intellectual growth.*
B. Economic growth is impacted by bias and prejudice.
C. Current thoughts on bias and prejudice lack insight.
Source: Carriveau, R.S. (2011), Connecting the Dots: Developing Student Learning Outcomes and Outcome-Based Assessments
14
The Link Between Outcomes and Items
Calculating Outcome Attainment Values
1. 87
2. 90
3. 65
4. 58
5. 63
6. 52
7. 66
8. 77
9. 84
10 93
11. 96
12. 88
13. 82
14. 88
15. 90
16. 80
17. 92
18. 81
19. 81
20. 82
Specific Outcome
1.1.1
Avg = 81
Specific Outcome
1.1.2
Ave = 60
General Outcome
1.1
Avg = 81
Specific Outcome
1.1.3
Avg = 88
Specific Outcome
1.2.1
Avg = 85
Goal
1
Avg = 83
General Outcome
1.2
Avg = 85
Specific Outcome
1.2.2
Avg = 85
Source: Carriveau, R.S. (2011), Connecting the Dots: Developing Student Learning Outcomes and Outcome-Based Assessments
15
ExamSoft “Categories”
Use the “Categories” feature to create three levels
of outcome statements
16
Tag Learning
Outcomes for
each row
Weight
rows
17
Reporting Outcome Attainment
at the Program Level
Program
Goal
#1
% Correct
Goal 1
% Correct
Goal 2
% Correct
GLO 1.2
% Correct
GLO 2.3
% Correct
sLO 1.2.3
% Correct
# Exams
40
35
17
11
5
3
# Items
615
282
21
71
2
14
Group Average
85.23%
84.3%
83.8%
86.0%
92.9%
86.5%
Student A
75.77%
71.84%
72.73%
50%
100%
100%
Student B
89.71%
88%
90.91%
100%
100%
92.3%
Student C
83.82%
82%
100%
100%
84.6%
50%
Student D
77.45%
80%
72.73%
75%
100%
84.62%
18
Activity
• Write a Three Level Outcome Statement
19
Core Principles
• What are the essential principles every student
should take away from the course?
• Who are your students and what are you
preparing them for?
• Can your course teach important lessons
beyond your own discipline-specific content?
• How will you assess student attainment?
20
Course Objectives
•
Goal 1 – Students will understand molecular structure and its implication
for basic chemical reactivity
•
Goal 2 – Students will understand the concept of isomerism in organic
chemistry
•
Goal 3 – Students will understand various mechanistic pathways
commonly encountered in organic chemistry
•
Goal 4 – Students will understand how spectroscopy can be used to
determine molecular structure
•
Goal 5 – Students will understand the relationship of organic chemistry
to their everyday lives
•
Goal 6 – Students will be aware of the impact of the globalization of
scientific research on US competitiveness in science and technology and
be motivated to pursue and obtain degrees in STEM majors
•
Goal 7 – Students will demonstrate elements of collaboration, leadership,
innovation, problem solving, creativity, teamwork, and critical thinking
21
Objectives and Professional Standards
• American Chemical
Society’s Criterion
Referencing Project
underway, but not
completed.
• Lumina Foundation – DQP
• University Goals
• Currently based on
important topics from ACS
study guide
22
23
24
Online Content
252 podcasts totaling ~23h of recorded material produced covering all topics
from both semesters posted on UNT’s iTunesU site
25
STEM Incentives Program
• Information on graduate school in the
sciences
• Careers in chemistry
• Where my classmates are now
• Need for diversity in STEM
• Science and Engineering Indicators 2012
• Activity demonstrating the globalization of
scientific research
– List authors’ home institution for first 20
articles in the first issue of JACS in 1982,
1992, 2002, and 2012
26
STEM Incentives Program
27
Engaged Learning Activities
*Office of Science and Technology Policy – Engage to Excel (2012)
28
Class Time – Small Group Work
• 192 students divided into 32 groups of 6
• Each group meets 2hr/wk
• Can observe skills related to course goals 5-7
29
Group Assignments
30
31
Formative Assessment On Demand
32
More Robust Analysis
Learning Objective
1. Students will understand molecular structure and its
implication for basic chemical reactivity
1.1 Students will know and apply the naming system for
organic compounds
1.1.1 Students will recognize and correctly name
molecules containing functional groups
1.1.2 Students will correctly name stereoisomers
1.2 Students will understand the relationship between
structure, hybridization, resonance, and aromaticity
1.2.1 Students will identify the correct hybridization
state for C, N, O and other relevant atoms
1.2.2 Students will identify factors that lead to
stabilization in resonance structures
1.3 Students will understand the role of acidity⁄basicity in
reactions
1.3.1 Students will predict products of an acid⁄base
reaction
1.3.2 Students will identify acid⁄base conjugate pairs
1.3.3 Students will use pKa values to predict relative
acidity
# Assessments
# Items
Group Average
11
66
73.36%
11
23
71.06%
9
23
71.24%
2
1
70.56%
2
21
81.80%
2
18
81.19%
2
3
84.86%
4
23
70.67%
2
2
73.89%
2
6
89.73%
4
15
66.86%
33
Tracking Student Learning in Real Time
First Half of
Semester
Number Group
of Items Average
1.1.1 Students will recognize and correctly name
molecules containing functional groups
1.3.3 Students will identify acid⁄base trends for
common functional groups
2.1.1 Students will recognize the mechanisms for
electrophilic and nucleophilic aromatic substitution
reactions
2.1.2 Students will predict the site of substitution
based on substituents present on the aromatic ring
2.1.5 Students will predict products of reactions given
specific starting materials
3.1.1 Students will predict reaction products based on
starting materials
3.1.2 Students will differentiate between reversible
and irreversible addition reactions
5.2.1 Students will analyze spectral data to predict
molecular structure
Second Half of
Semester
Number
Group
of Items Average
10
91%
5
97%
10
82%
2
69%
2
38%
1
84%
2
44%
1
85%
2
80%
12
87%
35
76%
29
83%
8
87%
4
93%
12
84%
9
88%
34
Rob Petros – Improved from 56% to 71%
Fourth Exam Performance (split into two section)
Remediation Assignment Performance (sections combined)
35
Rob Petros – Early intervention
Student Feedback
• Student feedback
released on
learning outcome
performance and
areas of
strength/weakness
• Displayed score
upon exam exit
36
Lessons Learned
1. More students can be reached through
engaged learning environments
2. Real-time analysis of student attainment data
can be used to improve student performance
3. Setting expectations for students is critical
4. Participation must be part of students’ grades
5. Groups of all male or all female students
appeared to be less engaged
6. Ideally, roles should be assigned to each group
member that can then be rotated over the
course of the semester
37
Engagement in the Class
• The purpose of the UNT Student Engagement Inventory
(SEI) is to provide information about student engagement
that instructors can use to make meaningful continuous
improvements to their course.
• Part 1, Personal Internal Motivation, is a measure of the
students’ personal internal predisposition to be engaged in
learning.
• Part 2, Classroom Interaction is a measure of the students’
perception of engagement based on interactions in the
teacher-created classroom learning environment.
• Part 3, Social Interaction, is a measure of the social
interaction among classmates in interactive classroom
experiences.
38
SEI – Classroom Interaction
• Part 2 – Classroom Interaction total scale
score.
– I am satisfied with what I learned in my class.
– Opportunities to interact with my instructor
were provided in my course.
– I interact positively with my instructors.
– I am encouraged in my class to reflect on
what I learn.
– I work harder in my class than I expected to.
39
SEI – Social Interaction
• Part 3 – Social Interaction in the classroom
total scale score.
– I actively participate when a group project is
assigned in a class (in-class or on-line).
– Opportunities to collaborate with fellow students
are provided in my class.
– I actively participate in study groups.
– My class requires the use of electronic media for
discussions.
– I contribute to class discussions (in-class or online).
40
Review
• Identify relevant student and faculty data related to STEM
course redesign
• Describe effective, learner focused instructional practices
• Describe the connection between outcome based education and
improvement of instruction
• Describe outcome based assessment strategies
• Identify basic milestones for success
• Discuss how various assessment and instructional best
practices can be applied to a variety of settings and institutions
• Develop strategies for implementing instructional and
assessment best practices at course, department and
institutional level.
• Develop a basic outline of ideas for incorporating outcome
based, learner centered instruction and assessment into their
own courses, programs, or institutions.
41
Contact
• Dr. Rob Petros
– robby.petros@unt.edu
• Dr. Mike Simmons
– mike.simmons@unt.edu
42
Planning
• Learning Strategies worksheet
• Teaching Strategies worksheet
• Assessment Strategies worksheet
43