3/10/2015
The Challenges of Spreading and
Sustaining Research-Based
Instruction in Undergraduate STEM
Charles Henderson
Western Michigan University
homepages.wmich.edu/~chenders
Awards: #0715698, #1022186,
#0623009, #0723699
RUME 2015
February 19, 2015
Abstract
There have been many calls for the reform of introductory
Science, Technology, Engineering and Mathematics (STEM)
courses. These calls have resulted in a cadre of researchers
who study the teaching and learning of undergraduate STEM
and have developed instructional methods that improve
student learning. There currently exists a substantial gap
between research-based knowledge of ‘best practice’
instructional methods and the teaching practices of typical
STEM faculty. This talk will connect data about the spread of
research-based instructional strategies in college-level STEM
to ideas from the change literature. Recommendations will be
made for how to decrease the knowledge-practice gap.
1
3/10/2015
Starting Point
• Standard Instructional Practices in STEM are not
working
• We know a lot about:
– effective teaching and learning of STEM
subjects
– how to apply this knowledge in individual
classrooms
• An Important Question is:
– How can we encourage the spread of researchbased teaching to all instructors and
classrooms?
An important, highly replicable, result from DBER is
that active learning instructional strategies increase
student performance.
Effect sizes by discipline (from a metaanalysis of 225 studies – Freeman et al., 2014).
Learning Gains (Hedges’s g)
% Decrease in Failure Rates
©2014 by National Academy of Sciences
2
3/10/2015
STEM change agents often think about change
in terms of development and dissemination*
Desired Situation
Initial Situation
Development
Dissemination
• An individual or small group
• Develop polished product
• Conduct rigorous testing to
show efficacy
• Talks
• Papers
• Web site
*Henderson, C., Beach, A., & Finkelstein, N. D. (2011). Facilitating change in undergraduate STEM instructional practices: An analytic
review of the literature. Journal of Research in Science Teaching, 48(8), 952–984. doi:10.1002/tea.20439
An Example from Physics: Peer
Instruction
Class is a series
of ~15 min cycles
Brief Lecture (5-10 min)
Concept Question (1 min)
Students Answer Individually (1-2 min)
Students Discuss in Small Groups (2-4min)
Students Answer Again
Whole Class Discussion (2+ min)
Adapted from: http://perusersguide.org/guides/Section.cfm?G=Peer_Instruction&S=What
3
3/10/2015
Dissemination: Talks
From: http://mazur-www.harvard.edu/
Mazur has given over 600 talks about Peer
Instruction (Mazur, April 2013, Personal communication).
Dissemination: Significant Materials Available
• 253 page book with detailed
implementation
recommendations and disk
with ready-to-go materials:
– In-class questions
– Reading quizzes
– Exam questions
• Publisher has distributed book
for free to large numbers of US
physics faculty.*
– 18,700 copies shipped since
1996
– 12,700 free
*From Mazur, 2009 AAPT Winter Meeting
4
3/10/2015
Ideally, this would lead to an ever increasing
number of users
Development
Dissemination
time
Users (%)
Use would grow, slowly at first, then faster and
eventually reach a saturation point
Time
5
3/10/2015
Users (%)
Actual results may vary
Time
According to development and dissemination,
individual instructors must go through the
Innovation Decision Process*
Knowledge
Persuasion
Decision
Implementation
Confirmation
Change agents need to support this process
Knowledge
Persuasion
Support
12
*Rogers, Diffusion of Innovations
6
3/10/2015
Important Questions for Change Agents at each
Stage of the Innovation Decision Process
• Knowledge: How do instructors learn about
new instructional practices?
• Persuasion: What motivates instructors to try
something new? What do instructors perceive
as affordances and barriers?
• Support: What leads to successful vs.
unsuccessful use?
Overview of Presentation
What do we know about knowledge,
persuasion, and support?
• The big picture (w/ M. Dancy)
– In general, where are the biggest losses on the
innovation-decision process?
• Specific research-based instructional
strategies
– Peer Instruction (w/ M. Dancy, C. Turpen)
– SCALE-UP (w/ M. Dancy, B. Beichner, K. Foote)
7
3/10/2015
The Big Picture
Data Collection: Web Survey
– Administered by American Institute of Physics Statistical
Research Center, Fall 2008
– Random sample:
• 1) two year colleges
• 2) four year colleges with a physics B.A.
• 3) four year colleges with a physics graduate degree
– 722 useable responses (response rate 50.3%)
– Questions about knowledge and use of 24 ResearchBased Instructional Strategies (RBIS)
• Henderson, C. & Dancy, M. (2009) The Impact of Physics Education Research on the Teaching of
Introductory Quantitative Physics in the United States, Physical Review Special Topics: Physics Education
Research, 5 (2), 020107.
• Dancy, M. & Henderson, C. (2010) Pedagogical Practices and Instructional Change of Physics
Faculty, American Journal of Physics, Physics, 78 (10),15
1056-1063.
Where are the biggest shortfalls?
(Survey data from 722 Physics Instructors)
Knowledge
Persuasion
Decision
Implementation
Confirmation
Knowledge
Persuasion
Support
Know
about 1
or more
RBIS
Have
tried 1 or
more
RBIS
Currently
use 1 or
more
RBIS
16
8
3/10/2015
Impact Current Dissemination Strategies on
Undergraduate Physics Instruction
32%
Discontinuation
Lack of
Lack of Persuasion
Knowledge
Lack of
Support
Correlated Variables†
Variable
READ (teaching-related journals)
NFW (Physics New Faculty Workshop)
ATND (talks/workshops)
MORE (interest in using more RBIS)
GEN (gender)
SATF (satisfied with meeting goals)
PSTN (full-time, permanent vs. other)
RSH2 (research publications)
SIZE (class size)
INST (type of institution)
CRSE (alg- or calc-based course)
DGRE (highest degree)
ENC (departmental encouragement for teaching)
GOAL (instructional goals)
JOB (% of job related to teaching)
PEER (frequency of talk w/ peers about teaching)
RANK (academic rank)
RSH1 (research presentations)
RSH3 (research grants)
YEAR (years of teaching experience)
†Controlling
for other study variables using a logistic regression model.
Have
Knowledge
*
*****
**
Persuaded to Continued
Try
Use (Support)
*
***
*
*
*
*
*
*Strength of effect is based on size of odds ratios (each * ~ odds ratio of 2).
9
3/10/2015
The Physics NFW is effective traditional dissemination
27%
Discontinuation
33%
Discontinuation
Summary – Part 1: Big Picture
• Traditional dissemination seems to work
– Well for Knowledge
– OK for Persuasion
– Poorly for Support
• Intensive one-time workshops like the Physics
New Faculty Workshop appear to be
particularly effective forms of traditional
dissemination, but still are not effective at
support.
10
3/10/2015
Part Two: Specific Strategies
Peer Instruction and SCALE-UP
Peer Instruction
• Moderate change
in pedagogy
• No change in
resources needed
SCALE-UP
• Large change in
pedagogy
• Large change in
resources needed
Knowledge vs. Use
(self-report from web survey)
Most Known
Most Used
1. Peer Instruction (64%)
1. Peer Instruction (29%)
2. Physlets (56%)
2:1
5. Just in Time Teaching (48%)
2. Ranking Tasks (15%)
5. Physlets (13%)
6. Just in Time Teaching (8%)
9. Ranking Tasks (39%)
10. SCALE-UP (34.5%)
10:1
15. SCALE-UP (3.3%)
Henderson, C. & Dancy, M. (2009) The Impact of Physics Education Research on the Teaching of Introductory Quantitative Physics in the
United States, Physical Review Special Topics: Physics Education Research, 5 (2), 020107.
11
3/10/2015
Interviews with instructors who know about
Peer Instruction
722 Instructors Surveyed
Knows about Peer Instruction (64%)
NonUser
Former
User
User
Randomly-Selected Interview Sample
N=14
N=6
N=15
Topics Covered in Interviews
- How faculty learned about Peer Instruction,
- Instructional practices in most recent intro
physics course,
- How and Why started using Peer Instruction,
- How was Peer Instruction implemented,
- Institutional & Departmental Context.
12
3/10/2015
Knowledge: Problems Communicating
about Innovations
• Names of innovations mean very different
things to different people
• Innovations are commonly modified during
implementation
– Few instructors (between 6% and 47%) use Peer
Instruction as described by the developer.
– In many cases instructors are not aware of these
differences.
25
From Interviews
A significant minority of faculty (9/35 =26%) only
demonstrated use of the term peer instruction in a
literal way, typically as any in-class or out-of-class
activity where students worked with one-another.
Texas Tech: Drop in Peer Tutoring is available
free to all currently enrolled students.
13
3/10/2015
From Web Survey – Use of ‘Essential
Features’ of Peer Instruction
Features of Peer Instruction (measured on survey)
Traditional Lecture (for nearly every class or multiple times every class)
Students discuss ideas in small groups (multiple times every class)
Students solve/discuss qualitative/conceptual problem (multiple times every class)
Whole class voting (multiple times every class)
Conceptual questions (used on all tests)
Self-Described
Users of Peer
Instruction
55%
27%
27%
38%
64%
6%
21%
35%
Uses all 5 components
Uses 4 or more of the 5 components
Uses 3 of more of the 5 components
Use of ‘essential features’ was even lower for Cooperative Group Problem Solving.
27
From Telephone Interviews (N=35)
Relationship between self-described user status and Use of Peer Instruction Features
10
Users
Former Users
Non-Users
9
# of interviewees
8
7
4/9
1/9
6
7/15 (47%) selfdescribed users use at
least 7 PI features.
6/9
5
4
3
2
1
0
0
1
2
3
4
5
6
7
8
9
Number of Peer Instruction features used by self-described
28
users, former users, and knowledgeable
non-users.
14
3/10/2015
Communication Difficulties: Self-described user
status does not help much in understanding
actual instructional behavior.
SelfDescribed
User Status
Knows about Peer Instruction
Non-User
N=14
9 5
Researcher
Described
User Status
Former User
N=6
1
5
User
N=15
8 7
Non-User
N=10
Mixed User
N=18
High User
N=7
0 Features
1-6 Features
7-9 Features
Inappropriate Assimilation
The addition of the concept of Peer Instruction often
does not result in qualitatively different thinking about
teaching
For Piaget, learning
occurs via assimilation
(quantitative change in
mental structure) or
accomodation
(qualitative change).
Figure from by Dan MacIsaac, Wednesday, May 1, 1991
http://physicsed.buffalostate.edu/danowner/kuhnpiaget/KP1.html
Submitted to Dr S. Abell as partial requirement for EDCI 591S
15
3/10/2015
Conclusions (so far)
• Development and dissemination change
strategies in physics have resulted in
– Widespread knowledge about RBIS (~9/10 of faculty)
– Faculty motivation to try RBIS (~3/4 of faculty)
• Of those who try
– ~ 1/3 discontinue use
– ~ 1/3 engage in inappropriate assimilation
– ~ 1/3 continue use
Knowledge: Informal discussions are the most
common first exposure to Peer Instruction. Other
forms of exposure are used later.
* 24/35 respondents identified their ‘first exposure’.
16
3/10/2015
Knowledge: Local colleagues are the most
common informal discussion partners.
* 24/35 respondents identified their ‘first exposure’.
Knowledge: Exposure via local colleagues may
be related to higher levels of high use
Source of any % High Users
% Mixed
% Non-Users
exposure to PI
(N=7)
Users (N=18)
(N=10)
Grad school
14.3
16.7
0.0
colleague
Local colleague
71.4
44.4
30.0
Non-Local
14.3
38.9
30.0
Colleague
Colleague 14.3
16.7
0.0
unclear
17
3/10/2015
Conclusions (so far)
• Personal interactions with colleagues:
– are a very important source of knowledge about Peer
Instruction
– are the most common first exposure to Peer
Instruction
– may lead to higher levels of use
Persuasion:
Affordances of Peer Instruction (from interviews)
% total
Dissatisfaction with lecture
57.1
Evidence of effectiveness from personal experience
54.3
Gets students active
48.6
Departmental support or encouragement
45.7
Evidence of effectiveness from data
42.9
Intuitively makes sense to me
37.1
Provides feedback to the instructor
34.3
Gets students working together
31.4
Encourages depth of understanding
25.7
Students learn by hearing a peer’s explanation
25.7
Students learn by giving an explanation to a peer
22.9
Forces more students to participate
20.0
18
3/10/2015
Persuasion:
Constraints of Peer Instruction (from interviews)
Requires time and energy to change
Content coverage concerns, personal belief
Difficulty to getting students engaged
Students aren’t capable of doing it
In personal experience it did NOT work
Structural, lack of resources
Structural, class size
Structural, lack of appropriate classroom
Trouble finding good questions
Difficulty getting student buy-in
Current practices are effective
Intuitively don’t think that PI will work
Content coverage concerns, external requirements
Content coverage concerns, institutional expectations
% total
57.1
48.6
48.6
37.1
34.3
34.3
31.4
31.4
31.4
28.6
25.7
25.7
22.9
20.0
Persuasion: Affordances and Constraints
of using Peer Instruction
Average # of Codes
12.0
10.0
Reasons
aligning with
Perceivedfor
affordances
of PI PI
Perceivedfor
constraints
of PI with PI
Reasons
NOT aligning
8.0
6.0
4.0
2.0
0.0
High Use
Mixed Use
Non-Use
Researcher Assigned Implementation Group
19
3/10/2015
Persuasion:
Variations by Implementation Group
NON (N=10)
MIXED (N=18) HIGH (N=7)
Most Prevalent Affordances
Most Prevalent Constraints
1. Dissatisfaction with traditional lecture (86%)
2. Evidence of effectiveness, personal experience (71%)
3. Forces more students to participate (71%)
4. Evidence of effectiveness, data (57%)
PI makes intuitive sense (57%)
Provides feedback to the instructor (57%)
1. Dissatisfaction with traditional lecture (78%)
2. Gets students active in class (67%)
3. Evidence of effectiveness, personal experience (56%)
4. Departmental support or encouragement (50%)
1. Difficulty of getting students engaged (100%)
2. Trouble finding good questions (57%)
1. Evidence of effectiveness personal experience (50%,)
1. Time and energy required to change (90%)
2. Student deficiencies (60%)
3. Personal commitment to content coverage (50%)
Structural, Lack of resources (50%)
Structural, Class size (50%)
Current practice effective (50%)
External requirement of content coverage (50%)
1. Time and energy required to change (56%)
2. Personal commitment to content coverage (50%)
Conclusions (so far): Persuasion
• Most interviewees in all user groups are persuaded
that Peer Instruction is effective
– Personal experience is more persuasive than data
• Lower users are concerned about logistical problems
– Time required to change
– Lack of time to cover necessary content
• Higher users are concerned about improving quality
of implementation
– Getting students involved
– Finding instructional resources
20
3/10/2015
SCALE-UP
SCALE-UP Involves redesigning the
classroom and pedagogy
21
3/10/2015
SCALE-UP
• Developed by
Bob Beichner in
1993 for Physics
• Influenced
teaching
practice in a
minimum of 314
departments at
189 higher
education
institutions*
*Foote, K. T., Neumeyer, X., Henderson, C., Dancy, M. H., & Beichner, R. J. (2014). Diffusion of researchbased instructional strategies: the case of SCALE-UP. International Journal of STEM Education, 1(1), 10.
SCALE-UP is Spreading
SCALE-UP Departments in Existence
120
# of SCALE-UP Dpmts in Existence
• Survey of 659
SCALE-UP users
at 2- and 4-year
colleges in the US
• 63% of reported
departments
using SCALE-UP
are outside the
originating
discipline of
physics, and 20%
are outside of
STEM.
100
Physics
Chemistry
Biology & Health
Engineering
Math, Stats, Comp. Sci.
Non-STEM
80
60
40
20
0
2007
2008
2009
2010
Year
2011
2012
2013
*Foote, Neumeyer, Henderson, Dancy, Beichner (submitted) Diffusion of Research-Based Instructional
Strategies: The Case of SCALE-UP
22
3/10/2015
Knowledge: Like Peer Instruction, interactions
with colleagues are the most common sources
of knowledge about SCALE-UP
Sources of Knowledge about SCALE-UP
Type of Colleague
Exposure
Percentage of SCALE-UP users
40%
35%
30%
25%
20%
15%
10%
5%
0%
Colleague
Talk/Workshop
Web
Literature
Other/don't
know
(How did you receive and share information about SCALE-UP?)
Knowledge about SCALE-UP travels most within institutions (same
department is more common than different department).
Neumeyer, X., Foote, K. T., Beichner, R., Dancy, M. H., & Henderson C. (2014). Examining the Diffusion of Research-Based Instructional Strategies Using
Social Network Analysis: A Case Study of SCALE-UP. Proceedings of the 2014 ASEE Annual Conference.
23
3/10/2015
Beichner Cluster
The SCALE-UP diffusion network is highly centralized around Bob
Beichner.
Neumeyer, X., Foote, K. T., Beichner, R., Dancy, M. H., & Henderson C. (2014). Examining the Diffusion of Research-Based Instructional Strategies Using
Social Network Analysis: A Case Study of SCALE-UP. Proceedings of the 2014 ASEE Annual Conference.
Two Case Studies
Interviews with stakeholders at two intuitions
reporting successful, ongoing implementation of
SCALE-UP in multiple departments
Foote, K. T., Neumeyer, X., Henderson C., Dancy, M. H., & Beichner, R. (in press). SCALE-UP Implementation and Intra-Institutional
Dissemination: A Case Study of Two Institutions. Proceedings of the 2014 Physics Education Research Conference.
24
3/10/2015
Initiation
• U of Iowa (TILE) –Top Down
– Video of SCALE-UP reform was circulating around
campus, generating interest.
– Major flood resulted in need to reconstruct
classrooms and federal funding was available.
– Provost created team to decide on funding use, they
build SU classrooms and implemented training
program for faculty.
• Clemson – Bottom Up
– Math and engineering faculty were collaborating on
funded reform efforts and heard about SU through
NSF reform initiative and Beichner’s post-doc.
– High failure rates in gatekeeper courses were a
concern of upper administration who provided funds
for SU classrooms at request of faculty champions.
Spread
U of Iowa (7 TILE classrooms used by 60 departments)
• Rooms centrally controlled --> all departments could use
• Center for Teaching ran mandatory training program
• Some department chairs encouraged their faculty
• Faculty and student excitement about successful courses helped
motivate faculty
Clemson (10 SU classrooms used by 10 departments, all general
engineering courses are SU)
• In 2006, math department head decided all introductory calculus
courses would be SCALE-UP
• Spread from math and general engineering to civil and
mechanical engineering because of an interdisciplinary grant
• Instructors invited visitors to observe classes spread outside
STEM
25
3/10/2015
Commonalities
• Key figures with existing interest in reform
were introduced to SCALE-UP through
informal mechanisms and colleague
connections.
• Financial support for redesigned rooms was
available due to supportive administrators.
• Administrative support and the visibility of
redesigned classrooms contributed to success.
Implications
• Cross departmental collaboration assists
dissemination.
• Administration-faculty partnerships facilitate
the adoption of SCALE-UP.
• Redesigned classrooms add visibility and
sustainability.
26
3/10/2015
Summary: Knowledge
• Instructors learn about new instructional
strategies from colleagues (locally and
nationally)
• Other sources of information then become
important
• Inappropriate assimilation hinders
communication
Summary: Persuasion
• Instructors are most persuaded by personal
experiences
• Data appears to be primarily useful to confirm
decisions, not sway opinions
• Time to learn and implement is a major
barrier
• Fears of lower content coverage is a major
barrier
27
3/10/2015
Summary: Support
• We don’t do support very well
• A local community of users, administrative
encouragement/resources, and visibility (e.g.,
a SCALE-UP room) appears to be helpful in
supporting sustained use
STEM change agents often think about change
in terms of development and dissemination*
Development
Initial Situation
Dissemination
Desired Situation
*Henderson, C., Beach, A., & Finkelstein, N. D. (2011). Facilitating change in undergraduate STEM instructional practices: An analytic
review of the literature. Journal of Research in Science Teaching, 48(8), 952–984. doi:10.1002/tea.20439
28
3/10/2015
We can build a better model:
The Designing for Sustained Adoption Framework
http://www.increasetheimpact.com/
We can build a better model:
The Designing for Sustained Adoption Framework
• Analysis of typical development and
dissemination practices
– 75 NSF CCLI grant proposals funded in 2009
• Case studies of well-propagated innovations
– PhET, PLTL, Peer Instruction
• Synthesis of the related literature
29
3/10/2015
We can build a better model
(Solitary)
Development
(Mass Market)
Dissemination
Developing a strong product requires significant
feedback from potential users
Interactive
Development
(Mass Market)
Dissemination
Lesson: Curriculum developers need
to collaborate with and build in
opportunities for feedback from
target audience from the very
beginning.
30
3/10/2015
The most powerful forms of dissemination are
interactive
Interactive
Development
Interactive
Dissemination
Lesson: Curriculum developers
need to go beyond papers and
talks, develop opportunities for
personal interactions
Support is required for successful adoption
Interactive
Development
Interactive
Dissemination
Support
Lesson: The rate of “adopt and
drop” and inappropriate
assimilation is high. Many adopters
run into implementation
difficulties and need support for
successful and sustained use.
31
3/10/2015
The instructional system imposes significant
constraints. The more changes that the new instruction
requires in the instructional system, the harder it will
be to get adoption
Interactive
Development
Interactive
Dissemination
Initial System
Support
Desired System
Individual instructors and their classroom practices
are just one part of the instructional system.
Interactive
Development
Individual
Department
Institution
Extra-Institution
Initial System
Interactive
Dissemination
Lesson: It is essential to
understand what changes
the new instruction
requires from the
instructional system.
Support
Individual
Department
Institution
Extra-Institution
Desired System
32
3/10/2015
A more complete model to guide development
and dissemination activities
Interactive
Development
Interactive
Dissemination
Support
Individual
Individual
Department
Department
Institution
Institution
Extra-Institution
Extra-Institution
Initial System
Desired System
Most Important ‘Take-Away’ Message
Typical Practice
(Solitary)
Development
(Mass Market)
Dissemination
Successful Practice
Interactive
Development
Interactive
Dissemination
Support
http://www.increasetheimpact.com
33
3/10/2015
END
34