Using systems thinking to frame the evaluation
of a complex educational intervention:
InTeGrate as a case study
Kim Kastens, Carol Baldassari, Jacqueline DeLisi and Cathryn Manduca
American Geophysical Union
December 17, 2014
This work is supported by a National Science Foundation (NSF) collaboration between the
Directorates for Education and Human Resources (EHR) and Geosciences (GEO) under grant DUE - 1125331
• What is InTeGrate?
• Elements of systems thinking to be
discussed
• Using systems thinking to design,
implement and evaluate InTeGrate
InTeGrate: Interdisciplinary Teaching of
Geoscience for a Sustainable Future
• NSF STEP Center = STEM Talent Expansion Program
• Undergraduate education
• $10M 5years finishing year 3
Two required goals:
•
•
National impact on increasing number of students in STEM
pipeline
Address a national grand challenge: in InTeGrate’s case,
environmental sustainability and resource limitations.
Materials development:
• Modules or full courses
• Developed by 3-4 person
faculty teams
• Each team member must be
from a different institution
• Each team member must
commit to testing the material
• 36 teams
• 109 team members, 23 states
• Two year process
o 6 published
o 9 tested in 35 courses
with 1338 students
InTeGrate
Curriculum
Development
& Revision
Rubric
Guiding principles (non-negotiable):
Pedagogical excellence:
•
•
•
•
•
•
•
•
•
•
Grand challenge facing society
Interdisciplinary problem solving
Geoscientific habits of mind
Authentic geoscience data
Systems thinking
Learning objectives & goals
Assessment & measurement
Resources & materials
Learning strategies
Alignment of module/course elements
Workshops:
• 13 in-gathering workshops
completed
• 276 participants
• Switching now to
dissemination workshops
Website:
• 1038 public-facing pages
• 35,000 visitors in last year
• serc.carleton.edu/integrate
Implementation Programs:
• Larger scale than a single
course
• Innovative approaches to
increase number and
diversity of students
developing Earth literacy OR
• Prepare diverse workforce
equipped to bring geoscience
to bear in addressing societal
issues
• 7 in place, goal 26
Complex system
Elements of systems thinking we are using
• Systems mapping
• Non-linear causal chains
• Constant “sensing” of the system and its
environment for decision-support
• Emergent phenomena
• Feedback loops
• Plastic behavior: lasting change
Systems mapping
(related to conjecture mapping
and logic modeling)
Inputs
...leads to …
…favors…
Activities
…influences…
…contributes to …
…shapes the planning for…
Outputs
Outcomes
For each link:
• Is there a mechanism?
• Is there empirical evidence?
Conjecture mapping of InTeGrate January 2012
For each link:
• Is there a mechanism?
• Is there empirical evidence?
Elements of systems thinking
• Systems mapping
• Non-linear causal chains
• Constant “sensing” of the system and its
environment for decision-support
• Emergent phenomena
• Feedback loops
• Plastic behavior: lasting change
Caus
e
x
Effect
Action
Nudg
e
Influence
Action
Nudg
e
Effect
Intervention
x
Outcome
Outcome 1
Activity
Outcome 2
Outcome 3
InTeGrate’s design conjectures that any given desirable outcome will require
multiple nudges or influencers.
Frequent intense
discourse with
teammates
around learning
goals, content,
pedagogy
Leads
to
Individual and
group reflection
on what is and is
not working
Leads
to
Frequent
immersion in
rubric, web site
and other guiding
materials
Substantial
interactions with
assessment team
consultant (yr 1)
and development
team leader
Leads
to
Instructors who
have internalized
InTeGrate values
and methods
Leads
to
become
Leads
to
forge strong
collaborative ties
that can grow into
Instructors
incorporate
InTeGrate values
and methods into
the rest of their
teaching practice
Advocates for
InTeGrate’s ideas,
programs and
materials
Enduring
community of
practice
Evaluation inserts probes at critical flows, seeking evidence for the veracity and
robustness of the conjectured flow.
“Having goals and objectives formalized forced me to think differently about the
design of course. …InTeGrate’s process forces you to think about how to start,
where are you going towards. … I’m thinking about redoing all my classes.”
Instructors who
have internalized
InTeGrate values
and methods
Leads
to
Evaluation
target
become
forge strong
collaborative ties
that can grow into
Instructors
incorporate
InTeGrate values
and methods into
the rest of their
teaching practice
Advocates for
InTeGrate’s ideas,
programs and
materials
Enduring
community of
practice
Elements of systems thinking
• Systems mapping
• Non-linear causal chains
• Continuous “sensing” of the system and its
environment for decision-support
• Emergent phenomena
• Feedback loops
• Plastic behavior: lasting change
Complex problems require constant “sensing” of the status of the
system and its environment…
Meadows, D. (1999). Leverage Points: Places to
Intervene in a Systems: The Sustainability Institute.
http://faculty.cs.byu.edu/~mike/mikeg/WORKSHOP/panel.html
Use of technology to support decision-making
http://serc.carleton.edu/admin/review/process_overview.php?process_id=3
Elements of systems thinking
• Systems mapping
• Non-linear causal chains
• Constant “sensing” of the system and its
environment for decision-support
• Emergent phenomena
• Feedback loops
• Plastic behavior: lasting change
Emergent phenomena
Larger entities, patterns, and regularities arise through interactions
among smaller or simpler entities that themselves do not exhibit such
properties.
Towards
STEM
careers
http://www.pbs.org/wgbh/nova/sciencenow/3410/03-ever-01.html
(GLE &
embedded
assessments)
Non-Geo majors with increased
understanding of Earth and
environment (aka increased
“geoliteracy”)
Teach Geo modules and
courses that are
pedagogically excellent
and oriented towards
societally important
Earth-related problems
Attitudinal survey;
GLE essay questions
Progress on
nationally- urgent
problems
concerning
resources and
environment
More Geo graduates who are
equipped and disposed to
address societal problems of
resources and environment
Attitudinal
survey
More students attracted to
Geoscience majors and careers
Inputs
STEM (Geo)
Talent Expansion
Activities
Outputs
Outcomes
Elements of systems thinking
• Systems mapping
• Non-linear causal chains
• Continuous “sensing” of the system and its
environment for decision-support
• Emergent phenomena
• Feedback loops
• Plastic behavior: lasting change
Feedback loops
(A) causes or
influences (B)….
…and then (B) in turn
causes or influences
(A).
http://www.smashingmagazine.com/2013/02/15/designing-great-feedback-loops/
Balancing feedback loops: Materials development teams are
required to have 3 or 4 people, from different types of
institutions, and all developers must teach and test materials.
Materials are being
developed to be
suitable for a wide
range of students
and institutions
Materials suitable
for a wide range
of students and
institutions
Note that feedback
loop does not pass
through the
leadership team…
(-)
One instructor/
developer proposes
activity optimized
for his/her locale,
institution, students
Another
instructor/
developer
objects
The group
compromises
(-)
Narrowly suitable
activity is included
in the test module
Narrowly suitable
activity works suboptimally in other
test schools
Materials are
widely
adopted/
adapted
Module is revised to
suit wider range of
students & settings.
... Individuals
throughout the
InTeGrate ecosystem
are empowered to
detect and act on
opportunities and fix
problems.
Elements of systems thinking
• Systems mapping
• Non-linear causal chains
• Continuous “sensing” of the system and its
environment for decision-support
• Emergent phenomena
• Feedback loops
• Plastic behavior: lasting change
As practitioners of an historical science, geoscientists know that events
leave traces, even long after the causal forces have ended.
What will be the traces left behind on the landscape of higher education
when the InTeGrate event has ended?
Potential types of traces include. . .
•
•
•
•
Shared vision: of education, of geosciences
Tested, reusable materials and tools
Effective, transferable processes
People who do things differently: faculty, students,
graduates
• Relationships among people: Friendships, collegial
relationships, and trust
Shared vision: The field of geosciences offers insights and
approaches that are fundamental to tackling the 21st century’s grand
challenges…
From a non-geoscientist:
“I think the materials did an excellent job of acquainting the students
with the meaning of environmental justice. The materials also taught
them how important science is to social justice.”
From a geoscientist:
“I always had this sense that these grand challenges were a better
way to teach the basic stuff. InTeGrate’s structure, their guiding
principles, affirmed that idea. …”
Tested & effective tool:
Curriculum Development & Revision Rubric
• “The rubric really helped me focus my materials,
resources, and activities on geoscientific habits of
mind ….”
• “The rubric pushed us in directions that we might not
have gone—the interaction between earth science
and economic, societal, and policy issues.”
Effective, transferable
management processes:
Use of data and
technology to create a
decision-support system
People who do things differently:
Graduates who make different life choices
….most accurately depicts your
level of interest in a career in Earth
or Environmental Sciences….
36%
Relationships among people: Friendships,
collegial relationships, and trust
Fosters strong collaborative
ties that can grow into
Enduring
community of
practice
“I’m really happy to have this opportunity to be involved in the
project. I was inspired to work with great colleagues, share new
ideas, and be involved in a new paradigm in curriculum
development. I learned a lot and I would love to do a project like
this again.”
Recap
• Education is a complex system
• Systems thinking (or complexity theory) is a well developed discipline
with a useful toolkit for thinking about education reform
•
•
•
•
•
•
Systems mapping
Non-linear causal chains
Continuous “sensing” of the system for decision-support
Emergent phenomena
Feedback loops
Plastic behavior: lasting change
• InTeGrate is an ambitious work in progress. Early indicators are that it
may succeed in leaving lasting traces upon the landscape.
•
•
•
•
•
Shared vision
Tested, reusable materials and tools
Effective, transferable processes
People who do things differently: faculty, students, graduates
Relationships among people: Friendships, collegial relationships, and trust
Additional thinking from:
•
•
•
•
•
•
Anne Egger
Ellen Iverson
David McConnell
David Steer
InTeGrate Advisory Board
InTeGrate Leadership Team
Literature insights from:
•
•
•
•
•
•
Donella Meadows
Margaret Hargreaves
William Sandoval
John Kania, Mark Kramer & Patty Russell
Hallie Preskill & Srik Gopal
Michael Quinn Patton