Using systems thinking to frame the evaluation of a complex educational intervention: InTeGrate as a case study Kim Kastens, Carol Baldassari, Jacqueline DeLisi and Cathryn Manduca American Geophysical Union December 17, 2014 This work is supported by a National Science Foundation (NSF) collaboration between the Directorates for Education and Human Resources (EHR) and Geosciences (GEO) under grant DUE - 1125331 • What is InTeGrate? • Elements of systems thinking to be discussed • Using systems thinking to design, implement and evaluate InTeGrate InTeGrate: Interdisciplinary Teaching of Geoscience for a Sustainable Future • NSF STEP Center = STEM Talent Expansion Program • Undergraduate education • $10M 5years finishing year 3 Two required goals: • • National impact on increasing number of students in STEM pipeline Address a national grand challenge: in InTeGrate’s case, environmental sustainability and resource limitations. Materials development: • Modules or full courses • Developed by 3-4 person faculty teams • Each team member must be from a different institution • Each team member must commit to testing the material • 36 teams • 109 team members, 23 states • Two year process o 6 published o 9 tested in 35 courses with 1338 students InTeGrate Curriculum Development & Revision Rubric Guiding principles (non-negotiable): Pedagogical excellence: • • • • • • • • • • Grand challenge facing society Interdisciplinary problem solving Geoscientific habits of mind Authentic geoscience data Systems thinking Learning objectives & goals Assessment & measurement Resources & materials Learning strategies Alignment of module/course elements Workshops: • 13 in-gathering workshops completed • 276 participants • Switching now to dissemination workshops Website: • 1038 public-facing pages • 35,000 visitors in last year • serc.carleton.edu/integrate Implementation Programs: • Larger scale than a single course • Innovative approaches to increase number and diversity of students developing Earth literacy OR • Prepare diverse workforce equipped to bring geoscience to bear in addressing societal issues • 7 in place, goal 26 Complex system Elements of systems thinking we are using • Systems mapping • Non-linear causal chains • Constant “sensing” of the system and its environment for decision-support • Emergent phenomena • Feedback loops • Plastic behavior: lasting change Systems mapping (related to conjecture mapping and logic modeling) Inputs ...leads to … …favors… Activities …influences… …contributes to … …shapes the planning for… Outputs Outcomes For each link: • Is there a mechanism? • Is there empirical evidence? Conjecture mapping of InTeGrate January 2012 For each link: • Is there a mechanism? • Is there empirical evidence? Elements of systems thinking • Systems mapping • Non-linear causal chains • Constant “sensing” of the system and its environment for decision-support • Emergent phenomena • Feedback loops • Plastic behavior: lasting change Caus e x Effect Action Nudg e Influence Action Nudg e Effect Intervention x Outcome Outcome 1 Activity Outcome 2 Outcome 3 InTeGrate’s design conjectures that any given desirable outcome will require multiple nudges or influencers. Frequent intense discourse with teammates around learning goals, content, pedagogy Leads to Individual and group reflection on what is and is not working Leads to Frequent immersion in rubric, web site and other guiding materials Substantial interactions with assessment team consultant (yr 1) and development team leader Leads to Instructors who have internalized InTeGrate values and methods Leads to become Leads to forge strong collaborative ties that can grow into Instructors incorporate InTeGrate values and methods into the rest of their teaching practice Advocates for InTeGrate’s ideas, programs and materials Enduring community of practice Evaluation inserts probes at critical flows, seeking evidence for the veracity and robustness of the conjectured flow. “Having goals and objectives formalized forced me to think differently about the design of course. …InTeGrate’s process forces you to think about how to start, where are you going towards. … I’m thinking about redoing all my classes.” Instructors who have internalized InTeGrate values and methods Leads to Evaluation target become forge strong collaborative ties that can grow into Instructors incorporate InTeGrate values and methods into the rest of their teaching practice Advocates for InTeGrate’s ideas, programs and materials Enduring community of practice Elements of systems thinking • Systems mapping • Non-linear causal chains • Continuous “sensing” of the system and its environment for decision-support • Emergent phenomena • Feedback loops • Plastic behavior: lasting change Complex problems require constant “sensing” of the status of the system and its environment… Meadows, D. (1999). Leverage Points: Places to Intervene in a Systems: The Sustainability Institute. http://faculty.cs.byu.edu/~mike/mikeg/WORKSHOP/panel.html Use of technology to support decision-making http://serc.carleton.edu/admin/review/process_overview.php?process_id=3 Elements of systems thinking • Systems mapping • Non-linear causal chains • Constant “sensing” of the system and its environment for decision-support • Emergent phenomena • Feedback loops • Plastic behavior: lasting change Emergent phenomena Larger entities, patterns, and regularities arise through interactions among smaller or simpler entities that themselves do not exhibit such properties. Towards STEM careers http://www.pbs.org/wgbh/nova/sciencenow/3410/03-ever-01.html (GLE & embedded assessments) Non-Geo majors with increased understanding of Earth and environment (aka increased “geoliteracy”) Teach Geo modules and courses that are pedagogically excellent and oriented towards societally important Earth-related problems Attitudinal survey; GLE essay questions Progress on nationally- urgent problems concerning resources and environment More Geo graduates who are equipped and disposed to address societal problems of resources and environment Attitudinal survey More students attracted to Geoscience majors and careers Inputs STEM (Geo) Talent Expansion Activities Outputs Outcomes Elements of systems thinking • Systems mapping • Non-linear causal chains • Continuous “sensing” of the system and its environment for decision-support • Emergent phenomena • Feedback loops • Plastic behavior: lasting change Feedback loops (A) causes or influences (B)…. …and then (B) in turn causes or influences (A). http://www.smashingmagazine.com/2013/02/15/designing-great-feedback-loops/ Balancing feedback loops: Materials development teams are required to have 3 or 4 people, from different types of institutions, and all developers must teach and test materials. Materials are being developed to be suitable for a wide range of students and institutions Materials suitable for a wide range of students and institutions Note that feedback loop does not pass through the leadership team… (-) One instructor/ developer proposes activity optimized for his/her locale, institution, students Another instructor/ developer objects The group compromises (-) Narrowly suitable activity is included in the test module Narrowly suitable activity works suboptimally in other test schools Materials are widely adopted/ adapted Module is revised to suit wider range of students & settings. ... Individuals throughout the InTeGrate ecosystem are empowered to detect and act on opportunities and fix problems. Elements of systems thinking • Systems mapping • Non-linear causal chains • Continuous “sensing” of the system and its environment for decision-support • Emergent phenomena • Feedback loops • Plastic behavior: lasting change As practitioners of an historical science, geoscientists know that events leave traces, even long after the causal forces have ended. What will be the traces left behind on the landscape of higher education when the InTeGrate event has ended? Potential types of traces include. . . • • • • Shared vision: of education, of geosciences Tested, reusable materials and tools Effective, transferable processes People who do things differently: faculty, students, graduates • Relationships among people: Friendships, collegial relationships, and trust Shared vision: The field of geosciences offers insights and approaches that are fundamental to tackling the 21st century’s grand challenges… From a non-geoscientist: “I think the materials did an excellent job of acquainting the students with the meaning of environmental justice. The materials also taught them how important science is to social justice.” From a geoscientist: “I always had this sense that these grand challenges were a better way to teach the basic stuff. InTeGrate’s structure, their guiding principles, affirmed that idea. …” Tested & effective tool: Curriculum Development & Revision Rubric • “The rubric really helped me focus my materials, resources, and activities on geoscientific habits of mind ….” • “The rubric pushed us in directions that we might not have gone—the interaction between earth science and economic, societal, and policy issues.” Effective, transferable management processes: Use of data and technology to create a decision-support system People who do things differently: Graduates who make different life choices ….most accurately depicts your level of interest in a career in Earth or Environmental Sciences…. 36% Relationships among people: Friendships, collegial relationships, and trust Fosters strong collaborative ties that can grow into Enduring community of practice “I’m really happy to have this opportunity to be involved in the project. I was inspired to work with great colleagues, share new ideas, and be involved in a new paradigm in curriculum development. I learned a lot and I would love to do a project like this again.” Recap • Education is a complex system • Systems thinking (or complexity theory) is a well developed discipline with a useful toolkit for thinking about education reform • • • • • • Systems mapping Non-linear causal chains Continuous “sensing” of the system for decision-support Emergent phenomena Feedback loops Plastic behavior: lasting change • InTeGrate is an ambitious work in progress. Early indicators are that it may succeed in leaving lasting traces upon the landscape. • • • • • Shared vision Tested, reusable materials and tools Effective, transferable processes People who do things differently: faculty, students, graduates Relationships among people: Friendships, collegial relationships, and trust Additional thinking from: • • • • • • Anne Egger Ellen Iverson David McConnell David Steer InTeGrate Advisory Board InTeGrate Leadership Team Literature insights from: • • • • • • Donella Meadows Margaret Hargreaves William Sandoval John Kania, Mark Kramer & Patty Russell Hallie Preskill & Srik Gopal Michael Quinn Patton