AN ABSTRACT OF THE THESIS OF
Laura Ferguson for the degree of Master of Science in Marine Resource Management
in presented on September 28, 2015.
Title: Characterizing and Assessing the Researcher-Stakeholder Engagement Process
for Water Sustainability: The Willamette Water 2100 Project.
Abstract approved:
______________________________________________________
Samuel Chan
Natural resource management and policy is ideally informed by the best
available science. Natural resource researchers ideally participate in broader impacts
activities to extend the reach of their best available research. However, there are
many cultural, institutional, and practical barriers to participating in broader impact
activities and to incorporating science into natural resource use decisions. Researcherstakeholder engagement is one proposed solution to overcome such barriers and to
achieve both broader impact and science-based policy goals. This research explores
the research-stakeholder engagement process as a means to achieve those ends.
The objective of this study was to document the perceptions of participants in
a transdisciplinary researcher-stakeholder engagement process in order to identify its
impacts as well as barriers and pathways to its successes. Literature has documented
many researcher-stakeholder engagement process case studies where researchers offer
lessons learned and speculate on their impacts, but few offer data on the engagement
process structure, the stakeholder perspective of the engagement process, or the
impacts of collaboration between academic research teams and scientific
stakeholders. This work addresses these gaps by taking a closer look at how one team
of researchers engaged with its stakeholders and voicing the perceptions of
stakeholders in addition to researchers.
An exploratory sequential mixed methods approach was used for an in-depth
case study of the researcher-stakeholder engagement experience in Willamette Water
2100 (WW2100), a five-year transdisciplinary research project investigating the
biophysical and socioeconomic drivers of future water scarcity. Attendance records
characterize the individuals participating in each engagement event. Twenty-six semistructured interviews with key participants were collected, transcribed and analyzed
to identify recurring themes. An online survey of all researchers and stakeholders
engaged with the project (n=137; response rate = 49%) was then conducted to
document their perceptions their motivations to, expectations for, participation in, and
outcomes of WW2100.
The results presented here were intended to be representative of motivations,
expectations, challenges, successes, and outcomes salient to all WW2100
participants. Researchers and stakeholders were motivated to participate for social,
knowledge, and utility reasons and held different expectations for the roles they
would play, the researcher-stakeholder engagement process itself, and the resulting
research results. Four types of challenges were identified: lack of a shared vision,
differing professional languages, research complexities, and project management.
Participants identified successful outcomes including: overcoming challenges,
facilitating learning, greater understanding, conversation among diverse perspectives,
and improving and extending research results. Researcher-stakeholder engagement in
natural resource research can create more relevant science and achieve scientific
broader impact goals.
This research offers novel evidence of researcher-stakeholder engagement
impacts and proposes more specific criteria for broader impact activity evaluation.
©Copyright by Laura Ferguson
September 28, 2015
All Rights Reserved
Characterizing and Assessing the Researcher-Stakeholder Engagement Process for
Water Sustainability: The Willamette Water 2100 Project
by
Laura Ferguson
A THESIS
submitted to
Oregon State University
in partial fulfillment of
the requirements for the
degree of
Master of Science
Presented September 28, 2015
Commencement June 2016
Master of Science thesis of Laura Ferguson presented on September 28, 2015
APPROVED:
Major Professor, representing Marine Resource Management
Dean of the College of Earth, Ocean and Atmospheric Sciences
Dean of the Graduate School
I understand that my thesis will become part of the permanent collection of Oregon
State University libraries. My signature below authorizes release of my thesis to any
reader upon request.
Laura Ferguson, Author
ACKNOWLEDGEMENTS
It takes a whole village to raise a graduate student. I am grateful for the support of my
village throughout this degree’s journey. I entered the Marine Resource Management
Program with a mind to explore all the topics that fell within the interdisciplinary
degree. I was fortunate to be advised by Sam Chan who accepted me as his student,
encouraged my exploration, and supported me in classes, research, and work-life
balance. Sam pushed me to think critically, locally and globally, and was instrumental
to my development and success while at OSU. I am also grateful to my committee
members, Mary Santelmann and Bryan Tilt, for their continued support,
encouragement, and guidance. Thank you, Mary, for the many conversations on
theories and best practices in stakeholder engagement and for the hands-on qualitative
data analysis practice. Thank you, Bryan, for striking the qualitative data spark in
your Anthropology class, for your encouragement throughout the coding process, and
for the list of resources which strengthened this work. Thank you to Skip Rochefort,
GCR, for interesting and though-provoking questions. Thank you to Bo Shelby for
your help developing the survey instrument and leading me to think more deeply.
Thank you to Flaxen Conway for your tireless dedication to the MRM program and
its students. Thank you for your guidance in my education, support in pursuing new
opportunities, and for your presence as a role model. Thank you to the faculty at OSU
who taught me skills and provided knowledge and advice through coursework and
individual meetings. Thank you to the Graduate School and NSF for providing
financial support for this research, as well as Oregon Sea Grant who not only
provided financial support but also a professional community from which to learn.
Thank you to Lori Hartline and Robert Allan for the needed administrative reminders,
future employment daydreams and smiles. Thank you to the WW2100 research team
for allowing me into many privileged meetings and to all WW2100 participants for so
openly sharing your perceptions and opinions with me. I am grateful to my family
who love and support me no matter what. I am grateful to my friends who love and
support me like a family. Para Miguel, gracias por comprenderme y apoyarme en
todo, siempre fortalecido por amor. My successes and achievements are our successes
and achievements. Thank you for working so hard with me to arrive here.
TABLE OF CONTENTS
Page
CHAPTER ONE: INTRODUCTION, CONTEXT, & METHODS ............................. 1
Broader Impacts .................................................................................................................... 3
Trends in Collaborative Researcher-Stakeholder Engagement ............................................ 5
Defining the Case: Willamette Water 2100 Researcher-Stakeholder Engagement ........... 11
Willamette Water 2100 Model ....................................................................................... 12
Willamette Water 2100 Process Structure ..................................................................... 13
Methods .............................................................................................................................. 15
Attendance Record Analysis ........................................................................................... 15
Qualitative Interview Phase ............................................................................................ 16
Quantitative Survey Phase .............................................................................................. 18
Ethical Considerations......................................................................................................... 20
Literature Cited ................................................................................................................... 21
CHAPTER 2: JOURNAL ARTICLE [for submission to Ecology and Society]........ 30
Abstract ............................................................................................................................... 30
Introduction ........................................................................................................................ 30
Methods .............................................................................................................................. 34
Attendance Record Analysis ........................................................................................... 35
Semi-Structured Interviews ............................................................................................ 35
Online Survey .................................................................................................................. 36
Statistical analyses .......................................................................................................... 37
Results ................................................................................................................................. 38
WW2100 Participant Characterization ........................................................................... 38
Participant Motivations .................................................................................................. 40
Participant Expectations ................................................................................................. 43
Differences between Participant Groups........................................................................ 48
Discussion and Conclusions ................................................................................................ 50
References .......................................................................................................................... 56
TABLE OF CONTENTS
Page
CHAPTER 3: JOURNAL ARTICLE [for submission to The International Journal of
Science in Society] ...................................................................................................... 64
Abstract ............................................................................................................................... 64
Introduction ........................................................................................................................ 64
Methods .............................................................................................................................. 71
Qualitative semi-structured interviews .......................................................................... 71
Quantitative survey......................................................................................................... 73
Results ................................................................................................................................. 74
Researcher-stakeholder engagement process structure................................................ 74
Researcher-stakeholder engagement process challenges.............................................. 78
Research-stakeholder engagement process impacts ..................................................... 83
Discussion ........................................................................................................................... 89
Challenges of Transdisciplinary Projects ......................................................................... 89
Impacts of Stakeholder Engagement in Transdisciplinary Projects ................................ 92
Conclusions ......................................................................................................................... 97
References .......................................................................................................................... 98
CHAPTER 4: RESULTS, DISCUSSION & CONCLUSIONS ............................... 107
Who is participating in the researcher-stakeholder engagement process? ..................... 108
What are participants’ motivations and expectations in participating? .......................... 109
What are participants’ perceptions of the process? ........................................................ 112
Challenges ..................................................................................................................... 112
Successes and Impacts .................................................................................................. 114
Engagement Process Structure to achieve impacts ...................................................... 120
Limitations ........................................................................................................................ 125
Recommendations ............................................................................................................ 126
Conclusions ....................................................................................................................... 127
Literature Cited ................................................................................................................. 128
APPENDICES .......................................................................................................... 136
LIST OF FIGURES
Figure
Page
1.1 Map of the Willamette basin and land use category distribution ………………..12
2.1. Individual participation in LAN events, organized by representative category...39
2.2. Motivations of survey respondents...…………………...……………………….40
2.3. Expectations for stakeholders and research team members and whether they were
met ..……………………………..……………………………………………….47
3.1. Spectra of research approach philosophies…...…………………………………80
4.1. Timeline of stakeholder engagement events and positive and negative perceptions
of these events ….……………………………………………………………..123
LIST OF TABLES
Table
Page
1.1
Lessons learned and impacts from cases of stakeholder engagement in
transdisciplinary research.….……………………………………...……….....9
1.2
Group membership and number of participants interviewed …………..…...17
1.3
Total respondents, total surveys sent, and response rate for each respondent
category.…………..……………………………………………………….....18
2.1
Total respondents, total surveys sent, and response rate for each respondent
category………………………………………………………………………36
2.2
Expectations for the WW2100 model and engagement process and whether or
not they were met...………………………………………………………......46
2.3
Expectations for stakeholder roles by respondent category……………....….48
2.4
Research team and stakeholder group expectations for research
team roles…………………………………………………………………….49
2.5
Belonging to two expectation groups by professional group in WW2100
researcher-stakeholder engagement process….……………………………...50
2.6
Expectations for the researcher-stakeholder engagement process and resulting
model by respondent category…………… …………………………………51
3.1
Lessons learned and impacts from previous cases of stakeholder engagement
in transdisciplinary research…………………………………………………69
3.2
Representation, expertise, and number of participants
interviewed..…………………………………………………………………72
3.3
Summary of researcher-stakeholder engagement formats in
WW2100………………………………………………. ……………………76
3.4
Codebook summary of challenges and successes of WW2100 researcherstakeholder engagement process………… ………………………………….89
3.5
Revised broader impact (BI) framework and examples from WW2100
outcomes………………………………………………………………….….94
LIST OF TABLES (Continued)
4.1
Revised broader impact (BI) framework and examples from WW2100
outcomes………………… ………………………………………………...119
LIST OF APPENDICES
Appendix
Page
A.
Semi-Structured Interview Guide……..…………………………...……….137
B.
Survey Instrument and Results……………………………………………..138
C.
Verbal Consent Guide……………………………………………………...168
D.
Survey Letter of Invitation…………………………………………………169
E.
Participating Research Team University Departments and Organizations…170
F.
Supplemental Survey Results Tables……………………………………….173
G.
Exploratory Factor Analysis of Researcher-Stakeholder Process and Model
Expectations………………………………………………………………...175
H.
Cronbach Reliability Analyses Index Analysis…………………………….176
LIST OF APPENDIX TABLES
E.1.
Page
Participating research team university departments and stakeholder
organizations………………………………………………………………..153
F.1.
Motivations of survey respondents…………………………………………156
F.2.
Expectations for research team member and stakeholder roles…………….156
F.3.
Expectations for stakeholder roles and whether they were met……………157
F.4.
Expectations for research team member roles and whether or not they were
met………………………………………………………………………….157
G.1.
Exploratory factor analysis of researcher-stakeholder engagement process and
model expectations…………………………………………………………158
H.1.
Cronbach alpha reliability analyses for participation indices………………159
H.2.
Cronbach alpha reliability analyses for model utility, process utility, feeling
heard, and model understanding indices……………………………………160
1
CHAPTER ONE: INTRODUCTION, CONTEXT, & METHODS
As the evidence for global climate change grows, communities are preparing for and
adapting to the oncoming impacts on climate and variability. However, climate change and the
associated natural resource management impacts possess the characteristics of a wicked problem
(Rittel & Webber, 1973). There are no immediate or ultimate tests of solutions to the suite of
climate change problems; they can be explained in many ways; each item in the suite can be
considered a symptom of the other; the stakes are high; and there is no shared definition of the
problem being faced (Rittel & Webber, 1973). Furthermore, global climate change implies
global climate impacts. Transboundary natural resources will be impacted in unpredictable ways.
Where shared natural resources were already difficult to manage (Dietz, Ostrom, & Stern, 2003),
natural resource managers are further challenged by the uncertainty of climate change impacts
(Lawler et al., 2010). As a result, systems-approach research may be useful in addressing natural
resource management in the face of climate change.
Scientific studies and associated recommendations to managers and policy makers are
among the current options for presenting evidence for global climate change to the public
(Intergovernmental Panel on Climate Change, 2013). For example, there are several biodiversity
risk assessments that explore how certain species will respond to changes in temperature and
precipitation (e.g. Brainard et al., 2013; McClure et al., 2013). There are also studies which aim
to characterize system responses to unique aspects of climate. For instance, a study by Chang,
Praskievicz, & Parandvash (2014) showed that water consumption in Portland, Oregon was more
tightly correlated with temperature than with precipitation. Yet, the water consumption in
Portland does not exist in isolation. How then can climate change impacts be managed when they
span many of society’s organizational boundaries (Stubbs & Lemon, 2001)?
The ability to incorporate climate change and natural resource research into management
and policies is limited. Several studies have investigated barriers to using climate research to
make decisions and take actions. Reasons managers and policy makers have cited for not
utilizing scientific information include uncertainty, conflicting priorities, institutional limitations,
miscommunication or lack of effective communication, differing values, and the lack of results
suited to local conditions (Callahan, Miles, & Fluharty, 2013; Gregory, Arvai, & Gerber, 2013;
Rayner, Lach, & Ingram, 2005; Smith, Strzepek, Rozaklis, Ellinghouse, & Hallett, 2009; Weible
2
& Sabatier, 2009; Yang, Wu, & Shen, 2013). Additionally, science is not incorporated into
policies because there are scientific studies supporting both sides of a policy debate; each of
which can easily identify limitations in the methods and assumptions of the other (Fuller, 2011).
Science is not used by non-scientists in climate and resource management for many reasons.
On the other hand, funding agencies continue to award grants to scientific research
projects, in part on the basis of whether the project exhibits strong potential for broader impacts,
and resource managers are expected to incorporate science and climate change adaptation into
planning and practices (Halofsky et al., 2011). There is thus a need for research scientists and
non-scientists to work together to achieve these goals. Upon discovering that only 1/3 of research
they funded was used to address coastal management problems, the Cooperative Institute for
Coastal and Estuarine Environmental Technology (CI-CEET) evaluated their projects and found
that user involvement and user trust were key factors present in the cases where funded research
was able to affect change (Riley et al. 2011). In other words, research needs to be credible,
salient, and legitimate (Cash et al., 2003). Research which incorporates the expertise of multiple
disciplines and engages with science users is proposed and explored as a way to produce
credible, salient, and legitimate results.
The purpose of this thesis is to characterize one case of interdisciplinary research which
has engaged with science users throughout the research process. Although there are many
examples which provide lessons learned and outcomes from similar cases, they do not
thoroughly discuss the structure nor represent participants’ perceptions of the researcherstakeholder engagement process. This study characterizes both researcher and stakeholder
experiences and reflections on the process as it pertains to challenges and successes in order to
identify barriers and pathways to a successful process. Where previous cases suggest potential
impacts of their processes based on researcher reflection, this study identifies outcomes based on
researcher and stakeholder reports and places them in the context of the National Science
Foundation broader impacts criteria. The study offered here characterizes the experience of both
researchers and stakeholders in this process by asking three questions:
1)
Who is participating?
2)
What are their motivations and expectations for participating?
3)
What are their perceptions of the process?
3
Broader Impacts
On July 10, 1997 the National Science Foundation (NSF) established two equally
important merit criteria to evaluate proposals for funding: Intellectual Merit and Broader
Impacts. Intellectual Merit refers to a project’s potential to advance knowledge within and across
scientific disciplines, while Broader Impacts refers to a project’s potential to benefit society and
contribute to achieving specific societal outcomes. A task force evaluated these sets of criteria in
2011 and confirmed that they were appropriate, emphasizing the need for broader participation in
science to enhance scientific literacy and benefit society (National Science Board, 2011). As
with all aspects of research to receive government funding through NSF, spending on Broader
Impacts must be reported to congress according to the America COMPETES Reauthorization
Act of 2010 (National Science Foundation, 2012). Broader Impacts, then, are a priority for the
nation in funding scientific research.
The NSF provides guidelines to researchers and reviewers to help them develop and
evaluate Broader Impacts proposal plans. First, researchers must include a separate section in the
proposal outlining the Broader Impacts plan of the proposed research or they will be returned to
them without review. Within this section researchers should describe how the proposed research
may benefit society through specific outcomes (National Science Foundation, 2012). The focus
of Broader Impacts complements the Intellectual Merit criterion of knowledge creation by
focusing on knowledge integration and transfer. Broader Impacts may be accomplished through
the research itself, through activities related to research, or to activities that are complementary
to the project (National Science Board, 2011). According to the guidelines for principal
investigators, examples of Broader Impacts activities include:
“innovations in teaching and training…contributions to the science of
learning…development and/or refinement of research tools; computation
methodologies, and algorithms for problem solving; development of databases to
support research and education; broadening the participation of groups
underrepresented in science, mathematics, engineering and technology; and
service to the scientific and engineering community outside of the individual’s
immediate organization” (National Science Foundation, 2013).
Ultimately, researchers should ensure that their proposed Broader Impacts activities speak to the
questions developed by the National Science Board to guide proposal reviewers (National
Science Board, 2011, p. 4).
4
1) How well does the activity advance the discovery and understanding while
promoting teaching, training, and learning?
2) How well does the proposed activity broaden the participation of
underrepresented groups (e.g., gender, ethnicity, geographic, etc.)?
3) To what extent will it enhance the infrastructure for research and education,
such as facilities, instrumentation, networks, and partnerships?
4) Will the results be disseminated broadly to enhance scientific and
technological understanding?
5) What may be the benefits of the proposed activity to society?
Despite review and updates to the guidelines, the Broader Impacts requirement remains
under scrutiny. The list of examples provided in the guidelines for authors was perceived as a
broad checklist which researchers struggled to fulfill in their focused research proposals
(National Science Board, 2011). The new guidelines have replaced the list of illustrative
examples with the general list cited above, leaving researchers without role models for broader
impacts activities (National Science Foundation, 2012; National Science Foundation, 2013).
Reviewers maintain that it is more difficult to assess the outcomes of Broader Impacts than
Intellectual Merit because they are not as clear or consistent across projects and institutions
(National Science Board, 2011).
Given the assessment inconsistency across institutions and projects, researchers struggle
not only to propose Broader Impacts activities but to execute approved activities. One study of
projects funded by NSF found that 65% had Broader Impacts statements and 19% of those only
included one of five possible broader impact activities. The most popular broader impacts
activities were (a) teaching and training followed by (b) broader dissemination of results with
37% and 22% of funded projects participating in such activities, respectively (Nadkarni &
Stasch, 2013). Traditionally, broader impacts have followed this pattern. Supporting graduate
students and publishing results on websites are familiar, tangibly beneficial to scientists, and
require little effort. They are grounded in typical academic methods of scientific knowledge
distribution. However, “since intellectual merit and broader impacts are now cast as integrated
and interdependent criteria within NSF’s review process, there is some expectation that scientists
and stakeholders are both engaged in the research enterprise and mutually benefit from it”
(Frodeman et al., 2013, p. 153). Researchers realize the importance of reaching out to society and
5
have enjoyed engaging in broader impact activities (Pearson et al., 1997) but struggle to
implement and engage various publics without explicit direction, successful role models, and
consistent assessment criteria.
“The promise of applying systems science as a bridge between hard and soft
systems approaches is realized as all stakeholders join to review technologies,
policies, underlying assumptions, and worldviews and re-assess the main goals
and questions on which policies and practice are based”
(Pahl-wostl, 2007, p. 60).
“Stakeholder engagement has evolved from a marginal concern to a driving
force.” (Lynam, de Jong, Sheil, Kusumanto, & Evans, 2007)
Trends in Collaborative Researcher-Stakeholder Engagement
Scientific research has responded in two ways to the two challenges presented above.
First, to address wicked climate and natural resource management problems, scientific research
is moving towards systems approach studies, integrating multiple disciplines to explore all parts
of a system. Second, in the quest for scientific broader impacts research teams are reaching out to
and engaging with their stakeholders. The two methods combined can be called collaborative
researcher-stakeholder engagement. Cases of collaborative researcher-stakeholder engagement
are reviewed in Ferguson et al. (2015, in preparation).
In climate and water research, transdisciplinary studies and stakeholder engagement
efforts have become more common. Such studies seek to engage stakeholders in climate and
natural resource research and there are myriad ways to do so (Mader, Mader, Zimmermann,
Görsdorf-Lechevin, & Diethart, 2013). Distinct methods are used: forming steering groups,
conducting surveys and/or individual interviews, holding small or large group sessions, one to
two events in one year, or several events over multiple years. Transdisciplinary projects, then,
can be considered participatory research projects. These methods tend to include nonscientists in
science and technology (Lengwiler, 2008) and are characterized by a blurring of the lines
between research and co-generation of knowledge. Those with a stake in the outcome of the
research participate actively in a co-researcher role (Mackenzie, Tan, Hoverman, & Baldwin,
2012). Despite interest in the literature, applications of participatory methods are rare
6
(Kastenhofer, Bechtold, & Wilfing, 2011). Lengwiler (2008) offers a review of historical
approaches to participatory modern science.
Participation in research typically includes “intentional collaborations in which members
of the public engage in the process of research to generate new science-based knowledge (Shirk
et al., 2012, 29). Recently participatory research has been utilized in projects modeling or
planning for future scenarios of climate change. Participatory modeling can be a way to integrate
multiple sources of knowledge and can facilitate social learning (Pahl-wostl, et al., 2007b; Pahlwostl, et al., 2007c). For a review and assessment of the many tools for participatory research,
see Lynam et al. (2007).
Projects like those outlined above can be organized according to the degree of
stakeholder interaction in the project. Stakeholder interaction:
“refers to the degree to which representatives of the constituency base are
involved in aspects of the research: defining the problem, formulating research
questions, selecting methods, conducting research, analyzing findings, developing
usable knowledge, testing/evaluating research results, participating in
dissemination of results, and participating in identifying next research steps”
(Lemos & Morehouse, 2006, p. 61)
Stakeholder engagement in natural resource research can be organized according to when in the
scientific process stakeholders are involved and as a function of openness to stakeholder
participation or as a function of information flow. Stakeholders may be involved in the first
phase of research by directing the questions asked. Shirk et al. (2012) considers this a contractual
relationship between stakeholders and research teams. European science shops are an example of
contractual stakeholder engagement. In the science shop model, stakeholders propose questions
to the shop which then directs the question to the appropriate department of the university. Once
answered, the results are delivered to the stakeholders in a way that they may be directly
implemented (Farkas, 1999). For a review of science shops see Leydesdorff and Ward (2005).
Openness to including stakeholders beyond that initial phase has been classified on several scales
varying from consultation to co-production (Kloprogge & van der Sluijs, 2006) and from
contractual to co-created (Shirk et al., 2012). Information may flow one way in information and
outreach or may flow two ways as in consultation and decision-influencing (Mader et al., 2013).
A review and assessment of the various tools and timings with which to engage stakeholders can
be found in Lynam et al. (2007).
7
One mechanism through which to incorporate stakeholder information into scientific
research from beginning to end is the Learning and Action Network (LAN). LANs link
stakeholders to research teams throughout the investigative process. LANs are not unique to nor
did they originate in the natural resource research sector. Businesses have used LANs for
collaboration and conflict mitigation, defining them as “set[s] of relationships which lay over and
complement formal organizational structures linking individuals together by the flow of
knowledge, information, and ideals” (Clarke & Roome, 1999, p. 297). LAN meetings in a
business context reduced stereotypes held by different actors and as a result of two-way
communication among diverse perspectives led to knowledge creation (Clarke & Roome, 1999).
Moving from business to the health sector, knowledge is produced and applied through social
interaction in a community of practice of academic and non-academic citizens (Li et al., 2009;
Spiegel et al., 2011). Although links are not always apparent to members of a LAN (Clarke &
Roome, 1999), they generate the benefits of knowledge production and understanding.
A new LAN definition is required to extend the concept from the business and health
context to natural resource research and management. Podolny and Page (1998) define a LAN as
“any collection of actors (N ≥ 2) that pursue repeated, enduring exchange relations with one
another and, at the same time, lack a legitimate organizational authority to arbitrate and resolve
disputes that may arise during the exchange” (p. 59). According to this definition, participation
in a LAN is a conscious decision and group-regulated. Forming such a network requires trust in
each other and in each other’s knowledge (Mader et al., 2013). Without an organizational
authority but with mutual trust, all LAN members can participate on equal ground to gain the
benefits associated with LANs. This ideal form of a LAN is not necessarily present in natural
resource research as research teams often facilitate network formation. For instance, one research
group brought together people who do not normally interact to form an air quality adaptive
management network (Stubbs & Lemon, 2001).
Regardless of the way they form, LANs in a natural resource researcher-stakeholder
engagement context facilitate information exchange, knowledge creation, and social learning.
Network formation may have many benefits including: effective climate change communication
and discussion of climate uncertainty (Bartels et al., 2013), knowledge acquisition, result
legitimacy in the eyes of the stakeholders (Podolny & Page, 1998), and increased sensitivity to
diverse stakeholder perspectives (Stubbs & Lemon, 2001). It is difficult to determine whether
8
knowledge created through networks is distinct from knowledge created by each actor alone;
however, there is much evidence of knowledge co-production occurring within LANs. For
instance, a LAN of agricultural producers formed the questions to direct research, which, when
answered, created new knowledge (Bartels et al., 2013). Ideally, LANs foster knowledge
creation by facilitating discussion and knowledge transfer among people of diverse perspectives,
which encourages new interpretations of the shared information (Podolny & Page, 1998).
The creation of LANs as a method for stakeholder engagement in research can produce
benefits that extend beyond the research itself. Forming a LAN can benefit those involved
simply by exposing members to current and future collaborators. Once formed, the network may
continue to share information and collaborate even after the specific project for which the LAN
was formed ends. The LAN facilitates social learning and encourages trust, commitment, and
issue reframing. These can develop into a shared LAN perspective of a common problem and a
shared action to address it (Sol, Beers, & Wals, 2013). From a decision-making perspective,
LANs can lead to collaborative planning where agencies tasked with managing a shared resource
can work together to apply relevant knowledge, establish facts, and set common goals. Action
networks can even influence policy adoption once the collaborative process has finished (van
Herk, Zevenbergen, Ashley, & Rijke, 2011).
Many of the cases discussed above offer lessons learned from their process of engaging
with stakeholders in addition to the impacts of stakeholder engagement on their research. Table
1.1 summarizes these impacts and the associated lessons learned. Although not explicitly stated
in publications, lessons learned sections are the result of project elements or entire engagement
processes going poorly. Each lesson stated here references a moment when that element was not
well-executed which negatively impacted the project. Some key lessons learned from a synthesis
of the literature include establishing clear roles and responsibilities (Lang et al., 2012;
Mackenzie et al., 2012; Voinov & Bousquet, 2010), arriving at a shared understanding of the
research goal early (Halofsky et al., 2011; Holman et al., 2008; Sol et al., 2013), allocating an
adequate amount of resources to the stakeholder engagement process (Halofsky et al., 2011;
Kloprogge & van der Sluijs, 2006; Lemos & Morehouse, 2006; Mackenzie et al., 2012), and
using a facilitator for scientist-stakeholder events (Kloprogge & van der Sluijs, 2006; Mackenzie
et al., 2012; Sol et al., 2013; Voinov & Bousquet, 2010).
9
Table 1.1. Lessons learned and impacts from cases of stakeholder engagement in transdisciplinary research.
Lessons learned
Clear roles and responsibilities
Allocate resources well
Be sensitive to stakeholder needs
Consider relationship to research funders
Focus on process rather than product
Accept uncertainty
Accept external expertise as credible
Engage early
Integrate qualitative and quantitative knowledge
Manage both stakeholder engagement and interdisciplinary portions
Produce non-normative publications
Make use of existing relationships
Necessary elements
Strong leadership
Collaborative research team
Mutual trust
Commitment to project
Transparency
Iterativity
Source
Lang et al., 2012; Mackenzie et al., 2012; Matso & Becker, 2014;
Voinov & Bousquet, 2010
Becu, Neef, Schreinemachers, & Sangkapitux, 2008; Kearney, Berkes,
Charles, & Wiber, 2007; Kloprogge & van der Sluijs, 2006; Lemos &
Morehouse, 2006; Mackenzie et al., 2012; Matso & Becker, 2014
Kloprogge & van der Sluijs, 2006; Lang et al., 2012; Lemos &
Morehouse, 2006; Mackenzie et al., 2012
Mackenzie et al., 2012
Dilling & Lemos, 2011; Kearney et al., 2007; Lautenbach, Berlekamp,
Graf, Seppelt, & Matthies, 2009; Voinov & Bousquet, 2010
Holzkämper, Kumar, Surridge, Paetzold, & Lerner, 2012; Voinov &
Bousquet, 2010
Mackenzie et al., 2012
Holman et al., 2008; Matso & Becker, 2014
Cross, McCarthy, Garfin, Gori, & Enquist, 2013
Daniell et al., 2010; Huntington et al., 2002; Lemos & Morehouse, 2006;
Matso & Becker, 2014
Leydesdorff & Ward, 2005
Huntington et al., 2002
Lemos & Morehouse, 2006; Manring, 2014; Sol et al., 2013
Dilling & Lemos, 2011; Kearney et al., 2007; Lang et al., 2012; Lemos
& Morehouse, 2006; Manring, 2014
Kloprogge & van der Sluijs, 2006; Lemos & Morehouse, 2006;
Mackenzie et al., 2012; Mader et al., 2013; Sol et al., 2013; Voinov &
Bousquet, 2010
Kearney et al., 2007; Sol et al., 2013
Johnson, 2011; Lang et al., 2012; Voinov & Bousquet, 2010
Dilling & Lemos, 2011; Halofsky et al., 2011; Holman et al., 2008; Lang
et al., 2012; Swart, Raskin, & Robinson, 2004; Voinov & Bousquet,
2010
10
Table 1.1. Lessons learned and impacts from previous cases of stakeholder engagement in transdisciplinary research. (Continued)
Untraditional metrics of success
Mid-size, diverse group
Shared reframing of issue/plan/goal
Facilitators/Boundary organizations
Visualizations
Frequent interaction
Impacts
Learn from one another
Improve understanding
Visualize future
Increased credibility
Incorporate managerial knowledge (accurate,
accessible, appropriate research)
Network building
Increase stakeholder self-efficacy
Future research emerges
Diverse dialogue
Increased legitimacy
Increased saliency
Mackenzie et al., 2012; Voinov & Bousquet, 2010
Bartels et al., 2013; Swart et al., 2004; Voinov & Bousquet, 2010
Dewulf, François, Pahl-wostl, & Taillieu, 2007; Fuller, 2011; Halofsky et al., 2011;
Kearney et al., 2007; Lang et al., 2012; Lautenbach et al., 2009; Lemos & Morehouse,
2006; Mackenzie et al., 2012; Matso & Becker, 2014; Sol et al., 2013
Cash et al., 2003; Dilling & Lemos, 2011; Johnson, 2011; Kearney et al., 2007;
Mackenzie et al., 2012; Robinson & Wallington, 2012; Sol et al., 2013
Sheppard et al., 2011
Johnson, 2011; Kloprogge & van der Sluijs, 2006; Lemos & Morehouse, 2006; Mader
et al., 2013
Bartels et al., 2013; Becu et al., 2008; Huntington et al., 2002; Lienert, Monstadt, &
Truffer, 2006; Tim Lynam, Drewry, Higham, & Mitchell, 2010; Manring, 2014;
Stubbs & Lemon, 2001
Becu et al., 2008; Cross, McCarthy, Garfin, Gori, & Enquist, 2013; Lienert et al., 2006
Becu et al., 2008; Lienert et al., 2006
Baker et al., 2004; Cash et al., 2003; Holman et al., 2008; Holzkämper et al., 2012;
Tim Lynam et al., 2010
Baker et al., 2004; Holman et al., 2008; Tim Lynam et al., 2010
Becu et al., 2008; Cross et al., 2013; Holzkämper et al., 2012; Leydesdorff & Ward,
2005; Manring, 2014; Stubbs & Lemon, 2001
Baker et al., 2004; Sheppard et al., 2011
Bartels et al., 2013; Becu et al., 2008; Halofsky et al., 2011
Becu et al., 2008; Cross et al., 2013; Halofsky et al., 2011; Huntington et al., 2002
Cash et al., 2003; Fuller, 2011
Cash et al., 2003
11
Similarly, previous cases identify necessary elements for project success (Table 1.1)
through the experience with failed and successful processes. Some cases posit that boundary
objects, spanners, or managers may serve to facilitate communication across disciplines (Fuller,
2011) and between researchers and stakeholders (Cash et al., 2003; Dilling & Lemos, 2011;
Johnson, 2011; Kearney et al., 2007; Robinson & Wallington, 2012). Successful stakeholder
engagement also requires mutual trust achieved through an iterative process and frequent
interactions (Dilling & Lemos, 2011; Halofsky et al., 2011; Holman et al., 2008; Johnson, 2011;
Kloprogge & van der Sluijs, 2006; Lemos & Morehouse, 2006; Mackenzie et al., 2012; Mader et
al., 2013; Sol et al., 2013; Swart et al., 2004; Voinov & Bousquet, 2010). Participatory research
methods and Learning and Action Networks are methods that can facilitate the researcherstakeholder engagement process.
Defining the Case: Willamette Water 2100 Researcher-Stakeholder Engagement
In the Willamette basin, Oregon, water scarcity is one of the predicted and already
observed consequences of global climate change. Nine years ago Bastasch (2006) stated, “To
make a long story short, Oregon’s out of easy water” (p. vi). For the responsible use of water and
the protection of other water-dependent resources, decision makers must adapt priorities and
create measures to address water scarcity, not only in Oregon, but worldwide.
The Willamette River Basin is situated in west-central Oregon between the east side of
the Coast mountain range and the west side of the Cascade mountain range (Figure 1.1). The
creeks and rivers between these two ranges drain to the Willamette River which discharges
approximately 27 million acre-feet of water annually to the Columbia River. The Willamette
basin accounts for 12% of the state’s area and is home to 70% of the population (Bastasch, 2006)
which is equal to 11,500 square miles and 2.6 million people (Bolte, 2014). As a result, the
Willamette basin is home to Oregon’s major economic enterprises such as agriculture, industry,
and forestry. These characteristics of the Willamette basin make it an important area to study but
a difficult one. It is precisely because of the size and diversity of the Willamette basin that the
modeling platform Envision is used to reflect the interacting processes of the basin in the
research project, Willamette Water 2100 (WW2100).
12
Figure 1.1. Map of the Willamette basin and land use category distribution.
Willamette Water 2100 Model
WW2100 uses the modeling platform, Envision, to predict water availability in the
Willamette basin to the year 2100. The project specifically asks: a) where are climate change and
human activity most likely to create conditions of water scarcity; b) where is water scarcity most
likely to exert the greatest impact on ecosystems and communities; and c) what strategies would
allow communities to prevent, mitigate, or adapt to scarcity most successfully? Researchers at
Oregon State University have partnered with the University of Oregon and Portland State
University in addition to basin-wide stakeholders to answer these questions.
The modeling platform allows for the integration of several sub-models to accurately
characterize the way in which different systems interact within the Willamette basin. The
research team projects water supply in the system by modeling climate and hydrological
processes as well as water demand due to urban, agricultural, forest and ecological needs and
use. All of these are influenced by the overarching drivers of climate change and human
population growth. Each sub-model is accompanied by assumptions describing system processes
within the model. For instance, agricultural and forest water demand is based on water lost to
evapotranspiration which changes as a result of different crop and forest cover types. Likewise,
urban water demand is estimated as a function of water, prices, household income, and
13
population. Fish, or ecological, water demands is based on the need for stream habitats with
appropriate water temperatures. Overall water supply is based on a precipitation dynamic model
which outputs to the hydrology model used by many water modelers nationwide. The resulting
modeled supply is allocated according to the sub-models of reservoir operations, which reflect
the current US Army Corps of Engineers federal operating rule, and current irrigation, municipal,
and instream water rights administered by the state of Oregon. All sub-models feed into the
Envision framework and interact to project a realistic assessment of water availability in the
Willamette basin over the next 85 years.
Willamette Water 2100 Process Structure
The geographic expanse and activity diversity of the Willamette River Basin is reflected
in the diverse team of researchers and stakeholders. The research team, which works directly
with the sub-models and the Envision framework, is composed of 26 principal academic
researchers from three Oregon universities. They represent many disciplines including
hydrology, ecological engineering, landscape architecture, climate science, snow science, applied
economics, environmental engineering, ecohydrology, water resources, geography,
environmental science, fisheries and wildlife, biological engineering, forest ecology, and law.
The research team collaborates with a group of approximately 215 expert citizens representing
state and federal institutions, policymaking groups, non-governmental organizations, and private
industry. Some examples of these citizens’ professions include: council members, water resource
managers, planners, irrigation district managers, reservoir and dam operators, biologists, farmers,
conservationists, and educators. Together with the research team, they form the Learning and
Action Network, WW2100’s information-sharing mechanism for stakeholder engagement and
broader impacts. From the beginning of the project to the end, the focus of the LAN has been on
collaboration and engagement between stakeholders and research team members rather than
traditional outreach.
Willamette Water 2100 offers a unique case to explore the researcher-stakeholder
engagement process and its outcomes after five years of collaboration. As an NSF-funded
project, WW2100 is fulfilling its broader impacts mandate through the formation of the LAN and
associated stakeholder engagement activities. The benefits of LANs in stakeholder engagement
processes are well-documented; however some knowledge gaps remain. First, stakeholder
14
involvement is given little discussion in modeling papers which reach conclusions about a
model’s usability for stakeholders without presenting results (ex. Lautenbach et al., 2009).
Second, there is a need for more structured assessment of factors which contribute to a project
reaching its goal (Lemos & Morehouse, 2006) and of factors of success, including its context and
how it is defined (Huntington et al., 2002). Third, the link between transdisciplinary research
and its impacts remains to be demonstrated (Lang et al., 2012). Finally, research is required to
find how the researcher-stakeholder interaction can be improved (Baker et al., 2004). Yet, it is
not enough for researchers in case studies such as this to reflect and offer lessons learned. If the
goal is public support, legitimacy, and democracy, the stakeholder perspective of the process is
important too (Kloprogge & van der Sluijs, 2006). The study offered here characterizes the
experience of both researchers and stakeholders in this process by asking:
1) Who is participating?
2) What are their motivations and expectations for participating?
3) What are their perceptions of the process?
The goals of this study are to identify the impacts of the WW2100 researcher-stakeholder
engagement process and to identify barriers and pathways to those successes. The first goal
works within the case study framework and focuses on individuals’ experiences, how they are
impacted personally, and if the personal and project goals were achieved. The second goal aims
to isolate transferable lessons from this researcher-stakeholder engagement process so that they
may be applied to similar future projects. While this study sought to understand the researcherstakeholder engagement process, the challenges and value of interdisciplinary research emerged
as well. This research, then, can offer a vision of what is necessary for a successful large-scale
interdisciplinary project and how the research group success is inextricably linked to the success
of the researcher-stakeholder engagement process.
15
Methods
This study takes an exploratory sequential mixed methods approach for an in-depth case
study of the participant experience in WW2100. The case study is an inquiry design utilized in
many fields which allows for an in-depth analysis of one case. The case may be a program, an
event, an activity, a process, or one or more individuals (Creswell, 2003). According to Berg and
Lune (2012), each study must fall within a broader category of events processes, or subjects so
that the presented study represents one case. WW2100 is one case of researcher-stakeholder
engagement in interdisciplinary natural resource research. Case studies also “require multiple
methods and/or sources of data through which [to] create a full and deep examination of the
case” (Berg & Lune, 2012, p. 325). Through multiple methods and by belonging to a broader
group, case studies thoroughly characterize one process to provide insight to other similar
processes.
For this reason an exploratory sequential mixed methods approach was employed to
analyze WW2100. First, attendance records were analyzed to characterize the composition of
participants at each researcher-stakeholder engagement event. Then, semi-structured interviews
led to an exploratory inductive qualitative understanding of the views of key researcherstakeholder engagement participants (Creswell, 2003). This phase required a grounded theory
approach where hypotheses were formed following data collection (Auerbach & Silverstein,
2003; Glaser & Strauss, 2009). A grounded theory approach in this phase led to an understanding
of the process not present in previous studies from those who had the most experience with it. A
quantitative phase numerically characterized the emerging concepts followed the qualitative
phase (Creswell, 2003). After understanding the process through the lens of those who had
participated in the process most, it was beneficial to understand how their perceptions fit with the
population of all participants. By utilizing multiple methods, data types, and data sources this
study increased its reliability and validity (Creswell, 2003) for a robust analysis of the WW2100
researcher-stakeholder engagement process.
Attendance Record Analysis
Researcher-stakeholder engagement event records were analyzed to characterize the
participation of invited individuals throughout the five year process. Individuals were
16
categorized according to the group they represented. The categories included federal, state,
county, city, regional government, non-profit, private industry, farming, utility, consultant,
contractor, intergovernmental group, press, K-12 educators, tribes, watershed council, and
university. Individual participation records informed interviewee selection in the following phase
and gave a quick snapshot of the composition of the WW2100 researcher-stakeholder
engagement events. This basic characterization indicated how participant composition at each
event may have changed over time or if one group participated more than others.
Qualitative Interview Phase
The exploratory inductive qualitative phase consisted of 26 semi-structured interviews
conducted towards the end of the researcher-stakeholder engagement process in a form of “en
route” reflection (Daniell et al., 2010). Interviewees were purposively selected (Patton, 2002)
based on their participation in the process and representativeness of various expertise.
Participants who had attended more events were more likely to be invited for an interview.
Participants with an expertise not yet represented were also invited for interviews. Twelve
participants from the stakeholder group and fourteen participants from the research team agreed
to be interviewed (Table 1.2). The interviewed stakeholders represented federal, state, and
county government organizations, as well as private industry and water utilities. The interviewed
researchers were from all three collaborating universities (Portland State University, Oregon
State University, and University of Oregon) and represented multiple disciplines including
economics, law, landscape architecture, ecology, hydrology, climate science, and biological and
ecological engineering. These interviewees were selected based on their experience with the
project and offered diverse perspectives on the researcher-stakeholder engagement process.
Interviews were conducted between January and March of 2015 (year five of WW2100).
Each interview followed a semi-structured interview guide (Appendix A), beginning with a
conversation about how the participant had been involved in WW2100. All questions on the
interview guide were answered to some degree during each interview but some topics were
developed more thoroughly in some interviews than others due to the organic nature of the semistructured interview and the diversity of research subjects. Each interview lasted on average 55
minutes (range: 26 – 89 minutes) and was conducted in person, via skype, or via telephone as the
interviewee preferred.
17
Table 1.2. Group membership and number of participants interviewed.
Stakeholder
Research Team
1 tribal representative
1 farmer
1 water utility manager
1 state agricultural agency representative
1 state water agency representative
1 private technology industry representative
2 federal reservoir agency representatives
1 federal forest agency representative
1 county government representative
1 irrigation district manager
1 city water agency representative
1 economist
4 Broader Impact Team members
1 landscape architecture researcher
1 lawyer
1 climate scientists
3 hydrologists
1 ecohydrologist
2 biological and ecological engineers
Semi-structured interviews were digitally audio recorded and transcribed using Express
Scribe Transcription software. This procedure ensured consistency among interviews and
allowed for open-coding analysis (McClellan, MacQueen, & Neidig, 2003). One interviewee
declined to be recorded. In this case, extensive notes of the conversation were taken, including
live verbatim transcriptions and shorthand conversation themes. The resulting transcripts were
sent to interviewees as a form of “member checking” (Miles, Huberman, & Saldana, 2014). In
this way, interviewees could edit transcripts for accuracy and on rare occasion request that
segments be disregarded in analysis due to the sensitivity of the subject. The themes of any
omitted segments were present in other portions of the interview, and so were not lost to the
analysis.
Once transcribed, the interviews were analyzed through an open coding process with the
assistance of the computer software MaxQDA. Analysis began with this project’s research
concerns: what is the process like? What are its successes? What are its challenges? What is the
value of a researcher-stakeholder engagement process? With these concerns in mind, the text was
analyzed following Auerbach and Silverstein’s (2003) open coding process moving from text
relevant to the concerns, to identifying repeating ideas, and grouping ideas into themes which
18
eventually lead to a theoretical framework. Themes are categories or topics which descriptively
organize a group of repeating ideas (Auerbach & Silverstein, 2003). A first round of coding led
to categories as a result of the research concerns and a second round of coding offered a
refinement of the first round (Miles, Huberman, & Saldana, 2014). This process led to the
themes presented in the following chapters.
Although there is no objective “right” way to interpret qualitative interview data
(Auerbach & Silverstein, 2003; Miles, Huberman, & Saldana, 2014; Patton, 2002), there are
methods to improve qualitative data validity and reliability. First, the validity of the resulting
concepts was confirmed by two informal rounds of member checking (Creswell, 2003) wherein
the interviewees were presented with the preliminary themes and asked for feedback. Second, the
researcher’s codes were checked against the codebooks of two qualitative researchers to assess
inter-coder reliability (Creswell, 2003; Ryan & Bernard, 2003). Notable overlap between
codebooks indicated reliable themes. Because the coders were also participants in WW2100, this
inter-coder reliability assessment served as another form of member checking validity. Interview
data is open to interpretation; however, member checking and inter-coder reliability support the
validity and reliability of the analysis presented here.
Quantitative Survey Phase
The quantitative survey phase used a census design (Vaske, 2008) to survey all members
of the WW2100 listserv. Subjects were first informed of the survey by a research team leader as
a form of introduction and pre-invitation. Participants in WW2100 were then invited via e-mail
on May 11, 2015, and were reminded of the opportunity by e-mail 10 and 17 days after the initial
invitation. Each e-mail contained a link to a questionnaire administered online using the
Qualtrics software. Of the 281 people invited to participate, 137 responded (45 research team
members, 92 stakeholders), leading to a 49% response rate (Table 1.3).
Table 1.3. Total respondents, total surveys sent, and response rate for each respondent category.
Respondent Category
Research Team
Stakeholder Group
Overall
Total Respondents
45
92
137
Total Surveys Sent
72
209
281
Response Rate
62.5%
44%
49%
19
The online questionnaire was developed based on previous questionnaires administered
by WW2100 and a preliminary content analysis of the semi-structured interviews. These
contributed question themes, informed question design, and structured close-ended questions.
Through the 33-item questionnaire (Appendix B), respondents reported their perceptions in five
sections: professional information (capacity in the project, expertise, geographic location),
perceptions of current and future water use in the Willamette basin, expectations for WW2100,
participation in WW2100, and the outcomes of WW2100. Respondents were grouped into one of
two groups, ‘research team member’ or ‘stakeholder’ based on their responses to the first
question: “In what capacity are you acting in this project?” This acting capacity served as the
independent variable for several of the analyses.
Expectations for research team members to perform various roles in the project,
expectations for stakeholders to perform various roles, expectations for the process and the
model, and how well those expectations were met were taken directly from responses analyzed
as dependent variables. All variables were measured on a scale from 1 “strongly disagree” to 5
“strongly agree,” that they expected members of either group to perform or witnessed them
performing the roles in question.
Several indices were calculated from survey responses and used in further analysis. An
index of overall participation was calculated from four questions regarding participation in
WW2100. A communication participation score was first computed by calculating the mean of
several variables regarding the frequency of participation in various types of project
communication measured on an 8-point scale from 1 “never” to 8 “daily.” This score was then
combined with survey responses for the number of years involved in the project (0-5), number of
events attended (0-9), and the number of webinars attended (0-11). Because these variables were
on different scales, an overall participation index was calculated using the standardized z-scores
of the contributing variables and was used as an independent variable in further analyses.
Similarly, indices for model utility, process utility, feeling heard, and model understanding were
calculated from variables designated in the questionnaire designed to illuminate these concepts.
These indices were calculated from variables on 5-point scales from 1 “strongly disagree” to 5
“strongly agree” and used as dependent variables in further analysis.
Nonparametric statistical tests were used to more conservatively assess significant results
on a small sample size. A Mann Whitney U test compared role expectations between research
20
team members and stakeholders. Point biserial correlation effect sizes were calculated to
evaluate the strength of association among these variables. Wilcoxon Sign Rank tests examined
whether participants had different expectations for research team members and stakeholders, and
to answer whether expectations for research team members and stakeholders were met.
Wilcoxon Sign Rank tests were also used to examine whether participant expectations for the
process and model were met. Cohen’s d effect sizes were calculated in these cases to evaluate the
strength of the relationship among these variables.
Analyses regarding model utility, process utility, feeling heard, model understanding, and
overall participation first began with a Cronbach alpha reliability analyses to ensure that the
indices for these concepts were reliable. A value >.65 indicated that the variables were
measuring the same concept and thus could be combined into one index (Vaske, 2008).
Spearman rho (rs) correlations were used to assess the relationship between overall participation
and model utility, process utility, feeling heard, and model understanding. In all of the analyses,
statistical significance was tested at a .05 level and effect sizes were interpreted according to
Vaske (2008).
Ethical Considerations
Due to participation and interaction with human subjects in this research project, standard
verification protocol was used to ensure approval by the Institutional Review Board (IRB).
Human Research ethical training regarding consent procedure, confidentiality, data collection
and storage was completed prior to data collection with the Collaborative Institutional Training
Initiative (CITI). No vulnerable populations were interviewed or surveyed for this study. Consent
was obtained in one of two ways for this research. Interviewees granted verbal consent after
reviewing the approved IRB consent guide (Appendix C) and survey respondents implicitly
granted consent by submitting the completed questionnaire as outlined in the invitation letter
(Appendix D) and introduction to the questionnaire (Appendix B).
21
Literature Cited
Auerbach, C. F., & Silverstein, L. B. (2003). Qualitative data: An introduction to coding and
analysis. NYU Press.
Baker, J. P., Hulse, D. W., Gregory, S. V, White, D., Van Sickle, J., Berger, P. A., …
Schumaker, N. H. (2004). Alternative Futures for the Willamette River Basin, Oregon.
Ecological Applications, 14(2), 313–324.
Bartels, W. L., Furman, C. A., Diehl, D. C., Royce, F. S., Dourte, D. R., Ortiz, B. V., … Jones, J.
W. (2013). Warming up to climate change: a participatory approach to engaging with
agricultural stakeholders in the Southeast US. Regional Environmental Change, 13, S45–
S55. doi:10.1007/s10113-012-0371-9
Bastasch, R. 2006. The Oregon water handbook. Corvallis, Oregon: Oregon State University.
Becu, N., Neef, A., Schreinemachers, P., & Sangkapitux, C. (2008). Participatory computer
simulation to support collective decision-making: Potential and limits of stakeholder
involvement. Land Use Policy, 25(4), 498–509. doi:10.1016/j.landusepol.2007.11.002
Berg, B. L. & Lune, H. 2012. Qualitative research methods for the social sciences. 8th ed. Upper
Saddle River, NJ: Pearson.
Brainard, R. E., Weijerman, M., Eakin, C. M., McElhany, P., Miller, M. W., Patterson, M., …
Birkeland, C. (2013). Incorporating climate and ocean change into extinction risk
assessments for 82 coral species. Conservation Biology : The Journal of the Society for
Conservation Biology, 27(6), 1169–78. doi:10.1111/cobi.12171
Bolte, J. 2014. About. Retrieved November 29, 2014, from Envision: Integrated Modeling
Platform. http://envision.bioe.orst.edu/About.aspx.
Callahan, B., Miles, E., & Fluharty, D. (2013). Policy Implications of Climate Forecasts for
Water Resources Management in the Pacific Northwest. Policy Sciences, 32(3), 269–293.
Cash, D. W., Clark, W. C., Alcock, F., Dickson, N. M., Eckley, N., Guston, D. H., … Mitchell,
R. B. (2003). Knowledge systems for sustainable development. Proceedings of the National
Academy of Sciences of the United States of America, 100(14), 8086–8091.
doi:10.1073/pnas.1231332100
Chang, H., Praskievicz, S., & Parandvash, H. (2014). Sensitivity of Urban Water Consumption to
Weather and Climate Variability at Multiple Temporal Scales : The Case of Portland ,
Oregon. International Journal of Geospatial and Environmental Research, 1(1), Article 7.
Clarke, S., & Roome, N. (1999). Sustainable business: Learning - action networks as
organizational assets. Business Strategy and the Environment, 8, 296–310.
22
Creswell, J. W. 2003. Research design: Qualitative, quantitative, and mixed-methods
approaches. Thousand Oaks, CA: Sage.
Cohen, S. J. (2010). From observer to extension agent—using research experiences to enable
proactive response to climate change. Climatic Change, 100(1), 131–135.
doi:10.1007/s10584-010-9811-z
Cross, M. S., McCarthy, P. D., Garfin, G., Gori, D., & Enquist, C. (2013). Accelerating
adaptation of natural resource management to address climate change. Conservation
Biology : The Journal of the Society for Conservation Biology, 27(1), 4–13.
doi:10.1111/j.1523-1739.2012.01954.x
Daniell, K. A., White, I., Ferrand, N., Ribarova, I. S., Coad, P., Rougier, J. E., … Burn, S.
(2010). Co-engineering Participatory Water Management Processes : Theory and insights
from Australian and Bulgarian interventions. Ecology and Society, 15(4), 11. Retrieved
from http://www.ecologyandsociety.org/vol15/iss4/art11/
Dewulf, A., François, G., Pahl-wostl, C., & Taillieu, T. (2007). A framing approach to crossdisciplinary research collaboration: Experiences from a large-scale research project on
adaptive water management. Ecology and Society, 12(2), 14.
Dietz, T., Ostrom, E., & Stern, P. C. (2003). The struggle to govern the commons. Science (New
York, N.Y.), 302(5652), 1907–12. doi:10.1126/science.1091015
Dilling, L., & Lemos, M. C. (2011). Creating usable science: Opportunities and constraints for
climate knowledge use and their implications for science policy. Global Environmental
Change, 21(2), 680–689. doi:10.1016/j.gloenvcha.2010.11.006
Eccles, J. S., & Wigfield, A. (2002). Motivational beliefs, values, and goals. Annual Review of
Psychology, 53, 109 – 132.
Farkas, N. (1999). Dutch Science Shops: Matching community needs with university R & D.
Science Studies, 2, 33–47.
Ferguson, L. F. (2015, in preparation). Collaborative Science-Stakeholder Engagement.
Unpublished annotated bibliography.
Freitag, A. (2014). Naming, framing, and blaming: Exploring ways of knowing in the
deceptively simple wuestion “What is water quality?” Human Ecology, 42, 325–337.
doi:10.1007/s10745-014-9649-5
Frodeman, R., Holbrook, J. B., Bourexis, P. S., Cook, S. B., Diederick, L., & Tankersley, R. A.
(2013). Broader Impacts 2.0: Seeing - and Seizing—the Opportunity. Bioscience, 63(3),
153–155. doi:10.1525/bio.2013.63.3.2
23
Fuller, B. (2011). Enabling problem-solving between science and politics in water conflicts:
impasses and breakthroughs in the Everglades, Florida, USA. Hydrological Sciences
Journal, 56(4), 576–587. doi:10.1080/02626667.2011.579075
Glaser, B. G. & Strauss, A. L. 2009. The discovery of grounded theory: Strategies for qualitative
research. Transaction Publishers.
Gregory, R., Arvai, J., & Gerber, L. R. (2013). Structuring decisions for managing threatened
and endangered species in a changing climate. Conservation Biology : The Journal of the
Society for Conservation Biology, 27(6), 1212–21. doi:10.1111/cobi.12165
Grin, J., & van de Graaf, H. (1996). Technology assessment as learning. Science, Technology &
Human Values, 21(1), 72–99.
Halofsky, J. E., Peterson, D. L., Furniss, M. J., Joyce, L. A., Millar, C. I., & Neilson, R. P.
(2011). Workshop approach for developing change adaptation strategies and actions for
natural resource management agencies in the United States. Journal of Forestry, (June),
219–225.
Hansen, J. A., & Lehmann, M. (2006). Agents of change: universities as development hubs.
Journal of Cleaner Production, 14(9-11), 820–829. doi:10.1016/j.jclepro.2005.11.048
Hildén, M. (2011). The evolution of climate policies – the role of learning and evaluations.
Journal of Cleaner Production, 19(16), 1798–1811. doi:10.1016/j.jclepro.2011.05.004
Holman, I. P., Rounsevell, M. D. A., Cojacaru, G., Shackley, S., McLachlan, C., Audsley, E., …
Richards, J. A. (2008). The concepts and development of a participatory regional integrated
assessment tool. Climatic Change, 90(1-2), 5–30. doi:10.1007/s10584-008-9453-6
Holzkämper, A., Kumar, V., Surridge, B. W. J., Paetzold, A., & Lerner, D. N. (2012). Bringing
diverse knowledge sources together - a meta-model for supporting integrated catchment
management. Journal of Environmental Management, 96(1), 116–27.
doi:10.1016/j.jenvman.2011.10.016
Huntington, H. P., Brown-schwalenberg, P. K., Frost, K. J., Fernandez-gimenez, M. E., Norton,
D. W., & Rosenberg, D. H. (2002). Observations on the workshop as a means of improving
communication between holders of traditional and scientific knowledge. Environmental
Management, 30(6), 778–792. doi:10.1007/s00267-002-2749-9
Intergovernmental Panel on Climate Change. 2013. Working group 1, Summary for
policymakers. Available at http://www.climatechange2013.org/images/upoad/WGIAR5SPM_Approved27Sep2013.pdf.
Johnson, T. R. (2011). Fishermen, scientists, and boundary spanners: Cooperative research in the
U.S. Illex squid fishery. Society & Natural Resources: An International Journal, 24(3),
242–255. doi:10.1080/08941920802545800
24
Kastenhofer, K., Bechtold, U., & Wilfing, H. (2011). Sustaining sustainability science: The role
of established inter-disciplines. Ecological Economics, 70(4), 835–843.
doi:10.1016/j.ecolecon.2010.12.008
Kearney, J., Berkes, F., Charles, A., & Wiber, M. (2007). The role of participatory governance
and community-based management in integrated coastal and ocean management in Canada.
Coastal Management, 35(1), 79–104. doi:10.1080/10.1080/08920750600970511
Kloprogge, P., & van der Sluijs, J. P. (2006). The inclusion of stakeholder knowledge and
perspectives in integrated assessment of climate change. Climatic Change, 75, 359–389.
doi:10.1007/s10584-006-0362-2
Landry, R., Amara, N., & Lamari, M. (2001). Utilization of social science research knowledge in
Canada. Research Policy, 30, 333–349.
Lang, D. J., Wiek, A., Bergmann, M., Stauffacher, M., Martens, P., Moll, P., … Thomas, C. J.
(2012). Transdisciplinary research in sustainability science: practice, principles, and
challenges. Sustainability Science, 7(Supplement 1), 25–43. doi:10.1007/s11625-011-0149x
Lautenbach, S., Berlekamp, J., Graf, N., Seppelt, R., & Matthies, M. (2009). Scenario analysis
and management options for sustainable river basin managementt: Application of the Elbe
DSS. Environmental Modelling & Software, 24, 26–43. doi:10.1016/j.envsoft.2008.05.001
Lawler, J. J., Tear, T. H., Pyke, C., Shaw, M. R., Gonzalez, P., Kareiva, P., … Pearsall, S.
(2010). Resource management in a changing and uncertain climate. Frontiers in Ecology
and the Environment, 8(1), 35–43. doi:10.1890/070146
Lemos, M. C., & Morehouse, B. J. (2006). The co-production of science and policy in integrated
climate assessments. Global Environmental Change, 15(2005), 57–68.
doi:10.1016/j.gloenvcha.2004.09.004
Lengwiler, M. (2008). Participatory approaches in science and technology: Historical origins and
current practices in critical perspective. Science, Technology & Human Values, 33(2), 186–
200.
Lester, S. E., McLeod, K. L., Tallis, H., Ruckelshaus, M., Halpern, B. S., Levin, P. S., …
Parrish, J. K. (2010). Science in support of ecosystem-based management for the US West
Coast and beyond. Biological Conservation, 143(3), 576–587.
doi:10.1016/j.biocon.2009.11.021
Leydesdorff, L., & Ward, J. (2005). Science shops: a kaleidoschope of science-society
collaborations in Europe. Public Understanding of Science.
doi:10.1177/0963662505056612
25
Li, L. C., Grimshaw, J. M., Nielsen, C., Judd, M., Coyte, P. C., & Graham, I. D. (2009). Use of
communities of practice in business and health care sectors: A systematic review.
Implementation Science, 4(27), 1–9. doi:10.1186/1748-5908-4-27
Lienert, J., Monstadt, J., & Truffer, B. (2006). Future scenarios for a sustainable water sector : A
case study from Switzerland. Environmental Science & Technology, 40(2), 436–442.
doi:10.1021/es0514139
Lynam, T., de Jong, W., Sheil, D., Kusumanto, T., & Evans, K. (2007). A review of tools for
incorporating community knowledge, preferences, and values into decision making in
natural resources management. Ecology and Society, 12(1).
Lynam, T., Drewry, J., Higham, W., & Mitchell, C. (2010). Adaptive modelling for adaptive
water quality management in the Great Barrier Reef region, Australia. Environmental
Modelling and Software, 25(11), 1291–1301. doi:10.1016/j.envsoft.2009.09.013
Mackenzie, J., Tan, P. L., Hoverman, S., & Baldwin, C. (2012). The value and limitations of
Participatory Action Research methodology. Journal of Hydrology, 474, 11–21.
doi:10.1016/j.jhydrol.2012.09.008
Mader, M., Mader, C., Zimmermann, F. M., Görsdorf-Lechevin, E., & Diethart, M. (2013).
Monitoring networking between higher education institutions and regional actors. Journal
of Cleaner Production, 49, 105–113. doi:10.1016/j.jclepro.2012.07.046
Mahmoud, M., Liu, Y., Hartmann, H., Stewart, S., Wagener, T., Semmens, D., … Winter, L.
(2009). A formal framework for scenario development in support of environmental
decision-making. Environmental Modelling and Software, 24(7), 798–808.
doi:10.1016/j.envsoft.2008.11.010
Manring, S. L. (2014). The role of universities in developing interdisciplinary action research
collaborations to understand and manage resilient social-ecological systems. Journal of
Cleaner Production, 64, 125–135. doi:10.1016/j.jclepro.2013.07.010
Martin-Sempere, M. J., Garzon-Garcia, B., & Rey-Rocha, J. (2008). Scientists’ motivation to
communicate science and technology to the public: surveying participants at the Madrid
Science Fair. Public Understanding of Science, 17(3), 349–367.
doi:10.1177/0963662506067660
Matso, K. E., & Becker, M. L. (2014). What can funders do to better link science with decisions?
Case studies of coastal communities and climate change. Environmental Management,
54(6), 1356–71. doi:10.1007/s00267-014-0347-2
McClellan, E., MacQueen, K. M., & Neidig, J. L. 2003. Beyond the qualitative interview: Data
preparation and transcription. Field Methods, 15(1): 63 - 84.
26
McClure, M. M., Alexander, M., Borggaard, D., Boughton, D., Crozier, L., Griffis, R., … Van
Houtan, K. (2013). Incorporating climate science in applications of the US endangered
species act for aquatic species. Conservation Biology : The Journal of the Society for
Conservation Biology, 27(6), 1222–33. doi:10.1111/cobi.12166
Miles, M. B., Huberman, A.M., & Saldana, J. (2014). Qualitative data analysis: A methods
sourcebook (3rd ed.). Thousand Oaks: Sage Publications.
Nadkarni, N. M., & Stasch, A. E. (2013). How broad are our broader impacts? An analysis of the
National Science Foundation’s Ecosystem Studies Program and the Broader Impacts.
Frontiers in Ecology and the Environment2, 11(1), 13–19. doi:10.1890/110106
National Science Board. (2011). Merit Review Criteria. Review and Revisions. Retrieved from
papers2://publication/uuid/910994F5-3EE1-4236-8346-5602486DA1D2
National Science Foundation. (2012). Proposal and award policies and procedures guide.
Retrieved from http://www.nsf.gov/pubs/policydocs/pappguide/nsf13001/nsf13_1.pdf
Pahl-wostl, C. (2007). Transitions towards adaptive management of water facing climate and
global change. Water Resources Management, 21, 49–62. doi:10.1007/s11269-006-9040-4
Pahl-wostl, C., Craps, M., Dewulf, A., Mostert, E., Tabara, D., & Taillieu, T. (2007). Social
learning and water resources management. Ecology and Society, 12(2), 5.
Pahl-wostl, C., Sendzimir, J., Jeffrey, P., Aerts, J., Berkamp, G., & Cross, K. (2007). Managing
change toward adaptive water management through social learning. Ecology and Society,
12(2), 30.
Patton, M. Q. (2002). Qualitative research and evaluation methods. 3rd edition. Thousand Oaks,
CA: Sage.
Pearson, G., Pearson, G., Pringle, S. M., Pringle, S. M., Thomas, J. N., & Thomas, J. N. (1997).
Scientists and the public understanding of science. Science, 6, 279–289.
Podolny, J. M., & Page, K. L. (1998). Network forms of organization. Annual Review of
Sociology, 24, 57–76.
Raddick, M. J., Bracey, G., Gay, P. L., Lintott, C. J., Murray, P., Schawinski, K., …
Vandenberg, J. (2010). Galaxy Zoo: Exploring the motivations of citizen science
volunteers. Astronomy Education Review, 9, 15. doi:10.3847/AER2009036
Rayner, S., Lach, D., & Ingram, H. (2005). Weather forecasts are for wimps*: Why water
resource managers do not use climate forecasts. Climatic Change, 69, 197–227.
Reed, M. S., Evely, A. C., Cundill, G., Fazey, I., Glass, J., & Laing, A. (2010). What is social
learning? Ecology and Society.
27
Riley, C., Matso, K., Leonard, D., Stadler, J., Trueblood, D., & Langan, R. (2011). How research
funding organizations can increase application of science to decision-making. Coastal
Management, 39(3), 336–350. doi:10.1080/08920753.2011.566117
Rittel, H. W. J., & Webber, M. M. (1973). Dilemmas in a general theory of planning. Policy
Sciences, 4(2), 155–169.
Robinson, C. J., & Wallington, T. J. (2012). Boundary work : Engaging knowledge systems in
co-management of feral animals on indigenous lands. Ecology and Society, 17(2), 16.
Rotman, D., Preece, J., Hammock, J., Procita, K., Hanse, D., Parr, C., … Jacobs, D. (2012).
Dynamic changes in motivation in collaborative citizen-science projects. In Session: Civic
and Community Engagement (pp. 217–226). doi:10.1145/2145204.2145238
Ryan, G. W. & Bernard, H. R. (2003). Techniques to identify themes. Field Methods, 15(1): 85 109.
Sheppard, S. R. J., Shaw, A., Flanders, D., Burch, S., Wiek, A., Carmichael, J., … Cohen, S.
(2011). Future visioning of local climate change: A framework for community engagement
and planning with scenarios and visualisation. Futures, 43(4), 400–412.
doi:10.1016/j.futures.2011.01.009
Shirk, J. L., Ballard, H. L., Wilderman, C. C., Phillips, T., Wiggins, A., Jordan, R., … Bonney,
R. (2012). Public participation in scientific research : A framework for deliberate design.
Ecology and Society, 17(2), 29. doi:10.5751/ES-04705-170229
Smith, J. B., Strzepek, K., Rozaklis, L., Ellinghouse, C., & Hallett, K. (2009). The Potential
Consequences of Climate Change for Boulder Colorado's Water Supplies.
Snover, A. K., Mantua, N. J., Littell, J. S., Alexander, M. A., McClure, M. M., & Nye, J. (2013).
Choosing and using climate-change scenarios for ecological-impact assessments and
conservation decisions. Conservation Biology: The Journal of the Society for Conservation
Biology, 27(6), 1147–57. doi:10.1111/cobi.12163
Sol, J., Beers, P. J., & Wals, A. E. J. (2013). Social learning in regional innovation networks:
trust, commitment and reframing as emergent properties of interaction. Journal of Cleaner
Production, 49, 35–43. doi:10.1016/j.jclepro.2012.07.041
Spiegel, J. M., Breilh, J., Beltran, E., Parra, J., Solis, F., Yassi, A., … Parkes, M. (2011).
Establishing a community of practice of researchers, practitioners, policy-makers and
communities to sustainably manage environmental health risks in Ecuador. BMC
International Health and Human Rights, 11 Suppl 2(Suppl 2), S5. doi:10.1186/1472-698X11-S2-S5
28
Star, S. L., & Griesemer, J. R. (1989). Institutional ecology, 'translations’ and boundary objects:
amateurs and professionals in Berkeley’s Museum of Vertebrate Zoology, 1907-39. Social
Studies of Science, 19(3), 387–420.
Stubbs, M., & Lemon, M. (2001). Learning to network and networking to learn: Facilitating the
process of adaptive management in a local response to the UK’s National Air Quality
Strategy. Environmental Management, 27(8), 321–334. doi:10.1007/s002670010152
Swart, R. J., Raskin, P., & Robinson, J. (2004). The problem of the future: sustainability science
and scenario analysis. Global Environmental Change, 14(2), 137–146.
doi:10.1016/j.gloenvcha.2003.10.002
Tuler, S. (1998). Learning through participation. Human Ecology Review, 5(1), 58–60.
Tullos, D., Brown, P. H., Kibler, K., Magee, D., Tilt, B., & Wolf, A. T. (2010). Perspectives on
the salience and magnitude of dam impacts for hydro development scenarios in China.
Water Alternatives, 3(2), 71–90.
Turnhout, E., Stuiver, M., Klostermann, J., Harms, B., & Leeuwis, C. (2013). New roles of
science in society: Different repertoires of knowledge brokering. Science and Public Policy,
40, 354–365. doi:10.1093/scipol/scs114
Van Herk, S., Zevenbergen, C., Ashley, R., & Rijke, J. (2011). Learning and Action Alliances
for the integration of flood risk management into urban planning: a new framework from
empirical evidence from The Netherlands. Environmental Science & Policy, 14(5), 543–
554. doi:10.1016/j.envsci.2011.04.006
Vaske, J. V. (2008). Surveey research and analysis: Applications in parks, recreation and human
dimensions. State College: PA: Venture.
Voinov, A., & Bousquet, F. (2010). Modelling with stakeholders. Environmental Modelling &
Software, 25(11), 1268–1281. doi:10.1016/j.envsoft.2010.03.007
Webler, T. (1998). Beyond Science: Deliberation and analysis in public decision making. Human
Ecology Review, 5(1), 61–62.
Weible, C. M., & Sabatier, P. A. (2009). Coalitions, science, and belief change: Comparing
adversarial and collaborative policy subsystems. Policy Studies Journal, 37(2), 195–212.
doi:10.1111/j.1541-0072.2009.00310.x
Wolf, B., Lindenthal, T., Szerencsits, M., Holbrook, J. B., & Heb, J. (2013). Evaluating research
beyond scientific impact: How to include criteria for productive interactions and impact on
practice and society. GAIA, 22(2), 104–114.
29
Yang, L., Wu, J., & Shen, P. (2013). Roles of science in institutional changes: The case of
desertification control in China. Environmental Science & Policy, 27(37), 32–54.
doi:10.1016/j.envsci.2012.10.017
30
CHAPTER 2: JOURNAL ARTICLE [for submission to Ecology and Society]
Willamette Water 2100: Exploring participant motivations and expectations in a
researcher-stakeholder engagement process
Abstract
Many barriers impede managers and policy makers from incorporating the ‘best available
science’ into decisions. Researcher-stakeholder engagement through Learning and Action
Networks (LANs) is one way to help overcome such cultural, institutional, and practical barriers.
In Willamette Water 2100 (WW2100), scientists and stakeholders studied biophysical and
socioeconomic drivers of future water scarcity in the Willamette basin to identify ways to
anticipate and respond to it. This study explores the participation, motivations, and expectations
of research team members and stakeholders in the WW2100 researcher-stakeholder engagement
process for water sustainability. Attendance records identify the participant composition at each
researcher-stakeholder engagement event while twenty-six key participant semi-structured
interviews and 137 completed online questionnaires illuminate their perceptions of that process.
Qualitative and quantitative analyses demonstrate that participation changes over time and that
participants are motivated to attend for social (ex. knowing other participants), knowledge (ex.
interest in the topic), and utility (ex. useful management tool) reasons. Nonparametric statistical
analyses show that research team members and stakeholders had similar expectations for the
roles each other would play but different expectations for the process and resulting model. For
instance, all participants expected research team members to interpret model outputs and
stakeholders to provide a ‘boots on the ground’ perspective but only researchers expected the
process to provide career experience. In most cases, role, model, and process expectations were
fulfilled though not always to the degree expected.
Keywords: broader impacts, expectations, task value, climate change, modeling
Introduction
Natural resource management and policy is ideally informed by the best available
science. Cultural, institutional, and practical barriers can impede the use of science in natural
resource use decisions. These include uncertainty in the results, conflicting priorities,
institutional limitations, miscommunication or lack of effective communication, differing values,
31
and the lack of results suited to local conditions (Callahan et al., 2013; Gregory et al., 2013;
Rayner et al., 2005; Riley et al., 2011; Smith et al., 2009; Weible & Sabatier, 2009; Yang et al.,
2013). Freitag (2014) found that university scientists, fishermen, and managers possess different
kinds of knowledge, which leads them to define water quality and frame its major issues, causes,
and solutions differently. For example, fishermen viewed sewage as one of the biggest issues
diminishing the “fishable, swimmable” quality of the water while scientists considered
sedimentation and hypoxia as the biggest issues for the chemical and biological welfare of the
water (Freitag, 2014). Similarly, hydropower government officials and academics expressed
different perspectives on the biophysical, socio-economic, and geopolitical impacts of dams
(Tullos et al., 2010). Differences between scientists and their stakeholders may exacerbate the
cultural, institutional, and practical barriers which impede the successful incorporation of
scientific research into management and policy decisions.
To overcome such differences and accompanying barriers, the National Science
Foundation mandates a broader impacts component to every research project that it funds
(National Science Foundation, 2012). Yet, a review in 2013 found that only 65% of NSF-funded
projects had broader impacts plans and most of these were poorly developed (Nadkarni &
Stasch, 2013). Some researchers do not know how best to engage with their stakeholders to
extend the impacts of their results beyond academia.
Researchers have followed NSF’s mandate down many paths, particularly in the case of
climate change, variability, and evaluation of alternative scenarios (Snover et al., 2013).
Stakeholders may consult with researchers in a decision making context, where the goal of the
project is to address an immediate need, create a specific tool, or adaptively manage for future
climate change (Cross et al., 2013; Holzkämper et al., 2012; Mackenzie et al., 2012).
Stakeholders may also consult in transdisciplinary research where they help direct and set
research agendas (Lienert et al., 2006). In sustainability research, multiple disciplines and
stakeholders network to find solutions as required for managing transboundary resources (Mader
et al., 2013; Stubbs & Lemon, 2001). In alternative futures research, researchers and stakeholders
collaborate to model and write scenarios that facilitate discussion and visualization of the future
(Sheppard et al., 2011). For a general review of modeling and alternative futures research, see
Voinov and Bousquet (2010). Many different methods can be used to engage with stakeholders,
32
such as forming steering groups, conducting surveys or interviews with individuals, small
groups, large groups, two hour workshops, or five year processes.
In order to identify effective strategies for broader impacts and stakeholder engagement,
it is important to consider the people involved in these projects. An initial assessment for
researcher-stakeholder engagement projects must first identify who is participating in the
research and who might be affected by the research (Wolf, Lindenthal, Szerencsits, Holbrook, &
Heb, 2013). However, such an assessment must extend beyond simple identification to consider
the roles and responsibilities of all participants (Tuler, 1998), their motivations, and their
expectations. Understanding the various roles, motivations, and expectations of participants may
be important to improve conflict mitigation in transdisciplinary research (Lang et al., 2012).
Recent articles have explored research team and citizen motivations for participating in
collaborative research. Researcher-stakeholder engagement participation motivations can be
informed by prior analyses of citizen science campaign participation motivations. A study on
scientists and citizens engaging in citizen science research identified four main participant
motivations: egoism, collectivism, altruism, and principalism (Rotman et al., 2012). Egoism
refers to personal gains though working together such as enhancing research for scientists and
learning about new ideas for citizens. Collectivism refers to a mutual benefit that each party
receives through collaboration. Altruism refers to a sense of giving something to the other party;
scientists see themselves as educating the public and the citizens believe they are benefitting the
scientists. Principalism refers to the internal belief that citizen-science engagement is worthwhile
on the principal that science should be accessible to everyone both within and beyond academia
(Rotman et al., 2012).
These broad categories which can define either group’s motivations begin to distinguish
themselves when examining either scientists or citizens and their unique motivations which
compose the broad categories. A survey of scientists in Madrid found that scientists’ greatest
motivations for engaging with the public was to increase interest and enthusiasm for science and
appreciation of scientists. Well-established scientists expressed a sense of duty in communicating
their findings to a larger audience while young scientists were motivated by personal satisfaction
and enjoyment of outreach and engagement events (Martin-Sempere, Garzon-Garcia, & ReyRocha, 2008). Latour and Woolgar (1979) asserted that scientists are ultimately motivated by,
and seek to enhance, their credibility. On the other hand, citizens who participate in citizen
33
science campaigns are motivated by the opportunity to contribute to research, to learn about a
topic of interest, to aid in new discoveries, and to find community with others (Raddick et al.,
2010). Still more motivations lead citizens to participate including enjoyment of the topic for its
beauty, fun, or marvel, or a basic desire to help because of an interest in science (Raddick et al.,
2010). Studies on motivations to participate in citizen-science engagement projects can guide an
exploration of participant motivations in researcher-stakeholder engagement processes.
Likewise, the expectancy-value theory can guide an exploration of participant
expectations in researcher-stakeholder engagement processes and how they may be related to a
process’s perceived value. According to this theory, an individual’s expectations and values
directly impact performance in a given task and are influenced indirectly by other peoples’
attitudes and expectations for him or her (Eccles & Wigfield, 2002). Eccles & Wigfield (2002)
identify four components of a task’s value: attainment, intrinsic, utility, and cost. Whether a
person values a task he or she performed is based on if the task was performed well, if he or she
enjoyed doing the task, how the task related to current and future goals, and the negative aspects
of engaging in the task. The concepts of expectations and values can be applied to understanding
an individual’s perception of stakeholder engagement through a researcher-stakeholder
engagement LAN.
Although it is important to identify motivations and expectations of individuals
participating in a researcher-stakeholder engagement process, it is equally important to identify
any differences among them. Attitude differences among scientists, managers, and government
officials are well documented in comparable natural resource management studies (ex. Riley et
al., 2011). Interviews with university scientists, fishermen and managers confirmed that each
group possessed different kinds of knowledge (academic in the scientific community,
experiential in the fisherman community, political in management community) and that these
differences led to different frames for water quality issues and potential solutions (Freitag, 2014).
In a wind turbine development project in Denmark, policy makers, technology developers, and
industry representatives defined problems differently, preferred different solutions, and held
different value systems leading them to arrive at different conclusions (Grin & van de Graaf,
1996). Differing experiences prior to the engagement process may manifest as different
motivations, expectations, and the meeting of those expectations among participants. Identifying
34
these cultural and political differences can provide clues on how to build a thriving
transdisciplinary project (Farkas, 1999).
This study investigates research team member and stakeholder expectations for and
values derived from participating in a researcher-stakeholder engagement process. The
occurrence of such processes is increasing but little is known about the people involved, their
motivations to participation, and their expectations of the researcher-stakeholder engagement
process. Egoism, relativism, altruism, and principalism are the reasons people participate in
similar citizen science campaigns (Rotman et al., 2012). In learning environments, expectations
play a key role in the value individuals ascribe to a task (Eccles & Wigfield, 2002) and in other
natural resource contexts research team members and stakeholders expressed different
perceptions of the same experience (ex. Grin & van de Graaf, 1996). To better understand the
individuals participating in transdisciplinary researcher-stakeholder engagement processes, this
study asks five questions of one case: (a) who is participating in the researcher-stakeholder
engagement process; (b) What are participants’ motivations for attending the researcherstakeholder engagement process; (c) What are participants’ expectations for the process; (d) were
those expectations met; (e) How do motivations and expectations differ between participant
groups?
Methods
Willamette Water 2100 (WW2100) provides a case study to inform current and future
researcher-stakeholder engagement processes for broader impacts to incorporate science into
management and policy decisions. WW2100 is a five-year collaborative effort across three
universities, 12 academic disciplines, and numerous state, federal, and private agencies. This
project seeks to model where and when water scarcity will occur in the Willamette basin through
the year 2100 as a result of climate change and human land and water use decisions. Over the
course of the study, academic researchers and expert water stakeholders have worked together as
part of the project’s broader impacts plan to model various scenarios exploring the future of
water in the Willamette basin. WW2100 is one of many water and climate interdisciplinary
collaborative research efforts with stakeholder engagement. Stakeholders participating in
WW2100 have contributed to output metrics of the model, assessing model assumptions, and
future scenarios. Together with the research team, they formed the WW2100 Learning and
35
Action Network (LAN) which emphasized scientific engagement between the two groups from
the beginning of the project to the end.
This study takes an exploratory sequential mixed methods approach (Creswell, 2003) for
an in-depth case study of the participants of WW2100. Attendance records were analyzed to
characterize the composition of the participants at each researcher-stakeholder engagement
event. Semi-structured interviews of key participants provided qualitative data regarding
motivations and expectations and also influenced the design of the quantitative survey which was
administered to all members of the researcher-stakeholder engagement listserv. Using multiple
methods, data types, and data sources allowed for a robust analysis of the researcher-stakeholder
engagement process by increasing reliability and validity (Creswell, 2003).
Attendance Record Analysis
Researcher-stakeholder engagement event records were analyzed to characterize the
participation of invited individuals throughout the five year process. Individuals were
categorized according to the group they represented. The categories included federal, state,
county, city, regional government, non-profit, private industry, farming, utility, consultant,
contractor, intergovernmental group, press, K-12 educators, tribes, watershed council, and
university. Participants were assigned as research team members when affiliated with one of the
three participating universities or as stakeholders in all other cases. A comprehensive list of all
participating stakeholder organizations and research team university departments can be found in
Appendix E, table E.1. Individual participation records informed the selection of interviewees in
the following phase and gave a quick snapshot of the makeup of the WW2100 researcherstakeholder engagement events. This basic characterization can indicate how participant
composition at each event may change over time or if one group participates more than others.
Semi-Structured Interviews
Semi-structured retrospective interviews regarding expectations, motivations, outcomes,
and general reflections on the researcher-stakeholder engagement process were conducted.
Interviewees were purposively selected (Patton, 2002) based on their participation in the process
and representativeness of the various expertise. Fourteen research team members representing all
36
universities and disciplines and twelve stakeholders representing many agencies and interests
were interviewed. Each interview lasted between 26 and 89 minutes (average: 55 minutes) and
were conducted in person, via skype, or via telephone according to the interviewee’s preference.
Interviews were digitally audio recorded, transcribed using Express Scribe Transcription
software, and sent to the interviewees as a form of “member checking” (Miles, Huberman, &
Saldana, 2014).
Interview transcripts were analyzed through an open coding process (Auerbach &
Silverstein, 2003) with the aid of the computer software MaxQDA. Transcripts were read with
the research concerns (roles, motivations, expectations, outcomes) in mind and following a
refinement of the first round of coding (Miles, Huberman, & Saldana, 2014), led to the themes
presented in the following section. These themes also contributed to the survey development for
the third phase of data collection.
Online Survey
The quantitative portion of this study utilized a census design (Vaske, 2008) to survey all
members of the WW2100 listserv. Participants in WW2100 were invited by e-mail to complete
an online questionnaire via an online survey website. Of the 281 people invited to participate,
137 responded (45 research team members, 92 stakeholders), leading to a 49% response rate
(Table 2.1). Acceptable response rates range from 35% to 70% (Vaske, 2008). A wave analysis
was conducted to check for response bias and found that average weekly survey return items did
not change over time (Creswell, 2003). Given the response rate, conclusions about the WW2100
participant population can be inferred with 90% confidence (Vaske, 2008).
Table 2.1. Total respondents, total surveys sent, and response rate for each respondent category
Respondent category
Research team
Stakeholder group
Total
Total respondents
45
92
137
Total surveys sent
72
209
281
Response rate
62.5%
44%
49%
Through the 33-item questionnaire, respondents reported their perceptions in five
sections: professional information (capacity in the project, expertise, geographic location),
current and future water use in the Willamette basin, expectations for WW2100, participation in
37
WW2100, and the outcomes of WW2100. Respondents were grouped into one of two groups
‘research team member’ or ‘stakeholder’ based on their responses to the first question: “In what
capacity are you acting in this project?” This acting capacity served as the independent variable
for several of the analyses. Expectations for research team members to perform various roles in
the project, expectations for stakeholders to perform various roles in the project, expectations for
the process and the model, and how well those expectations were met were taken directly from
responses and analyzed as dependent variables. All variables were measured on a scale from 1
“strongly disagree” to 5 “strongly agree,” that they expected members of either group to perform
or witnessed them performing the roles in question.
Statistical analyses
IBM SPSS statistical software was used for all analyses. A Mann Whitney U Test
compared the role, process, and model expectations between research team members and
stakeholders. Point biserial correlation effect sizes were calculated to evaluate the strength of
association among these variables. Wilcoxon Sign Rank tests were used to examine whether
participants had different expectations for research team members and stakeholders, and to
answer whether expectations for research team member and stakeholder roles were met.
Wilcoxon Sign Rank tests were also used to examine whether participant expectations for the
process and model were met. Cohen’s d effect sizes were calculated in these cases to evaluate the
strength of the relationship among these variables.
An Exploratory Factor Analysis was conducted to group expectations of both the
researcher-stakeholder engagement process and the Envision model into expectation factors to
aid in the discussion of different expectation types. A K-means cluster analysis was used to
group respondents into one of two groups according to their responses to process and model
expectations. Membership in stakeholder or research team groups was compared to either of the
two expectation groups through chi-square analysis.
38
Results
WW2100 Participant Characterization
Participation in LAN events is recorded and displayed in Figure 2.1. University affiliation
is the most represented organization at any given researcher-stakeholder engagement event.
Among stakeholders, the most represented groups were state, county, federal, and city
government and the least represented were the tribes, farmers, watershed councils and non-profit
organizations. In some cases of low representation, a group is represented by only one individual
and in other cases, a group is represented only at certain events. For instance, non-profit
organizations, educators, and watershed councils were present at LAN events but not at
Technical Advisory Group (TAG) meetings. Tribal and private industry representatives were not
present at the initial LAN events but consistently attended TAG meetings towards the end of the
project.
Individual participation varied across events and representative categories. Some
individuals attended every, or nearly every, event, while others attended only one or two events.
Individuals from groups with fewer representatives tended to attend events more consistently.
The few farming representatives attended more consistently across all meetings than the
numerous individuals from the government agencies. There are a few exceptions to this pattern.
One county government official attended all but one event; a few university representatives
attended all but one event; one city government official attended all but two events. Participation
and attendance in WW2100 researcher-stakeholder engagement events varied among
organizations and individuals involved.
39
Figure 2.1. Individual participation in LAN events, organized by representative category. Events
1-5 are LAN events while events 6-10 are Technical Advisory Group meetings.
40
Participant Motivations
One question on the survey instrument asked respondents to agree or disagree that they
were motivated to participate by a list of seven suggested motivations. The results of this
question are shown in Figure 2.2 (Data are provided in Appendix F, Table F.1). Concern for
water in the future, professional relevance, and seeking new tools to address water issues were
the most highly rated motivations. Representing a larger group (i.e. agency, constituency,
organization, discipline) in the researcher-stakeholder engagement process and the regional focus
on the Willamette Valley of the project were the least motivating factors for participants overall.
The high rating for the “other” motivation category indicated that motivations could not be so
concisely characterized in a survey question. Therefore, the remainder of this section will focus
on the open-ended responses to what motivates participation in a researcher-stakeholder
engagement process.
Figure 2.2. Motivations of survey respondents. Error bars are standard deviations. To what extent
do you agree or disagree that you were motivated to participate in WW2100 because…
Three categories of motivation were identified from the semi-structured interviews with
key participants and the open-ended “other” responses to the survey question. Attendees were
motivated to participate in WW2100 for social reasons, for knowledge, and for the promised
research products.
Social motivations for participating in WW2100 centered around two themes.
Participants were invited in to the project by others with whom they had a pre-existing
41
relationship and participants were drawn to the project by its interdisciplinary strategy to address
climate and water resources. When asked what led them to be involved, the common refrain
from interviewees was “I was invited” or “I got asked.” Interviewees commonly traced their
participation to another person involved in WW2100 with whom they had worked on other
projects. One stakeholder exemplified this motivation to participate, attributing his participation
to a long-standing relationship with a research team member:
“And what I would say too is that I had a prior relationship with [a research
team member]. So I’ve known [him] for years…So when…he calls me up and
says, hey I’m working on this Willamette Water 2100. This is kind of the general
idea. We have a National Science Foundation grant to do some modeling but it
also has kind of a sociological aspect to it, are you interested? Yeah, that sounds
really interesting. I’d love to be interested.”
Interviewees also expressed that they were drawn to the project because of its interdisciplinary
approach to water resource management. Research team members were excited by the potential
to integrate various disciplines, to learn from each other, and to address a natural resource
problem requiring a collaborative approach. When asked what draws you to collaborative
projects, one research team member responded:
“Well they’re generally pretty fun. It’s nice to be able to be exposed to new stuff
every day. It’s nice to see how other disciplines look at the world. It’s nice to look
at these complex problems that involve lots of pieces that if you don’t take a
multidisciplinary approach, you’re spitting in the wind.”
Knowledge was another key motivation among interviewees. Participation in WW2100
researcher-stakeholder engagement process offered an opportunity to gain knowledge on water
as it related to individuals’ personal interests and/or other professional projects. One stakeholder
explained: “the water issues in Oregon are important. I mean, we all depend on water.”
Interviewees sought knowledge about water resources as they would relate to climate change,
human demand, ecological demand, and the future economy. Water “is our business”, said one
stakeholder, who attended to understand where the future of his business might go. WW2100
interviewees stated that they hoped to inform other projects with the knowledge they gained from
WW2100. Researchers working on reservoirs, other river basins, and at finer scales, as well as
policy makers interested in long-term planning, cross-county water issues, and state regulations
attended WW2100 motivated to inform their external projects. Many were interested in the
impacts of climate change on water resources and understanding how all the processes
42
surrounding water in the Willamette River Basin interact. One researcher summarized his
interest and why others might be interested in this way:
“This project is very interesting because we are looking at more human
dimensions like water demand and water management so how that will affect the
water sustainability.”
Participants not only sought information in participating in this process, but were motivated to
participate by the anticipated products that the process would produce. These include a tool to
model alternative water resource futures, a conversation on water policy and planning, and
achieving scientific broader impacts. WW2100 researcher-stakeholder engagement offered a way
to extend the research results beyond academia. One researcher expressed his motivation to
participate in the WW2100 in this way:
“[It’s] where I was really the most interested because a lot of times these cool
projects are done and then it’s just for the research. And it doesn’t really go
anywhere afterwards.”
Other participants were motivated by the research results themselves. The process would host
and provide fodder for a conversation about water policy and future planning as they relate to
climate change. One survey respondent expressed the perceived obligation to participate in water
policy and future planning conversations in this way:
“Decision makers and those that support decisions need to do more critical
thinking, problem-solving, planning, and policy implementation for a future so we
don't have "water emergencies" that focus solely on humans' unlimited
demand/use of water.”
The project would build a tool to explore alternative future scenarios according to population
growth, climate change, and land use decisions. This tool motivated many interviewees to be
involved in the project for the way it might prove useful. One research team member expressed
these hopes for the WW2100 modeling tool:
“There are big challenges for our society in this century…We have a chance of
doing something about it if we can see more clearly what’s going to happen in the
future or think more clearly about what might happen in the future. And this tool
gives us a way of doing that.”
Participants were drawn to WW2100 by many motivations. They expressed a concern for
water in the future and viewed WW2100 as a way to gain knowledge and tools to address future
water issues. Some participated because the research was relevant professionally, offered an
interdisciplinary approach to water issues, and a way to extend knowledge beyond the project
43
itself. Finally the most prevalent reason among interviewees for participating was simply
because they were invited to attend by a person they knew and trusted.
Participant Expectations
Participants in WW2100 expressed expectations for the process in which they would
engage, for the model the project would build, and for the outcomes it would produce. Some
participants indicated that they did not know what to expect or that they were surprised by
certain elements of the process. Some expectations were not met, some were, and some were
exceeded.
Participants in WW2100 expected that the process would be smooth, that stakeholders
would be engaged in research, and that it would be an opportunity for personal and professional
development. Interviewees expected that the process “would go more smoothly and more
quickly” and “that it would be a strong component of stakeholder involvement.” Expectations
from survey items were grouped into five factors by an exploratory factor analysis (Appendix
G). Four of the five resulting factors (interaction, progress, opportunity, monitor) refer to
expectations for the engagement process (Table 2.2). Survey respondents expected a certain
degree of interaction (mean = 3.35) and understanding (mean = 3.70) between research team
members and stakeholders throughout the process. They also expected to be involved throughout
the process of the project by being kept up to date on its progress (mean = 3.94) and learning to
improve the model (mean = 3.99). Through a transparent process, some participants expected
this to be a way to monitor research conducted at Oregon State University. Participants most
expected that the process would provide an opportunity for personal and professional
development by providing opportunities to work with others, to learn, and to satisfy their
curiosity.
The general sentiment among interviewees, and supported by the survey results, was that
these expectations were not delivered to the degree expected. Speaking of her expectation for
stakeholder engagement, one interviewee said:
“I really thought that we were going to have a robust range of input from people
that were not necessarily involved in water…from an academic point of view…but
that really didn’t come to fruition in as robust a way as I would have liked.”
44
Table 2.2 demonstrates the way in which many of the expectations for the process were not met
to the degree expected. All but one expectation were delivered to some degree by the process.
“Frequent interaction with stakeholders” was a significantly unmet expectation for the process
according to survey respondents (mean = 3.05). In no instance was any process expectation
exceeded, but several were met including “gain career experience (3.28),” “opportunity to share
what I know (3.65),” “stakeholders to make attempts to understand concerns for the project
(3.30).” All other item expectations were met but not to the degree expected. For example, “an
opportunity to learn” was the most expected opportunity provided by the process (mean = 4.38).
Respondents rated that the degree to which it was delivered was significantly less than expected
(mean = 3.98); however, a mean > 3 still indicates that the process provided an opportunity to
learn. The same is true for many of the other items that are met significantly less than expected.
WW2100 participants expected certain characteristics of the Envision water model.
Interviewees expected that the research would provide new numbers to update old research, that
the model would speak to specific interests and that it would be an accessible tool for nonresearch team members. Specific expectations for the model included addressing the value of
ecosystem services, the impact of climate on the availability of stored water, and the carrying
capacity of the Willamette Valley. One interviewee recalled her hope for a groundwater model:
“I was looking forward to the groundwater component, but later learned there
was not a detailed groundwater model or a water quality component.”
One factor in the survey, applicability, referred to respondents’ expectations for the model. That
the model results would be useful in participants’ jobs, that they would contribute to science and
that it would be a complete and integrated model of water in the region was highly expected.
As with expectations for the process, expectations for the model were met but not to the
degree expected. One interviewee reflected on the realization that the model would not do
everything she expected saying:
“There were a lot of things that I wanted to know and it turned out that the model
just wasn’t going to handle everything that a person wanted to know. And the
things that I wanted to know, while they may be really important to the decision
maker really…it was felt that those were really other grant opportunities, that it
wasn’t really directly applicable to this scarcity issue.”
Similarly, survey respondents indicated that their expectations for the applicability of the model,
while met, fell short of their expectations (Table 2.2). Substantial Cohen’s d effect sizes (Vaske,
2008) in Table 2.2 indicated that there was a strong association between expectation and delivery
45
for the researcher-stakeholder engagement process and model. The difference between
expectation and delivery was greater when a survey item was highly expected, as in the case of
the applicability factor items.
Some interviewees viewed this project as one among many contributing to water
sustainability in Oregon. They expressed expectations that WW2100 would build on previous
projects and would push technological innovation for future projects. In speaking of his
expectation for this project to be like an earlier project, one stakeholder stated:
“My experience there was again a good one, where there was a lot of give and
take…and you could see the changes. So I thought it'd be like that and it was and
is.”
One research team member reflected on his expectation for a model representing the complexity
of water in the Willamette Valley.
“I guess my expectation going into it was that it would be able to both push the
state of the art in terms of the science and modeling representation of a quite
complex system, and I think we’ve been somewhat successful in that.”
As shown in the excerpts, interviewees considered expectations that the project would contribute
to ongoing water sustainability projects met.
Many interviewees expressed that they had very low expectations or did not know what
to expect at the beginning of the process. As a result, they encountered some elements of the
project that they did not expect. The most prevalent theme among the unexpected was the role
participants played. Based on this outcome, questionnaire items were developed to evaluate the
expectations for different roles that stakeholders and research team members might play. Of the
suggested roles, participants most expected that stakeholders would “provide a boots-on-theground perspective” (mean = 4.19) and did not expect that they would “develop pieces of the
model” (mean = 2.54) or “write reports” (mean = 2.26). Participants expected that research team
members would fulfill all of the roles, most of all “develop pieces of the model” (mean = 4.64)
and least of all “provide a boots-on-the-ground” perspective” (mean = 3.11) (Figure 2.3). All
participants had significantly different expectations for the degree to which stakeholders would
fulfill the suggested roles and the degree to which research team members would fulfill the
suggested roles (Data and statistical test values are provided in Appendix F, Tables F.2, F.3, and
F.4).
46
Table 2.2. Expectations for the WW2100 model and engagement process and whether or not they
were met.
Progress
To use what we learn to improve the
model
Transparency on the project’s progress
To be kept up to date as the model
evolved
Some of my assumptions to change as the
project progressed
Opportunity
I expected to gain career experience
An opportunity to work with others in my
field
An opportunity to share what I know
An opportunity to learn
An opportunity to work with others
outside of my field
Satisfy my curiosity
Interaction
Frequent interaction with stakeholders
Stakeholders to make attempts to
understand my concerns for the project
Research team members to make attempts
to understand my concerns for the project
Frequent interaction with research team
members
Applicability
Model results that I could use in my job
An integrated model of water in the
Willamette Valley
Model results that would contribute to
science
Monitor
1
Expected
Met
Z-value
p-value
Effect
Size
Cohen’s
d
3.99
3.32
4.33
<.001
.76
3.96
3.40
4.23
<.001
.64
3.94
3.40
3.79
<.001
.60
3.93
3.58
2.60
.004
.44
3.31
3.28
.40
.692
.03
3.93
3.74
2.23
.026
.22
3.56
4.38
3.65
3.98
.90
4.43
.367
<.001
.09
.58
4.01
3.76
2.85
.004
.28
3.90
3.58
3.52
<.001
.34
3.35
2.88
4.17
<.001
.53
3.47
3.30
1.60
.109
.19
3.70
3.24
3.22
.001
.47
3.35
3.05
2.81
.005
.32
3.84
3.31
4.29
<.001
.56
4.34
3.62
5.49
<.001
.85
4.31
3.68
5.41
<.001
.78
3.56
3.47
.971
.332
.09
Cell values are means of reported expectations on a 5-point scale from 1 “strongly disagree” to 5 “strongly agree.
A mean value greater than 3 indicates that an item was expected and/or that the expectation was met.
Meeting role expectations varied among stakeholders and research team members and
according to the role in question. Stakeholders did not fully meet expectations for their roles to
“evaluate assumptions,” “provide a boots-on-the-ground perspective”, and “communicate with
47
stakeholders.” However, stakeholders exceeded expectations to “develop pieces of the model”
(mean = 3.85) and to “write reports” (mean = 3.48). All other stakeholder roles were met (Figure
2.3; Table A.3). Research team members met expectations for all but one role. Research team
members were somewhat expected to “provide a boots-on-the-ground perspective” (mean =
3.10) but did not meet that expectation (mean = 2.78). Although all other roles were reported
with means greater than 3, indicating that they were met, they were delivered to a degree
significantly less than expected (Figure 2.3; Table A.4). In some cases participants fulfilled roles
they did not expect to and in other cases, participants were not able to fulfill their expected roles.
Figure 2.3. Expectations for stakeholders and research team members and whether they were
met. Values are means ranging from 1 “strongly disagree” to 5 “strongly agree” that the group in
question was expected to or fulfilled the following roles. Asterisks indicate significant
differences at p < .05. Blue asterisks indicate significant difference between stakeholder
expectations and whether they were met. Orange asterisks indicate significant difference between
stakeholder expectations and whether they were met. Black asterisks indicate significant
difference between expectations for stakeholder and research team member roles.
48
Differences between Participant Groups
Participants in WW2100 belonged to one of two groups: research team or stakeholder
group. Role expectations for the researcher-stakeholder engagement process did not differ
between groups. However their expectations differed significantly regarding the engagement
process and the resulting model.
Research team members and stakeholders did not differ in their expectations for the roles
each other would fulfill in the engagement process. Table 2.3 indicates research team members
and stakeholders expected stakeholders to fulfill the suggested roles in the same way. Similarly,
Table 2.4 indicates that both groups expected research team members to fulfill all of the
suggested roles in the same way. Effect sizes indicate that there was a minimal to typical
relationship (Vaske, 2008) between group membership and role expectations. For example, there
was a minimal to typical relationship between group membership and expectation that
stakeholders would “develop pieces of the model” (rpb = .19). There were also minimal to typical
associations between group membership and expectations that research team members would
“provide a scientific perspective” (rpb = .21) and “guide research questions” (rpb = .16). In the
case of each other’s roles, research team members and stakeholders did not differ significantly in
their expectations.
Table 2.3. Expectations for stakeholder roles by respondent category.
Guide research questions
Develop pieces of the model
Write scenario assumptions
Evaluate scenario assumptions
Interpret model outputs
Write reports
Provide a “boots-on-the-ground”
perspective
Provide a scientific perspectives
Communicate with stakeholders
Communicate with stakeholders
who are not active WW2100
participants
1
Research Team
Stakeholder Group
U - value
pvalue
3.56
2.25
3.13
3.97
3.38
2.28
3.65
2.68
3.20
3.80
3.31
2.28
.56
1.93
.28
.62
.27
.25
.575
.053
.784
.536
.787
.806
Effect
Size
(rpb)
.04
.19
.03
.08
.03
.00
4.21
4.10
.63
.532
.07
3.18
3.94
3.31
3.72
.53
1.20
.594
.228
.06
.11
3.81
3.82
.08
.937
.00
Cell values are means of reported expectations on a 5-point scale from 1 “strongly disagree” to 5 “strongly agree.”
49
Table 2.4. Expectations for research team member roles by respondent category.
Guide research questions
Develop pieces of the model
Write scenario assumptions
Evaluate scenario assumptions
Interpret model outputs
Write reports
Provide a “boots-on-the-ground”
perspective
Provide a scientific perspectives
Communicate with stakeholders
Communicate with stakeholders
who are not active WW2100
participants
1
Research Team
Stakeholder Group
U-value
pvalue
4.66
4.81
4.44
4.63
4.75
4.65
4.45
4.59
4.49
4.50
4.64
4.53
1.21
1.50
.38
.67
.85
.15
.226
.133
.708
.506
.398
.883
Effect
Size
(rpb)
.16
.18
.04
.10
.11
.08
2.97
3.21
.86
.389
.12
4.78
4.63
4.52
4.49
1.86
.95
.063
.342
.21
.11
4.22
4.22
.14
.891
.00
Cell values are means of reported expectations on a 5-point scale from 1 “strongly disagree” to 5 “strongly agree.”
Research team members generally had higher expectations than stakeholders for
WW2100 and its researcher-stakeholder engagement process (Table 2.5). A K-means cluster
analysis assigned survey respondents to one of two groups according to their expectations of the
items listed in Table 2.6. A greater proportion of research team members belonged to the ‘high
expectations’ group than to the ‘low to no expectations’ group. The reverse is true for
stakeholder survey respondents. A greater number of stakeholders held low to no expectations
than high expectations for the model and the engagement process.
Research team members and stakeholders differed in their expectations for three of the
five factors regarding the researcher-stakeholder engagement process (Table 2.6). Research team
members expected significantly greater inclusion in the progress of the project, opportunities
presented by the project and interaction with each other and stakeholders as a result of the
process. Stakeholders expected less than research team members in all but one case – monitoring
research at Oregon State University (stakeholder mean = 3.66 > research team mean =3.33).
Research team members and stakeholders agreed that they expected all but two items to some
degree. Stakeholders disagreed with researchers in that they did not expect to “gain career
experience” (stakeholder mean = 2.92) nor “frequent interaction with research team members”
(stakeholder mean = 2.98). Researchers and stakeholders agreed that they expected the process to
be “an opportunity to learn,” to “satisfy [their] curiosity,” that research team members and
50
stakeholders would “make attempts to understand [their] concerns for the project,” and that the
model would provide “results that [they] could use in [their] jobs.” Substantial effect sizes
(Vaske, 2008) demonstrated a strong relationship between group membership and expectations
for the opportunity factor (rpb = .39), namely to “gain career experience” (rpb = .50), and for
“frequent interaction with research team members” (rpb = .50).
Table 2.5. Belonging to two expectation groups by professional group in WW2100 researcherstakeholder engagement process.
Expectation
group
Low to no
expectations
High
expectations
1
Group Belonging1
Research
Stakeholder
team
Total
21
44
36
79
57
64
Χ2-value
p-value
4.71
.030
Effect Size
(φ)
.22
Cell values are counts of individual respondents.
Discussion and Conclusions
Participants in researcher-stakeholder engagement processes are diverse, representing
many organizations, motivations, and expectations. The composition of participants in any one
engagement event can be vastly different from that of another engagement event for the same
research project. As the composition of participants changes, so too can the driving motivations
for participation and the expectations of what engagement will achieve. Being transparent with
motivations and expectations in a project can contribute to developing trust (Mackenzie et al.,
2012). Identifying participant motivations and expectations in one researcher-stakeholder
engagement project may aid future projects to clarify participant motivations and expectations in
order to facilitate trust.
51
Table 2.6. Expectations for the researcher-stakeholder engagement process and resulting model
by respondent category.
Research
Team
Stakeholders
U-value
p-value
Progress
To use what we learn to improve the
model
Transparency on the project’s
progress
To be kept up to date as the model
evolved
Some of my assumptions to change as
the project progressed
4.17
3.86
2.31
.021
Effect
size
(rpb)
.24
4.28
3.87
2.10
.036
.24
4.23
3.87
1.88
.060
.21
4.03
3.87
.71
.476
.10
4.14
3.80
2.15
.031
.23
Opportunity
I expected to gain career experience
An opportunity to work with others in
my field
An opportunity to share what I know
An opportunity to learn
An opportunity to work with others
outside of my field
Satisfy my curiosity
4.20
4.06
3.68
2.92
3.73
5.03
<.001
<.001
.39
.50
4.26
3.75
2.82
.005
.28
3.94
4.43
3.38
4.33
2.86
.77
.004
.440
.30
.08
4.37
3.83
2.81
.005
.29
4.12
3.83
1.39
.165
.16
Interaction
Frequent interaction with stakeholders
Stakeholders to make attempts to
understand my concerns for the
project
Research team members to make
attempts to understand my concerns
for the project
Frequent interaction with research
team members
3.79
3.65
3.22
3.13
3.66
2.65
<.001
.008
.34
.27
3.57
3.26
1.50
.133
.15
3.94
3.48
1.88
.060
.22
4.00
2.98
4.83
<.001
.50
Applicability
Model results that I could use in my
job
An integrated model of water in the
Willamette Valley
Model results that would contribute to
science
4.33
4.07
1.44
.151
.17
3.84
3.83
.09
.925
.01
4.59
4.19
2.04
.042
.22
4.56
4.19
2.20
.028
.22
Monitor
3.33
3.66
1.69
.092
.16
1
Means and standard deviations are measured on a 5-point scale from 1 “Strongly Disagree” to 5 “Strongly Agree.”
52
Participants in WW2100 were motivated to participate for social reasons such as
collaboration and invitation, knowledge about the topic of interest or to inform other projects,
and for reaching broader audiences with the alternative futures modeling tool produced by the
project. The social motivations identified in this study fit well within the collectivism and
altruism motivations identified by Rotman et al. (2012). Scientists and stakeholders are
motivated to engage with each other in research because they see a mutual benefit
(collaboration/collectivism) and/or because they believe they can help the other
(invitation/altruism). That many interviewees were motivated to participate because of a personal
invitation reveals why participants came to believe they could benefit from or contribute to
WW2100. The remaining motivations this study identifies, knowledge, tool, and impact-seeking,
fall within the Rotman et al.'s (2012) egoism category of motivations. Rotman et al.’s (2012)
motivational categories in citizen science may be too superficial for long-term researcherstakeholder engagement processes. Participants were motivated to participate in the project by
what they could gain personally and professionally. However, their motivations are founded
deeper than exchanging knowledge and tools; they are founded on personal relationships with
others involved and previous experiences with similar projects.
Projects like WW2100 typically have three kinds of goals: outcomes for research,
outcomes for individuals, and outcomes for social-ecological systems like influencing policies
(Shirk et al., 2012). Participant expectations for the WW2100 process, the roles they would play
within it, and the resulting model reflect these typical researcher-stakeholder engagement process
goals. Expectations for the researcher-stakeholder engagement process were met, though
oftentimes not to the degree expected. Only one expectation for the process was not met and that
was the expectation to have frequent interaction with stakeholders. Participants had different
expectations for the roles stakeholders and research team members would play. On some
occasions, stakeholders exceeded role expectations; on others, they fell short of the role
expectations. Research team members met all expectations for their roles but to a degree that was
less than expected. Research team members were only slightly expected to “provide a boots-onthe-ground perspective” and this was the only role that they did not fulfill at all. Finally,
participants expected that the resulting model would be an accurate representation of water in the
Willamette Valley, that the model would contribute to science, and that it would provide results
useful to their jobs. All participants agreed that the model met these expectations but to a degree
53
significantly less than that which was expected. It is possible that because this study was
conducted during the last year and not after WW2100’s official close, expectations were in the
process of being fulfilled and a post-process assessment might have yielded different results.
Still, through the researcher-stakeholder engagement process and this study of met and unmet
expectations, research, management, and policy questions can be identified and addressed in
future projects.
In some instances motivations and expectations differed between participant groups. For
instance, only stakeholders expressed a motivation to participate in this project because it offered
a way to inform other projects. However, stakeholders and research team members did not differ
in the roles they expected each other to play, nor in their expectations for the research process
and resulting model.
There was a distinct difference between what all participants expected of research team
members and what all participants expected of stakeholders in WW2100. Research team
members were responsible for all of the suggested tasks but providing a ‘boots-on-the-ground’
perspective. Stakeholders, on the other hand, were expected to provide feedback and
communicate results rather than participate in asking questions and conducting research.
Similarly, stakeholders in a climate modeling project were expected to fill data gaps, develop
final scenarios and advise on visualization representation (Sheppard et al., 2011). Stakeholders
are commonly expected to contribute knowledge (Johnson, 2011) and based on the expectations
for researcher and stakeholder roles, most WW2100 participants expected the researcherstakeholder engagement process to be a “contributory” project rather than a “collaboration”
(Shirk et al., 2012).
Contrary to expectation, stakeholders in WW2100 developed pieces of the model and
wrote reports. When stakeholders are included in data interpretation, a project moves into the
“collaboration – co-creation” range (Shirk et al., 2012). Although stakeholders are not normally
included in data interpretation, in some cases they can be (Johnson, 2011) and when this occurs
both scientists and stakeholders practice analysis (Webler, 1998). When all collaborators share in
the decision-making powers of a project, true collaboration increases (Kearney et al., 2007).
The perceived role of researchers can evolve throughout an engagement process as well
(Becu et al., 2008). Some scientists expect to participate only as expert presenters in researcherstakeholder engagement (Bartels et al., 2013) and at the outset of WW2100, this may have been
54
true. However, comparing WW2100 role expectation to delivery, the degree to which
stakeholders and research team members fulfilled the role of communicating with stakeholders
converged. In other studies, researchers became like the stakeholders themselves through
collaboration (Becu et al., 2008). Proponents of collaboration applaud such blurring of the role
lines (Kearney et al., 2007), yet it should be mentioned that more conflicts can arise when project
team members play the same roles as stakeholders and as each other (Daniell et al., 2010).
Research team members held higher expectations for the process and model than did
stakeholders but, in all but one instance, both groups agreed on their expected process and
outcomes. As the project leaders, research team members may hold higher expectations for the
project because they know what WW2100 intends to achieve. In many similar cases, the
scientists invite partners outside of academia to join them (Lang et al., 2012). Unless they have
previous experience stakeholders do not know what to expect and therefore keep expectations
low. Stakeholders disagreed with research team members when they expected to gain career
experience through the process. This could be because the experience gained through
engagement is only indirectly related to stakeholder careers. As with roles, when research team
members and stakeholders expect the same elements but to a different degree, conflict may arise
as a result of differing methodological and quality standards (Lang et al., 2012).
Potential conflicts can be avoided by hosting early engagement meetings which focus on
developing shared expectations for the process and for each other (Tim Lynam et al., 2010).
Several case studies of interdisciplinary research emphasize the importance of establishing
expectations and clear roles and responsibilities at the outset of a project in their ‘lessons
learned’ sections (Lang et al., 2012; Mackenzie et al., 2012; Matso & Becker, 2014; Voinov &
Bousquet, 2010). In a co-modeling project in Thailand, stakeholder expectations for the
engagement process and its outcomes influenced their perceptions of the resulting model (Becu
et al., 2008). An engagement stage which develops shared expectations for the process,
participant roles, and products is especially important in academic-led research projects. For
without such a process, stakeholders develop independent ideas for what the process can do for
them which may also lead to conflict (Daniell et al., 2010).
Understanding motivations and expectations for participation in researcher-stakeholder
engagement projects is important not only to avoid conflict but also to improve the results
produced and the likelihood for future participation. Universities are viewed increasingly as
55
development hubs (Hansen & Lehmann, 2006). Although researchers tend to emphasize the
contributions of their research to science through publications, stakeholders require increased
applied research in natural resource management (Johnson, 2011). In order to address the needs
and expectations of participating groups, project scientists must first understand them. When a
project allows participants to meet their needs and expectations, they are more likely to
participate again (Eccles & Wigfield, 2002). In WW2100 participants were motivated to join the
project by previous positive engagement and outreach experiences with other participants. Thus,
engaging with stakeholders in transdisciplinary research which meets participant needs and
expectations may form relationships and empower stakeholders to where they are motivated to
participate in future projects. Contrarily, a project which does not meet needs and expectations
may dissuade potential participants from engaging with each other again.
To achieve, and to continue achieving broader impacts through researcher-stakeholder
engagement, projects must address what participants expect from the process. In this process,
participants expected to collaborate with one another through frequent interactions. They
expected to gain and to share knowledge and to play different roles to contribute to the project.
To inform natural resource management through broader impacts, research must address the
product expectations of their stakeholders. This project’s stakeholders expected a model which
accurately represented water in their region and which produced useful results. Future projects
can look to WW2100 as an example of what their stakeholders and research team members may
expect and then plan a facilitated session in their own process to clarify the roles and
expectations of each participant.
56
References
Auerbach, C. F., & Silverstein, L. B. (2003). Qualitative data: An introduction to coding and
analysis. NYU Press.
Baker, J. P., Hulse, D. W., Gregory, S. V, White, D., Van Sickle, J., Berger, P. A., …
Schumaker, N. H. (2004). Alternative Futures for the Willamette River Basin , Oregon.
Ecological Applications, 14(2), 313–324.
Bartels, W. L., Furman, C. A., Diehl, D. C., Royce, F. S., Dourte, D. R., Ortiz, B. V., … Jones, J.
W. (2013). Warming up to climate change: a participatory approach to engaging with
agricultural stakeholders in the Southeast US. Regional Environmental Change, 13, S45–
S55. doi:10.1007/s10113-012-0371-9
Becu, N., Neef, A., Schreinemachers, P., & Sangkapitux, C. (2008). Participatory computer
simulation to support collective decision-making: Potential and limits of stakeholder
involvement. Land Use Policy, 25(4), 498–509. doi:10.1016/j.landusepol.2007.11.002
Brainard, R. E., Weijerman, M., Eakin, C. M., McElhany, P., Miller, M. W., Patterson, M., …
Birkeland, C. (2013). Incorporating climate and ocean change into extinction risk
assessments for 82 coral species. Conservation Biology : The Journal of the Society for
Conservation Biology, 27(6), 1169–78. doi:10.1111/cobi.12171
Callahan, B., Miles, E., & Fluharty, D. (2013). Policy implications of climate forecasts for water
resources management in the Pacific Northwest. Policy Sciences, 32(3), 269–293.
Cash, D. W., Clark, W. C., Alcock, F., Dickson, N. M., Eckley, N., Guston, D. H., … Mitchell,
R. B. (2003). Knowledge systems for sustainable development. Proceedings of the National
Academy of Sciences of the United States of America, 100(14), 8086–8091.
doi:10.1073/pnas.1231332100
Chang, H., Praskievicz, S., & Parandvash, H. (2014). Sensitivity of urban water consumption to
weather and climate variability at multiple temporal scales: The case of Portland, Oregon.
International Journal of Geospatial and Environmental Research, 1(1), Article 7.
Clarke, S., & Roome, N. (1999). Sustainable business: Learning-action networks as
organizational assets. Business Strategy and the Environment, 8, 296–310.
Cohen, S. J. (2010). From observer to extension agent—using research experiences to enable
proactive response to climate change. Climatic Change, 100(1), 131–135.
doi:10.1007/s10584-010-9811-z
Creswell, J. W. 2003. Research design: Qualitative, quantitative, and mixed-methods
approaches. Thousand Oaks, CA: Sage.
57
Cross, M. S., McCarthy, P. D., Garfin, G., Gori, D., & Enquist, C. (2013). Accelerating
adaptation of natural resource management to address climate change. Conservation
Biology : The Journal of the Society for Conservation Biology, 27(1), 4–13.
doi:10.1111/j.1523-1739.2012.01954.x
Daniell, K. A., White, I., Ferrand, N., Ribarova, I. S., Coad, P., Rougier, J. E., … Burn, S.
(2010). Co-engineering participatory water management processes: Theory and insights
from Australian and Bulgarian interventions. Ecology and Society, 15(4), 11. Retrieved
from http://www.ecologyandsociety.org/vol15/iss4/art11/
Dewulf, A., François, G., Pahl-wostl, C., & Taillieu, T. (2007). A framing approach to crossdisciplinary research collaboration: Experiences from a large-scale research project on
adaptive water management. Ecology and Society, 12(2), 14.
Dietz, T., Ostrom, E., & Stern, P. C. (2003). The struggle to govern the commons. Science (New
York, N.Y.), 302(5652), 1907–12. doi:10.1126/science.1091015
Dilling, L., & Lemos, M. C. (2011). Creating usable science: Opportunities and constraints for
climate knowledge use and their implications for science policy. Global Environmental
Change, 21(2), 680–689. doi:10.1016/j.gloenvcha.2010.11.006
Eccles, J. S., & Wigfield, A. (2002). Motivational beliefs, values, and goals. Annual Review of
Psychology, 53, 109 – 132.
Farkas, N. (1999). Dutch Science Shops: Matching community needs with university R & D.
Science Studies, 2, 33–47.
Freitag, A. (2014). Naming, framing, and blaming: Exploring ways of knowing in the
deceptively simple question “What is water quality?” Human Ecology, 42, 325–337.
doi:10.1007/s10745-014-9649-5
Frodeman, R., Holbrook, J. B., Bourexis, P. S., Cook, S. B., Diederick, L., & Tankersley, R. A.
(2013). Broader Impacts 2.0: Seeing - and Seizing—the Opportunity. Bioscience, 63(3),
153–155. doi:10.1525/bio.2013.63.3.2
Fuller, B. (2011). Enabling problem-solving between science and politics in water conflicts:
impasses and breakthroughs in the Everglades, Florida, USA. Hydrological Sciences
Journal, 56(4), 576–587. doi:10.1080/02626667.2011.579075
Gregory, R., Arvai, J., & Gerber, L. R. (2013). Structuring decisions for managing threatened
and endangered species in a changing climate. Conservation Biology : The Journal of the
Society for Conservation Biology, 27(6), 1212–21. doi:10.1111/cobi.12165
Grin, J., & van de Graaf, H. (1996). Technology assessment as learning. Science, Technology &
Human Values, 21(1), 72–99.
58
Halofsky, J. E., Peterson, D. L., Furniss, M. J., Joyce, L. A., Millar, C. I., & Neilson, R. P.
(2011). Workshop approach for developing change adaptation strategies and actions for
natural resource management agencies in the United States. Journal of Forestry, (June),
219–225.
Hansen, J. A., & Lehmann, M. (2006). Agents of change: universities as development hubs.
Journal of Cleaner Production, 14(9-11), 820–829. doi:10.1016/j.jclepro.2005.11.048
Hildén, M. (2011). The evolution of climate policies – the role of learning and evaluations.
Journal of Cleaner Production, 19(16), 1798–1811. doi:10.1016/j.jclepro.2011.05.004
Holman, I. P., Rounsevell, M. D. A., Cojacaru, G., Shackley, S., McLachlan, C., Audsley, E., …
Richards, J. A. (2008). The concepts and development of a participatory regional integrated
assessment tool. Climatic Change, 90(1-2), 5–30. doi:10.1007/s10584-008-9453-6
Holzkämper, A., Kumar, V., Surridge, B. W. J., Paetzold, A., & Lerner, D. N. (2012). Bringing
diverse knowledge sources together - a meta-model for supporting integrated catchment
management. Journal of Environmental Management, 96(1), 116–27.
doi:10.1016/j.jenvman.2011.10.016
Huntington, H. P., Brown-schwalenberg, P. K., Frost, K. J., Fernandez-gimenez, M. E., Norton,
D. W., & Rosenberg, D. H. (2002). Observations on the workshop as a means of improving
communication between holders of traditional and scientific knowledge. Environmental
Management, 30(6), 778–792. doi:10.1007/s00267-002-2749-9
Johnson, T. R. (2011). Fishermen, scientists, and boundary spanners: Cooperative research in the
U.S. Illex squid fishery. Society & Natural Resources: An International Journal, 24(3),
242–255. doi:10.1080/08941920802545800
Kastenhofer, K., Bechtold, U., & Wilfing, H. (2011). Sustaining sustainability science: The role
of established inter-disciplines. Ecological Economics, 70(4), 835–843.
doi:10.1016/j.ecolecon.2010.12.008
Kearney, J., Berkes, F., Charles, A., & Wiber, M. (2007). The role of participatory governance
and community-based management in integrated coastal and ocean management in Canada.
Coastal Management, 35(1), 79–104. doi:10.1080/10.1080/08920750600970511
Kloprogge, P., & van der Sluijs, J. P. (2006). The inclusion of stakeholder knowledge and
perspectives in integrated assessment of climate change. Climatic Change, 75, 359–389.
doi:10.1007/s10584-006-0362-2
Landry, R., Amara, N., & Lamari, M. (2001). Utilization of social science research knowledge in
Canada. Research Policy, 30, 333–349.
Lang, D. J., Wiek, A., Bergmann, M., Stauffacher, M., Martens, P., Moll, P., … Thomas, C. J.
(2012). Transdisciplinary research in sustainability science: practice, principles, and
59
challenges. Sustainability Science, 7(Supplement 1), 25–43. doi:10.1007/s11625-011-0149x
Latour, B. and Woolgar, S. 1979. Laboratory life: The construction of scientific facts. Los
Angeles, London: Sage.
Lautenbach, S., Berlekamp, J., Graf, N., Seppelt, R., & Matthies, M. (2009). Scenario analysis
and management options for sustainable river basin managementt: Application of the Elbe
DSS. Environmental Modelling & Software, 24, 26–43. doi:10.1016/j.envsoft.2008.05.001
Lawler, J. J., Tear, T. H., Pyke, C., Shaw, M. R., Gonzalez, P., Kareiva, P., … Pearsall, S.
(2010). Resource management in a changing and uncertain climate. Frontiers in Ecology
and the Environment, 8(1), 35–43. doi:10.1890/070146
Lemos, M. C., & Morehouse, B. J. (2006). The co-production of science and policy in integrated
climate assessments. Global Environmental Change, 15(2005), 57–68.
doi:10.1016/j.gloenvcha.2004.09.004
Lengwiler, M. (2008). Participatory approaches in science and technology: Historical origins and
current practices in critical perspective. Science, Technology & Human Values, 33(2), 186–
200.
Lester, S. E., McLeod, K. L., Tallis, H., Ruckelshaus, M., Halpern, B. S., Levin, P. S., …
Parrish, J. K. (2010). Science in support of ecosystem-based management for the US West
Coast and beyond. Biological Conservation, 143(3), 576–587.
doi:10.1016/j.biocon.2009.11.021
Leydesdorff, L., & Ward, J. (2005). Science shops: a kaleidoschope of science-society
collaborations in Europe. Public Understanding of Science.
doi:10.1177/0963662505056612
Li, L. C., Grimshaw, J. M., Nielsen, C., Judd, M., Coyte, P. C., & Graham, I. D. (2009). Use of
communities of practice in business and health care sectors: A systematic review.
Implementation Science, 4(27), 1–9. doi:10.1186/1748-5908-4-27
Lienert, J., Monstadt, J., & Truffer, B. (2006). Future scenarios for a sustainable water sector : A
case study from Switzerland. Environmental Science & Technology, 40(2), 436–442.
doi:10.1021/es0514139
Lynam, T., de Jong, W., Sheil, D., Kusumanto, T., & Evans, K. (2007). A review of tools for
incorporating community knowledge, preferences, and values into decision making in
natural resources management. Ecology and Society, 12(1).
Lynam, T., Drewry, J., Higham, W., & Mitchell, C. (2010). Adaptive modelling for adaptive
water quality management in the Great Barrier Reef region, Australia. Environmental
Modelling and Software, 25(11), 1291–1301. doi:10.1016/j.envsoft.2009.09.013
60
Mackenzie, J., Tan, P. L., Hoverman, S., & Baldwin, C. (2012). The value and limitations of
Participatory Action Research methodology. Journal of Hydrology, 474, 11–21.
doi:10.1016/j.jhydrol.2012.09.008
Mader, M., Mader, C., Zimmermann, F. M., Görsdorf-Lechevin, E., & Diethart, M. (2013).
Monitoring networking between higher education institutions and regional actors. Journal
of Cleaner Production, 49, 105–113. doi:10.1016/j.jclepro.2012.07.046
Mahmoud, M., Liu, Y., Hartmann, H., Stewart, S., Wagener, T., Semmens, D., … Winter, L.
(2009). A formal framework for scenario development in support of environmental
decision-making. Environmental Modelling and Software, 24(7), 798–808.
doi:10.1016/j.envsoft.2008.11.010
Manring, S. L. (2014). The role of universities in developing interdisciplinary action research
collaborations to understand and manage resilient social-ecological systems. Journal of
Cleaner Production, 64, 125–135. doi:10.1016/j.jclepro.2013.07.010
Martin-Sempere, M. J., Garzon-Garcia, B., & Rey-Rocha, J. (2008). Scientists’ motivation to
communicate science and technology to the public: surveying participants at the Madrid
Science Fair. Public Understanding of Science, 17(3), 349–367.
doi:10.1177/0963662506067660
Matso, K. E., & Becker, M. L. (2014). What can funders do to better link science with decisions?
Case studies of coastal communities and climate change. Environmental Management,
54(6), 1356–71. doi:10.1007/s00267-014-0347-2
McClure, M. M., Alexander, M., Borggaard, D., Boughton, D., Crozier, L., Griffis, R., … Van
Houtan, K. (2013). Incorporating climate science in applications of the US endangered
species act for aquatic species. Conservation Biology : The Journal of the Society for
Conservation Biology, 27(6), 1222–33. doi:10.1111/cobi.12166
Miles, M. B., Huberman, A.M., & Saldana, J. (2014). Qualitative data analysis: A methods
sourcebook (3rd ed.). Thousand Oaks: Sage Publications.
Nadkarni, N. M., & Stasch, A. E. (2013). How broad are our broader impacts? An analysis of the
National Science Foundation’s Ecosystem Studies Program and the Broader Impacts.
Frontiers in Ecology and the Environment2, 11(1), 13–19. doi:10.1890/110106
National Science Board. (2011). Merit Review Criteria. Review and Revisions. Retrieved from
papers2://publication/uuid/910994F5-3EE1-4236-8346-5602486DA1D2
National Science Foundation. (2012). Proposal and award policies and procedures guide.
Retrieved from http://www.nsf.gov/pubs/policydocs/pappguide/nsf13001/nsf13_1.pdf
Pahl-wostl, C. (2007). Transitions towards adaptive management of water facing climate and
global change. Water Resources Management, 21, 49–62. doi:10.1007/s11269-006-9040-4
61
Pahl-wostl, C., Craps, M., Dewulf, A., Mostert, E., Tabara, D., & Taillieu, T. (2007). Social
learning and water resources management. Ecology and Society, 12(2), 5.
Pahl-wostl, C., Sendzimir, J., Jeffrey, P., Aerts, J., Berkamp, G., & Cross, K. (2007). Managing
change toward adaptive water management through social learning. Ecology and Society,
12(2), 30.
Patton, M. Q. (2002). Qualitative research and evaluation methods. 3rd edition. Thousand Oaks,
CA: Sage.
Pearson, G., Pearson, G., Pringle, S. M., Pringle, S. M., Thomas, J. N., & Thomas, J. N. (1997).
Scientists and the public understanding of science. Science, 6, 279–289.
Podolny, J. M., & Page, K. L. (1998). Network forms of organization. Annual Review of
Sociology, 24, 57–76.
Raddick, M. J., Bracey, G., Gay, P. L., Lintott, C. J., Murray, P., Schawinski, K., …
Vandenberg, J. (2010). Galaxy Zoo: Exploring the motivations of citizen science
volunteers. Astronomy Education Review, 9, 15. doi:10.3847/AER2009036
Rayner, S., Lach, D., & Ingram, H. (2005). Weather forecasts are for wimps*: Why water
resource managers do not use climate forecasts. Climatic Change, 69, 197–227.
Reed, M. S., Evely, A. C., Cundill, G., Fazey, I., Glass, J., & Laing, A. (2010). What is social
learning ? Ecology and Society.
Riley, C., Matso, K., Leonard, D., Stadler, J., Trueblood, D., & Langan, R. (2011). How research
funding organizations can increase application of science to decision-making. Coastal
Management, 39(3), 336–350. doi:10.1080/08920753.2011.566117
Rittel, H. W. J., & Webber, M. M. (1973). Dilemmas in a general theory of planning. Policy
Sciences, 4(2), 155–169.
Robinson, C. J., & Wallington, T. J. (2012). Boundary work : Engaging knowledge systems in
co-management of feral animals on indigenous lands. Ecology and Society, 17(2), 16.
Rotman, D., Preece, J., Hammock, J., Procita, K., Hanse, D., Parr, C., … Jacobs, D. (2012).
Dynamic changes in motivation in collaborative citizen-science projects. In Session: Civic
and Community Engagement (pp. 217–226). doi:10.1145/2145204.2145238
Sheppard, S. R. J., Shaw, A., Flanders, D., Burch, S., Wiek, A., Carmichael, J., … Cohen, S.
(2011). Future visioning of local climate change: A framework for community engagement
and planning with scenarios and visualisation. Futures, 43(4), 400–412.
doi:10.1016/j.futures.2011.01.009
62
Shirk, J. L., Ballard, H. L., Wilderman, C. C., Phillips, T., Wiggins, A., Jordan, R., … Bonney,
R. (2012). Public participation in scientific research: A framework for deliberate design.
Ecology and Society, 17(2), 29. doi:10.5751/ES-04705-170229
Smith, J. B., Strzepek, K., Rozaklis, L., Ellinghouse, C., & Hallett, K. (2009). The Potential
Consequences of Climate Change for Boulder Colorado’s Water Supplies.
Snover, A. K., Mantua, N. J., Littell, J. S., Alexander, M. A., McClure, M. M., & Nye, J. (2013).
Choosing and using climate-change scenarios for ecological-impact assessments and
conservation decisions. Conservation Biology: The Journal of the Society for Conservation
Biology, 27(6), 1147–57. doi:10.1111/cobi.12163
Sol, J., Beers, P. J., & Wals, A. E. J. (2013). Social learning in regional innovation networks:
trust, commitment and reframing as emergent properties of interaction. Journal of Cleaner
Production, 49, 35–43. doi:10.1016/j.jclepro.2012.07.041
Spiegel, J. M., Breilh, J., Beltran, E., Parra, J., Solis, F., Yassi, A., … Parkes, M. (2011).
Establishing a community of practice of researchers, practitioners, policy-makers and
communities to sustainably manage environmental health risks in Ecuador. BMC
International Health and Human Rights, 11 Suppl 2(Suppl 2), S5. doi:10.1186/1472-698X11-S2-S5
Star, S. L., & Griesemer, J. R. (1989). Institutional ecology, 'translations’ and boundary objects:
Amateurs and professionals in Berkeley’s Museum of Vertebrate Zoology, 1907-39. Social
Studies of Science, 19(3), 387–420.
Stubbs, M., & Lemon, M. (2001). Learning to network and networking to learn: Facilitating the
process of adaptive management in a local response to the UK’s National Air Quality
Strategy. Environmental Management, 27(8), 321–334. doi:10.1007/s002670010152
Swart, R. J., Raskin, P., & Robinson, J. (2004). The problem of the future: sustainability science
and scenario analysis. Global Environmental Change, 14(2), 137–146.
doi:10.1016/j.gloenvcha.2003.10.002
Tuler, S. (1998). Learning through participation. Human Ecology Review, 5(1), 58–60.
Tullos, D., Brown, P. H., Kibler, K., Magee, D., Tilt, B., & Wolf, A. T. (2010). Perspectives on
the salience and magnitude of dam impacts for hydro development scenarios in China.
Water Alternatives, 3(2), 71–90.
Turnhout, E., Stuiver, M., Klostermann, J., Harms, B., & Leeuwis, C. (2013). New roles of
science in society: Different repertoires of knowledge brokering. Science and Public Policy,
40, 354–365. doi:10.1093/scipol/scs114
Van Herk, S., Zevenbergen, C., Ashley, R., & Rijke, J. (2011). Learning and action alliances for
the integration of flood risk management into urban planning: A new framework from
63
empirical evidence from The Netherlands. Environmental Science & Policy, 14(5), 543–
554. doi:10.1016/j.envsci.2011.04.006
Vaske, J. V. (2008). Surveey research and analysis: Applications in parks, recreation and human
dimensions. State College: PA: Venture.
Voinov, A., & Bousquet, F. (2010). Modelling with stakeholders. Environmental Modelling &
Software, 25(11), 1268–1281. doi:10.1016/j.envsoft.2010.03.007
Webler, T. (1998). Beyond science: Deliberation and analysis in public decision making. Human
Ecology Review, 5(1), 61–62.
Weible, C. M., & Sabatier, P. A. (2009). Coalitions, science, and belief change: Comparing
adversarial and collaborative policy subsystems. Policy Studies Journal, 37(2), 195–212.
doi:10.1111/j.1541-0072.2009.00310.x
Wolf, B., Lindenthal, T., Szerencsits, M., Holbrook, J. B., & Heb, J. (2013). Evaluating research
beyond scientific impact: How to include criteria for productive interactions and impact on
practice and society. GAIA, 22(2), 104–114.
Yang, L., Wu, J., & Shen, P. (2013). Roles of science in institutional changes: The case of
desertification control in China. Environmental Science & Policy, 27(37), 32–54.
doi:10.1016/j.envsci.2012.10.017
64
CHAPTER 3: JOURNAL ARTICLE [for submission to The International Journal of
Science in Society]
Broadening broader impacts: Lessons from a researcher-stakeholder engagement process
for water sustainability
Abstract
Academic researchers are obligated by funding agencies to participate in broader impacts
activities to extend the reach of their research. Natural resource managers and policy makers
seek to use the best available science to develop their plans. Researcher-stakeholder engagement
is one suggested solution to achieve both of these goals. This study uses semi-structured
interviews and an online survey to explore the structure, challenges, and outcomes of one case
modeling current and future water resources in the Willamette basin. Research team members
planned field trips, large and small group workshops, webinars, and written online materials to
engage with water stakeholders throughout the course of the project. Challenges they
encountered included a lack of a shared vision, different professional languages, research
complexities, and project management. Outcomes of stakeholder engagement included
overcoming challenges, facilitating learning, greater understanding, conversation among diverse
perspectives, and improving and extending research results. Participation in researcherstakeholder events was positively correlated with beneficial broader impact outcomes. This paper
concludes with recommendations for a process to promote those outcomes.
Keywords: broader impacts, stakeholder engagement, natural resources, climate change,
modeling
Introduction
In 1997, the National Science Foundation (NSF) established Intellectual Merit and
Broader Impacts as two equal merit criteria for proposal evaluation. The Intellectual Merit
criterion evaluates a project’s potential to advance knowledge within and across scientific
disciplines. The Broader Impacts criterion evaluates a project’s potential to benefit society and to
achieve specific societal outcomes. In 2011, a task force confirmed that the Broader Impacts
criterion is important to enhance scientific literacy and to benefit society (National Science
65
Board, 2011). To aid researchers in developing Broader Impacts activities for their proposals and
reviewers evaluating proposals, the National Science Board produced a Broader Impacts guide.
This guide askes, to what extent does the proposed activity:
•
Advance discovery and understanding while promoting teaching, training, and
learning?
•
Broaden participation of underrepresented groups?
•
Enhance the infrastructure for research and education, such as facilities,
instrumentation, networks, and partnerships?
•
Broadly disseminate results to enhance scientific and technological
understanding?
•
Benefit society?
(National Science Board, 2011)
Despite these guidelines, the Broader Impacts criterion remains under scrutiny.
Reviewers maintain that the Broader Impacts guidelines are not as clear or consistent across
projects and institutions as the Intellectual Merit criterion and therefore are more difficult to
assess (National Science Board, 2011). And although researchers have enjoyed engaging in
broader impact activities (Pearson et al., 1997), they struggle to implement them and engage
various publics in creative ways. One study of NSF-funded projects found that only 65% of
projects had broader impact statements and that among those, 19% only included one of five
possible activity categories (Nadkarni & Stasch, 2013). Teaching/training was the most
frequently utilized method for broader impacts followed by broad dissemination of results
(Nadkarni & Stasch, 2013). Teaching classes, supporting graduate students, and sharing results
online (university websites, lab blogs, etc.) are familiar methods, tangibly beneficial to academic
researchers, and requiring little effort. However, there is an emergent paradigm through the two
integrated and interdependent NSF proposal criteria which creates an expectation that scientists
and science stakeholders engage in research together for their mutual benefit (Frodeman et al.,
2013).
This shift comes at a time where the natural resource management and policy climate is
seeking to incorporate the “best available science” (Lester et al., 2010). Climate change and the
associated natural resource management impacts exhibit the characteristics of a wicked problem.
66
There are no direct and immediate tests for solutions to the suite of climate change problems.
These problems can be explained in many ways, including as symptoms of one another. The
associated stakes are high and there is not a shared definition of the problem being faced (Rittel
& Webber, 1973). Global climate forecasts (Rayner et al., 2005), biodiversity risk assessments
(Brainard et al., 2013; McClure et al., 2013), and urban water use as it relates to precipitation and
temperature (Chang et al., 2014) are among the numerous studies intended to inform natural
resource managers and policy makers.
Yet there remains a large amount of uncertainty and inability to incorporate climate
change and natural resource research into management and policies. Managers and policy
makers cite reasons such as uncertainty, conflicting priorities, institutional limitations,
miscommunication or lack of effective communication, differing values, and lack of locally
relevant results (Callahan et al., 2013; Gregory et al., 2013; Rayner et al., 2005; Smith et al.,
2009; Weible & Sabatier, 2009; Yang et al., 2013) among the reasons for which they have not
utilized scientific information. The global climate impacts and the challenging uncertainty
surrounding them (Lawler et al., 2010) suggest that an extended systems-approach may be useful
to address natural resource management questions in the face of climate change.
There is a need for research scientists and non-scientists to collaborate to achieve their
interdependent goals. The NSF awards funding to research projects which exhibit strong
potential for broader impacts. Research managers are expected to incorporate science and
climate change adaptation into planning and practices (Halofsky et al., 2011). Research able to
affect change incorporates some degree of user involvement and cultivates user trust in the
research (Riley et al., 2011). To achieve broader impacts and be utilized, research must be
credible, salient, and legitimate (Cash et al., 2003). Transdisciplinary research, engaging
researchers of multiple disciplines and science stakeholders, is one proposed method to produce
credible, salient, and legitimate results.
Transdisciplinary research is one of several integrative approaches to research to address
real-world issues that do not fit neatly within one discipline (Dewulf et al., 2007).
Multidisciplinary research allows for consultation among many disciplines for a project but does
not necessarily require integrating the disciplines to achieve results (Schneider, 1997).
Interdisciplinary research creates connections between and among the disciplines (Mader et al.,
2013) “to tackle problems whose solutions cannot be achieved by any single discipline” (Lemos
67
& Morehouse, 2006, p. 62). However, “since intellectual merit and broader impacts are now cast
as integrated and interdependent criteria within NSF’s review process, there is some expectation
that scientists and stakeholders are both engaged in the research enterprise and mutually benefit
from it” (Frodeman et al., 2013, p. 153). Transdisciplinary research creates connections among
multiple disciplines, but extends those connections to include multiple “practice actors” (Lang et
al., 2012). For the purpose of this paper, transdisciplinary research will follow Lang et al.'s
(2012) definition as:
“a reflexive, integrative, method-driven scientific principle aiming at the solution
or transition of societal problems and concurrently of related scientific problems
by differentiating and integrating knowledge from various scientific and societal
bodies of knowledge” (p. 26).
Collaborative natural scientific research can be organized as falling within one or more of
the following four categories: sustainability science, climate change adaptive management,
decision-support tool construction, and/or alternative future exploration. Sustainability science
questions are driven by societal issues and seek to understand coupled social-ecological systems
and address uncertainty. The goal of sustainability science studies is to formulate policy
recommendations based on assessments of all parts of a coupled human natural system
(Kastenhofer et al., 2011). Climate change adaptive management research facilitates
collaborative planning around climate change and is inherently linked to policy (Mackenzie et
al., 2012). Adaptive management focuses on learning by acquiring facts, arriving at new
understandings (Tim Lynam et al., 2010), and adjusting policies to reflect new conditions
(Lawler et al., 2010). Decision-support tool construction research seeks to make integrated
management more tangible (Holzkämper et al., 2012) for an impending management or policy
decision. This research is focused on constructing credible, accurate, understandable, and
appropriate natural resource models to address the issue at hand (Holman et al., 2008).
Alternative future exploration brackets uncertainty by comparing multiple scenarios for topics of
interest (Swart et al., 2004). This research focuses on producing physically and politically
plausible scenarios (Mahmoud et al., 2009; Santelmann et al., 2001) to facilitate discussions and
visualize potential futures.
Any natural resource research collaboration may use stakeholder engagement as a
method to produce societally relevant results and to achieve broader impacts. Stakeholder
engagement involves working with groups of overlapping geographic or subject interests to
68
exchange or create knowledge to improve science and influence societal practices (Mackenzie et
al., 2012). For effective climate change decisions, the National Research Council called for
direct engagement between scientists and their stakeholders (National Research Council, 2006).
In their review of multiple research efforts, Dilling and Lemos (2011) found that useable science
is a function of the context of its intended use and the process followed to produce the science.
Nearly all cases producing usable science followed an iterative process between scientists and
science stakeholders. Cases of collaborative researcher-stakeholder engagement are reviewed in
Ferguson et al. (2015, in preparation), demonstrating the many contexts and methods in which
stakeholder engagement in scientific research is practiced.
Many of the cases in this review offer lessons learned and the impacts resulting from
their experience engaging stakeholders in scientific research. Table 3.1 summarizes the general
lessons learned, necessary elements, and impacts across the cases. Although helpful, very few of
the cases offer a thorough exploration of the participant experience in a researcher-stakeholder
engagement process. Most of the lessons learned are reflections made by the authors after
specific engagement events or following the conclusion of the process. Some cases make claims
about the impacts of their engagement processes without offering evidence within the paper to
support their conclusions. Absent from all of the cases is an assessment of the stakeholder
perspective following an engagement process.
This study seeks to address these gaps in knowledge surrounding stakeholder engagement
processes in natural resource research by characterizing the perceptions of all participants in a
researcher-stakeholder engagement process and by providing evidence for the impacts of
participating in such a process. This study asks: (a) what was the structure of the researcher-stakeholder engagement process; (b) what challenges did the research project and stakeholder
engagement process encounter; and, (c) what did the researcher-stakeholder engagement process
achieve? This study will identify barriers and pathways to success to offer recommendations for
improved natural resource researcher-stakeholder engagement and broader impacts.
69
Table 3.1. Lessons learned and impacts from previous cases of stakeholder engagement in transdisciplinary research.
Lessons learned
Clear roles and responsibilities
Allocate resources well
Be sensitive to stakeholder needs
Consider relationship to research funders
Focus on process rather than product
Accept uncertainty
Accept external expertise as credible
Engage early
Integrate qualitative and quantitative knowledge
Manage both stakeholder engagement and interdisciplinary
portions
Produce non-normative publications
Make use of existing relationships
Necessary elements
Strong leadership
Collaborative research team
Mutual trust
Commitment to project
Transparency
Iterativity
Source
Lang et al., 2012; Mackenzie et al., 2012; Matso & Becker, 2014; Voinov
& Bousquet, 2010
Becu, Neef, Schreinemachers, & Sangkapitux, 2008; Kearney, Berkes,
Charles, & Wiber, 2007; Kloprogge & van der Sluijs, 2006; Lemos &
Morehouse, 2006; Mackenzie et al., 2012; Matso & Becker, 2014
Kloprogge & van der Sluijs, 2006; Lang et al., 2012; Lemos &
Morehouse, 2006; Mackenzie et al., 2012
Mackenzie et al., 2012
Dilling & Lemos, 2011; Kearney et al., 2007; Lautenbach, Berlekamp,
Graf, Seppelt, & Matthies, 2009; Voinov & Bousquet, 2010
Holzkämper, Kumar, Surridge, Paetzold, & Lerner, 2012; Voinov &
Bousquet, 2010
Mackenzie et al., 2012
Holman et al., 2008; Matso & Becker, 2014
Cross, McCarthy, Garfin, Gori, & Enquist, 2013
Daniell et al., 2010; Huntington et al., 2002; Lemos & Morehouse, 2006;
Matso & Becker, 2014
Leydesdorff & Ward, 2005
Huntington et al., 2002
Lemos & Morehouse, 2006; Manring, 2014; Sol et al., 2013
Dilling & Lemos, 2011; Kearney et al., 2007; Lang et al., 2012; Lemos &
Morehouse, 2006; Manring, 2014
Kloprogge & van der Sluijs, 2006; Lemos & Morehouse, 2006;
Mackenzie et al., 2012; Mader et al., 2013; Sol et al., 2013; Voinov &
Bousquet, 2010
Kearney et al., 2007; Sol et al., 2013
Johnson, 2011; Lang et al., 2012; Voinov & Bousquet, 2010
Dilling & Lemos, 2011; Halofsky et al., 2011; Holman et al., 2008; Lang
et al., 2012; Swart, Raskin, & Robinson, 2004; Voinov & Bousquet, 2010
70
Table 3.1. Lessons learned and impacts from previous cases of stakeholder engagement in transdisciplinary research. (Continued)
Untraditional metrics of success
Mid-size, diverse group
Shared reframing of issue/plan/goal
Facilitators/Boundary organizations
Visualizations
Frequent interaction
Impacts
Learn from one another
Improve understanding
Visualize future
Increased credibility
Incorporate managerial knowledge (accurate,
accessible, appropriate research)
Network building
Increase stakeholder self-efficacy
Future research emerges
Diverse dialogue
Increased legitimacy
Increased saliency
Mackenzie et al., 2012; Voinov & Bousquet, 2010
Bartels et al., 2013; Swart et al., 2004; Voinov & Bousquet, 2010
Dewulf, François, Pahl-wostl, & Taillieu, 2007; Fuller, 2011; Halofsky et al., 2011;
Kearney et al., 2007; Lang et al., 2012; Lautenbach et al., 2009; Lemos & Morehouse,
2006; Mackenzie et al., 2012; Matso & Becker, 2014; Sol et al., 2013
Cash et al., 2003; Dilling & Lemos, 2011; Johnson, 2011; Kearney et al., 2007;
Mackenzie et al., 2012; Robinson & Wallington, 2012; Sol et al., 2013
Sheppard et al., 2011
Johnson, 2011; Kloprogge & van der Sluijs, 2006; Lemos & Morehouse, 2006; Mader
et al., 2013
Bartels et al., 2013; Becu et al., 2008; Huntington et al., 2002; Lienert, Monstadt, &
Truffer, 2006; Tim Lynam, Drewry, Higham, & Mitchell, 2010; Manring, 2014;
Stubbs & Lemon, 2001
Becu et al., 2008; Cross, McCarthy, Garfin, Gori, & Enquist, 2013; Lienert et al., 2006
Becu et al., 2008; Lienert et al., 2006
Baker et al., 2004; Cash et al., 2003; Holman et al., 2008; Holzkämper et al., 2012;
Tim Lynam et al., 2010
Baker et al., 2004; Holman et al., 2008; Tim Lynam et al., 2010
Becu et al., 2008; Cross et al., 2013; Holzkämper et al., 2012; Leydesdorff & Ward,
2005; Manring, 2014; Stubbs & Lemon, 2001
Baker et al., 2004; Sheppard et al., 2011
Bartels et al., 2013; Becu et al., 2008; Halofsky et al., 2011
Becu et al., 2008; Cross et al., 2013; Halofsky et al., 2011; Huntington et al., 2002
Cash et al., 2003; Fuller, 2011
Cash et al., 2003
71
Methods
This study takes a mixed methods approach to provide an in-depth case study analysis
(Berg & Lune, 2012) of one natural resource researcher-stakeholder engagement process,
Willamette Water 2100 (WW2100). WW2100 seeks to predict when and where water scarcity
might occur in the Willamette Valley, Oregon through the year 2100 as a result of climate
change, population growth, and water use decisions. Three universities and 26 principal
investigators representing multiple disciplines collaborated to create a model which integrates
sub-models of the processes in the Willamette watershed. These include hydrology, ecological
engineering, climate science, snow science, applied economics, environmental engineering,
water resource, forest ecology, fish and wildlife, and law. In addition to forming an
interdisciplinary team, WW2100 has engaged with approximately 215 water stakeholders
representing state and federal agencies, policymaking groups, non-governmental organizations,
and private industry throughout the research process. This case provides an example of a
transdisciplinary natural resource research project which uses stakeholder engagement to create
usable science and to achieve broader impacts.
An exploratory sequential mixed methods grounded theory approach was used to analyze
the WW2100 case (Creswell, 2003; Glaser & Strauss, 2009). First, semi-structured interviews
were conducted to develop an understanding of the views of key researcher-stakeholder
engagement process participants (Creswell, 2003). Then, a survey of all participants in the
researcher-stakeholder engagement process was conducted to provide a quantitative assessment
of the perception of the entire process. By utilizing two methods, data types, and data sources,
this study increased its reliability and validity (Creswell, 2003) for a robust analysis of the case.
Qualitative semi-structured interviews
Twenty-six semi-structured interviews were conducted in the fifth and final year of the
researcher-stakeholder engagement process. Interviewees were purposively selected (Patton,
2002) based on their participation in the process and representativeness of various expertise to
complete the “en route” reflection (Daniell et al., 2010). Participants who attended more events,
or who represented a perspective not yet recorded were more likely to be interviewed. Twelve
72
members of the stakeholder group and fourteen participants from the research team agreed to be
interviewed (Table 3.2).
Interviews were conducted between January and March, 2015.Each interview followed a
semi-structured interview guide (Appendix A) with questions intended to prompt reflection on
the process. Each interview lasted on average 55 minutes (range: 26 – 89 minutes) and was
conducted in person, via skype, or via telephone as the interviewee preferred. Interviews were
digitally audio recorded and transcribed using Express Scribe Transcription software to ensure
consistency among interviews and to allow for open-coding analysis (McClellan, MacQueen, &
Neidig, 2003). One interviewee declined to be recorded. In this case, extensive notes were taken
during the conversation including verbatim transcriptions and shorthand conversation themes.
The resulting transcripts were sent to interviewees as a form of “member checking” (Miles,
Huberman, & Saldana, 2014) to ensure transcription accuracy.
Table 3.2. Representation, expertise, and number of participants interviewed.
Stakeholder
Research Team
1 tribal representative
1 farmer
1 water utility manager
1 state agricultural agency representative
1 state water agency representative
1 private technology industry representative
2 federal reservoir agency representatives
1 federal forest agency representative
1 county government representative
1 irrigation district manager
1 city water agency representative
1 economist
4 Broader Impact Team members
1 landscape architecture researcher
1 lawyer
1 climate scientist
3 hydrologist
1 ecohydrologist
2 biological and ecological engineers
Following transcription, interviews were analyzed through an open coding process,
facilitated by the computer software MaxQDA. Open coding analysis began with the research
concerns, isolated relevant text from the raw interview, identified repeating ideas, grouped the
ideas into themes, and the themes into concepts (Auerbach & Silverstein, 2003). To ensure
73
validity, preliminary results were informally presented to the interviewees and asked for
feedback as a form of member checking (Creswell, 2003). Themes from analysis were checked
for reliability through an inter-coder reliability assessment (Creswell, 2003; Ryan & Bernard,
2003). Notable overlap between independent codebooks indicates reliable themes.
Quantitative survey
The quantitative phase used a census design (Vaske, 2008) to survey all members of the
WW2100 listserv. Subjects were invited via e-mail to complete an online questionnaire
administered using the Qualtrics website. Of 281 subjects invited, 45 research team members and
92 stakeholders (total = 137) responded for a 49% response rate. Acceptable response rates range
from 35% to 70% (Vaske, 2008) but a wave analysis was conducted to check for response bias.
Average weekly survey return items did not change over time and, given the response rate,
conclusions about the WW2100 participant population can be inferred with 90% confidence
(Creswell, 2003; Vaske, 2008).
The online questionnaire was developed based on previous questionnaires administered
by WW2100 and a preliminary content analysis of the semi-structured interviews. Respondents
to the questionnaire reported their participation in WW2100 and their perceptions of the
outcomes of WW2100. Respondents also were self-assigned to one of two groups, research team
member or stakeholder. This designation served as an independent variable in further analyses.
Data were analyzed using SPSS statistical analysis software. When appropriate, nonparametric
statistical tests were used for a conservative estimate of significance and relationship of the
sample.
Several indices were calculated from survey responses and used in later analyses. An
index of overall participation was calculated from several questions regarding participation in
WW2100 (Appendix H, table H.1). A communication participation score was first computed by
calculating the mean of five variables regarding the frequency of participation in various types of
project communication measured on an 8-point scale from 1 “never” to 8 “daily.” This score was
then combined with survey responses for the number of years involved in the project (0-5),
number of events attended (0-9), and number of webinars attended (0-11). Because these
variables are on different scales, an overall participation index was calculated using the
standardized z-scores of the contributing variables and was used as an independent variable in
74
further analyses. Similarly, indices for model utility, process utility, feeling heard, and model
understanding were calculated from variables designated in the questionnaire designed to
illuminate these concepts (Appendix H, table H.2). These indices were calculated from variables
on 5-point scales from 1 “strongly disagree” to 5 “strongly agree” and used as dependent
variables in further analyses of researcher-stakeholder engagement process outcomes.
Model utility, process utility, feeling heard, model understanding, and overall
participation indices first were assessed for reliability using Cronbach alpha reliability analysis.
Spearman rho (rs) correlations were then used to assess the relationship between overall
participation and model utility, process utility, feeling heard, and model understanding. In all of
the above analyses, statistical significance was tested at a .05 level
Results
This section presents interviewee and survey respondent perceptions of the researcherstakeholder engagement process structure, its challenges, and its successes. A summary of the
results explored below is presented in table 3.4.
Researcher-stakeholder engagement process structure
WW2100 engaged researchers and stakeholders from beginning to end of the research
project. A small group of principal investigators formed the Broader Impacts Team (BIT) whose
focus was to facilitate the stakeholder engagement process throughout. The BIT utilized various
formats to connect researchers and stakeholders with each other. Interviewees commenting on
the structure of the process emphasized the important role of the BIT and outlined the different
engagement stages as they perceived them.
The BIT played an important role in researcher-stakeholder engagement success. They
achieved this first by providing a forum for researcher-stakeholder interaction, coordinating the
necessary logistics for event planning. Then, members of the BIT facilitated communication
between the two groups during and between official events. The efforts of the BIT were not lost
to process participants and interviewees expressed how important having a team devoted to
stakeholder engagement was for the project’s success.
75
A team of principal investigators dedicated to stakeholder engagement ensured that there
was stakeholder engagement throughout the process and that adequate resources were allocated
to make it a success. One research team member stated the importance of the BIT in establishing
the researcher-stakeholder engagement process as a principal component of WW2100.
“The broader impacts team is really the group that set up all of the interactions
with the stakeholders and managed the stakeholder participation. I think that
there might have been more push back about including them but the broader
impacts team really made a lot of noise to make sure that this piece was integral
in the project.”
Often overlooked, coordinating a well-run event for research team members and
stakeholders is critical for continued engagement throughout the process. One stakeholder
expressed a desire to compliment the BIT for their work to coordinate a pleasant meeting, from
the location to the refreshments, and spoke of the consequences if such attention had not been
paid.
“Where the meetings are held is just one of the best places for this kind of
meeting I’ve ever seen. I don’t know how much planning there is that went into it
but I think that it did a very good job. I’d like to compliment whoever put it all
together…People start not wanting to show up if it’s in a venue that’s not nice. I
just think this one was done right.”
The BIT ensured that the research team dedicated time and resources to engage with stakeholders
and then organized events in such a way that stakeholders felt respected and valued to continue
attending.
Over the course of the project, different formats for interaction were used to connect
researchers and stakeholders (Table 3.3). Formal in-person events including field trips, large
group meetings, and small group meetings were held. Informal conversations and e-mails among
participants resulted from the formal events. The BIT also produced outreach materials such as
newsletters, webinars, and a website.
In the first year of the project, researchers and stakeholders were introduced to the
Willamette basin and its multiple water sources and demands through three field trips. These
field trips provided a good introduction to the topic and to each other, as well as a shared
experience from which to draw in future conversations. As a research team member recalled:
“I remember the first field trip that [a BIT member] organized was really useful.
And it got us thinking about a number of things; things that weren’t on our radar
screen, especially about the reservoirs and the management of the reservoirs and
a number of other things.”
76
Stakeholders also saw the benefit in taking time out of their days to tour the Willamette basin.
“I think the project was smart to take people out to some of these sites and see
some of the stuff with their own eyes and get a feel for it. I think there’s real
benefit associated with that. So I think the field trip component, although it’s a lot
of work, a lot of effort involved, I do think there’s a lot of value in doing some real
targeted field trips like the project did.”
Field trips primed the research and stakeholder groups for future interaction and sparked
conversations and thoughts for the modeling project and for future interactions.
Table 3.3. Summary of researcher-stakeholder engagement formats in WW2100. Adapted from
Wright et al., 2015)
Group
Learning Action Network (LAN) - ~215
self-identified listserv participants; 120
people attended at least one WW2100 event
Technical Advisory Group (TAG)
group of ~25 professionals chosen by
Research Team based on their expertise,
constituency affiliation, and representation;
charged with defining assumptions of two
stakeholder scenarios.
Regional Outreach – regional audiences of
water managers, policy makers and the
public
Outreach and Feedback Strategy
Field trips, workshops and webinars designed to foster
interaction and shared learning between researchers and
stakeholders, and provide regional feedback on model
and scenario design (project years 1-5; 2011-2015).
Six half-day meetings in project year 5 (2014-2015), as
well as phone calls, and emails on specific questions;
provided specific quantities for scenario assumptions,
and judgments on future land and water use policies and
practices.
35 invited presentations on the Willamette water system
and the WW2100 project; many invitations stemmed
from connections through the LAN and TAG.
Development of a project website with an overview and
an archive of materials from project outreach events
and recorded presentations –
http://water.oregonstate.edu/ww2100 .
In the second, third, and fourth year of the project, annual large group meetings (36-74
attendants) were held in a centralized location. These “Learning and Action Network” (LAN)
meetings were intended to continue the researcher-stakeholder conversation surrounding water
scarcity, to present research developments, and to solicit stakeholder feedback on the project’s
progress. Although some participants found the LAN meetings useful in their own right, others
did not speak as highly of them. One stakeholder commented on the session organization, saying
that it was “more of a presentation style in an auditorium so it wasn’t designed where they were
requesting a lot of dialogue and feedback.” Another stakeholder was not interested in the
broader conversation.
77
“There was a lot of foolishness in the early sessions. Feel good stuff. It didn’t get
down to the nuts and bolts. I didn’t care what the definition of conservation is or
whatever. Let’s get down to business.”
LAN large group meetings were composed of researcher plenary presentations, researcher poster
sessions, and facilitated breakout sessions. Interviewees expressed that the facilitated sessions
were the most effective and valuable component of LAN meetings.
In the fifth project year, six Technical Advisory Group (TAG) meetings were held. These
meetings were attended by members of the BIT, some research team member, and ~25
stakeholders representing diverse water use and regulation interests. One research team member
described the TAG meetings in this way:
“As the project has evolved, the shift has been more…towards working with
particularly the TAG group to define what we’ve been calling the stakeholder
scenarios. And so that’s a very different type of meeting. Those tend to be more
focused. They tend to be a little bit more directed to achieving a different
outcome. They tend to be a little bit more down in the weeds than the original
meetings were.”
Stakeholders also noted and appreciated the transition away from the large group presentation
format to a more dialogic interaction with research team members.
“That third phase, where they’re soliciting our input into the last runs of the
model, they’ve been very responsive and very open to what we have to say to
reality check the numbers in what they’re doing and how they’re running it. So
nice transition.”
The mention of researcher-stakeholder engagement process phases demonstrates how
participants noted the distinction among the three methods. Several interviewees discussed the
process in terms of its phases. The activities, meeting components, and their roles as attendees
shifted with the format. Having various formats for engagement benefitted the project for “all
three deliver different things.”
Less formal researcher-stakeholder interaction was also a part of this process. First, an
online presence maintained an “e-connection” between the research and its stakeholders
throughout. Stakeholders “appreciated the opportunity to do the webinars that have come out”
but found themselves somewhat overwhelmed by the number of e-mails. Informal conversations
also played a key role in the engagement process. One researcher described his experience in this
way.
78
“We got a request from some of the ag folks in the TAG meeting to talk about how
we were approaching planting dates, when farmers can get in the fields, stuff like
that. So we sat down with them and had a conversation about what they thought it
should look like based on their personal experience and we made changes in the
model to reflect that.”
Informal conversations, e-mails, meetings over breakfast, and telephone calls were the way
stakeholders were able to influence the research in a tangible way. However, these would not
have been possible without the connections established by the field trips, large group, and small
group meetings.
The coordination of the different types of events can be considered the structure built by
the BIT to support stakeholder engagement. However, individuals within the BIT worked as glue
to hold the structure together, facilitating conversations among participants during and between
events. One interviewee confirmed this important BIT role saying: “If [BIT member] hadn’t been
involved, there would have been no public input at all. I’m convinced of that.”
Eighteen of twenty-six interviewed confirmed the BIT’s value to the stakeholder
engagement process. They complimented the BIT for coordinating the process, for facilitating
conversations within and without meetings, for unending enthusiasm, and for being attentive to
participants’ needs. One stakeholder commented on the need for and the BIT’s ability to deliver
this unique role.
“There aren't a lot of people that have those kinds of skills, especially the really
technical scientific kind of people or people that are really into data and boxes
and models aren't always those kinds of people. So I think it's important on the
research academic side to realize who has those skills. [A BIT member] always
did a good job in the different workshops following up with people about
concerns…and give you a call and say I heard you had these concerns and what
do you think about this or I'm going to talk to this person.”
Using various formats and techniques was important to the success of the project but without the
BIT to plan and facilitate the researcher-stakeholder interactions, there would not have been a
WW2100 stakeholder engagement process.
Researcher-stakeholder engagement process challenges
Four types of challenges were identified for the WW2100 researcher-stakeholder
engagement process: lack of a shared vision, interdisciplinary-associated challenges, research
complexity, and project management challenges. Interviewees spoke of the potential causes of
79
these challenges and gave examples in which these challenges manifested. Often the challenges
were overcome but not without recognition and effort on the part of participants and project
leadership.
By far, “the biggest challenge [for WW2100] has been this lack of a shared vision in the
project.” Interviewees identified that participants had different research and stakeholder
engagement philosophies, disciplinary traditions, and experience with projects like this.
Combined with an unclear proposal, these project elements contributed to creating multiple
visions for the project. Multiple participants with divergent visions for the project led to
occasionally conflicting actions. Researchers and stakeholders found it difficult to connect with
one another. There was disagreement over when and how to involve stakeholders in the research,
what to do with stakeholder knowledge once obtained, and how to reconcile stakeholder
experiences with evidence found in peer-reviewed literature. At times, it appeared that the
researchers were pursuing different research goals.
Research team members found themselves on research philosophy spectra from applied
to theoretical science, from preferring quantitative data to qualitative data, and from a planning
to a projection approach to alternative futures modeling (Figure 3.1). One research team member
summarized the differences in approaches in this way:
“This Willamette Water 2100…was a pretty serious unresolved collision between
these two approaches in which the dominant designers of methodology came
from, and insisted on, an academic hypothesis-testing single variable at a time
changing approach and emphasized its strengths. Whereas the opportunities…to
couple more tightly to what the stakeholders think is important requires that
multiple assumptions, multiple parameters of the model change all at once so that
you make tests of larger regions of parameter space by changing a half a dozen
parameters at once. And that runs against the career commitments and frames of
reference of traditional scientific approach.”
Some research team members approached WW2100’s questions from a theoretical, predictive
projection perspective and others from an applied, scenario planning perspective. Then when
they came together, conflict arose. Not only were research team members in disagreement with
each other, their varying approaches to research were not aligned with the stakeholder group’s
either. One stakeholder noted how this challenge impacted the TAG in its mission:
“It seemed to be kind of difficult sometimes to develop the scenarios, perhaps
because the research team had different preconceived ideas than what the TAG
members had.”
80
Figure 3.1. Spectra of research approach philosophies.
Previous experience with interdisciplinary research and stakeholder engagement
influenced researcher and stakeholder expectations for the engagement process. Researchers with
previous experience working with stakeholders had a different idea for how the process would go
than those who had never worked with stakeholders beyond general outreach.
“I think some people came in with experience having done it in the past on
particular projects and had an idea but other people came from a really different
point of view and set of experiences and so there was a long time of
struggle…among the research team to figure out what was going to be acceptable
for the stakeholder involvement process.”
Disparate plans for the researcher-stakeholder engagement process were also traced back to the
research proposal.
“At the very beginning project, it was really quite unclear to me what we were
exactly going to end up doing in the project. I think that the proposal itself while
it reads very well,…it’s kind of spread…a little too thin.”
Research team members stated that the research proposal could have more explicitly outlined a
research plan rather than broad research goals so that once funded, the team would already have
the next steps in place.
Due to the different visions for the project, at times the project went in different
directions, causing conflict and requiring reconciliation. In the researcher-stakeholder
engagement process, much of this conflict centered around when and how to involve
stakeholders and their knowledge. Some participants preferred “early and often” stakeholder
engagement, while others preferred to produce results first in order to “have something to talk
81
about.” When stakeholders were asked for feedback on model assumptions and/or scenarios,
there was resistance to their suggestions. In part, this was due to the fact that the model was
already built and difficult to alter at that point.
“I think they started the process with all these workshops and just giving
presentations and… really had already done a lot of the work…and so when
people ask you or tell you that all your assumptions are off, they didn't really
want to go back and redo all their work.”
But there was also debate over how to incorporate stakeholder knowledge into the model and the
research in general. Should the stakeholders guide research questions? Should they vet the model
assumptions? Should stakeholders develop scenarios? Should this model incorporate stakeholder
observations or rely on peer-reviewed literature for data?
“There were academics who essentially were more convinced by peer-reviewed
papers even if the data that went into them were national in scale and therefore of
questionable relevance locally…There was actually a spectrum of affinity for
stakeholders and…I found myself arguing that their own data and modeling
should trump this peer-reviewed paper because of the way that the paper was
generalizing.”
The project also required reconciliation over the direction of the research goals. Without
a shared vision to guide a shared question, participants pursued individual questions for
academic publications or regarding processes or scales beyond what the Envision model was
capable of doing. This created conflict when participants discovered that their questions would
not be addressed in this project or at their desired scale or with their desired application. One
research team member discussed how forming a shared question in these cases can be somewhat
elusive.
“You have to focus on questions that are scientifically interesting. So there’s a
Venn diagram. Society has certain questions they want answered but not every
question society wants answered is really going to make good science and so you
have to try to find some overlap between those two. And it’s a matter of luck and
skill and a bunch of things. I’m not sure I have a plethora of those but hopefully
luck.”
Some of the differences in philosophies, goals, and experiences can be attributed to the
transdisciplinary nature of the research. Research which integrates multiple expertise inherently
brings with it the challenge (and opportunity) of diversity. In addition to different goals and
philosophies, interdisciplinary researchers must meet the challenge associated with having
different languages and working in large groups. This work can be slow moving and long in
82
duration and “it’s long in duration because when you bring together this many disciplines, you
have to figure out how to talk to one another.” Researchers and stakeholders from different
backgrounds struggled to come to a shared meaning of seemingly universal vocabulary words. It
required reconciling not just one or two vocabulary sets but at least twelve. As one researcher
commented on the size of the collaboration, “there's too many people and nobody took
responsibility.” Many interviewees commented on the number of people involved in this project
as a challenge. Some cited that collaborative research teams should not include more than 5-7
people and that large group engagement events only allow for general discussions when
specificity is needed. The number of people and the number of languages presented
interdisciplinary challenges for WW2100.
The complex interaction of all of the people involved was reflected in the complexity of
the research project which posed challenges in and of itself. Participants expressed that the
spatial and temporal extent of the modeling was challenging to achieve and to comprehend.
Some questioned if modeling the entire Willamette Basin was amenable to stakeholder
engagement because there was not “enough connectivity and homogeneity and common purpose
etc. within the stakeholder group for them to function cohesively.” Similarly, modeling and
imagining 85 years into the future was a challenge for many participants.
“It took probably a couple meetings before it really sunk in and started thinking
on that far out into the future. As a farmer businessman, you definitely think into
the future but I’m not sure that we think generations into the future.”
The modeling process proved to be a complex challenge. With thirteen interconnected submodels and 21 future scenarios, participants found the modeling goal ambitious and challenging.
“There is just a ridiculous huge amount of information that was trying to be put
together and to make that all try to talk and to make something come out of that
make any sense whatsoever was just a Herculean challenge.”
Such a large project brought basic project management challenges as well. Interpersonal
relationships and planning logistics were cited as frequent challenges within this category. With
many people involved, possessing different perspectives, it is no surprise that many interviewees
remarked on the challenge of working with different personalities. One person stated plainly,
“There were strong personality challenges.” Managing personality differences required
extensive effort and mediation within the project.
“I’ve had a lot of meetings with people to discuss differences of opinion, to work
out agreements,…to encourage people to work together, to demand people work
83
together, to plead that people work together, to provide resources to allow people
to do things that will then help somebody else.”
Differences aside, merely organizing the amount of people involved in the WW2100 researcherstakeholder engagement process was a logistical challenge. First, “they’re busy, we’re busy,” as
one interviewee stated. Sometimes schedules did not allow the sought-after engagement. This led
to an associated challenge of continuity from one meeting to the next. When there are scheduling
conflicts, the composition of researchers and stakeholders shifts from event to event.
“The dynamic of any particular gathering changes a little bit depending upon
who's there. And you can't have 100% attendance every time.”
The four types of challenges mentioned by key WW2100 participants were the project
lacking a shared vision, interdisciplinary challenges like reconciling different languages and
moving slowly, modeling complexity, and project management challenges. Many of the
challenges outlined above were overcome over the course of the project, the result of which
many interviewees cited as a success.
Research-stakeholder engagement process impacts
Although many challenges emerged from the interviews, a greater number of successes
were cited by interviewees and confirmed by survey respondents. Overcoming the
aforementioned challenges was considered a success by participants. Interviewees celebrated the
research and stakeholder engagement successes of the project. Survey results indicated that
participating in stakeholder engagement events was positively correlated with calculated indices
of an individual’s understanding and perception of the research results’ utility, perception of
feeling heard throughout the process, and value for the stakeholder engagement process.
The most prominent success of a research project is associated with the research
products, tools and publications. Half of the interviewees celebrated WW2100’s success in
building a model of water in the Willamette basin. Participants in WW2100 were proud of the
model they helped construct.
“I feel quite confident saying that in terms of just the model that this is without a
doubt the best look at water in the Willamette Valley that’s ever been done.”
Many researchers and stakeholders were also glad that “there were papers published and
written.” By the traditional metrics of research projects, WW2100 was successful.
84
An individual’s participation in researcher-stakeholder engagement events was “typically
to substantially” (Vaske, 2008) positively correlated with understanding (rs = .42, p < .001) and
“minimally to typically” positively correlated (Vaske, 2008) to perceiving the research results as
valuable (rs = .21, p = .002). Highly engaged participants understood the model and its
limitations and felt that the model provided a broader view of water in the Willamette Valley
while contributing to scientific knowledge and informing resource managers, policy makers, and
users. Engaging with research team members through model construction and scenario
development led stakeholders to understand the results and how they might be useful. “This
process was like wow, it was great, it was interesting but it also helped us understand the tool
more.” Through interaction with each other, stakeholders and researchers were able to identify
what questions the tool could answer and how those answers might be utilized in the future. One
stakeholder said of the project:
“The WW2100 effort would still provide useful information on a relative scale
and would illustrate general effects of different uses on water supply in the
Willamette Basin. The modeling would give us some good indications about
effects of increases in population, land use changes, and climate change.”
Those who participated more in the engagement process expressed ways in which the
results would be useful to scientific discovery and to water management and policy. The model
contributes interesting results within each of its sub-models: snow pack, climate, dam operations,
human water use, forest management, and water rights. The resulting Envision model was seen
by participants as a means to inform future water management projects and as a tool to facilitate
planning discussions. One stakeholder shared her need for the results of this project.
“If we can have a visual representation of possible changes, things to consider or
ponder, I think that’s exactly what we need. Because it’s very hard to visualize all
these changes on the landscape…I think this could be a very powerful planning
tool.”
Another stakeholder shared how the results could contribute to future discussions beyond the
immediate planning needs.
“It tells us the direction we need to go and the things, the issues we need to think
about…It’s a conversation starter. But if you don’t do this analysis, it becomes a random
conversation whereas this at least gives you, hey this is the status quo, nothing happens
or current projection, this is where we’re going to end up.”
WW2100 participants who attended more researcher-stakeholder engagement events expressed
greater understanding of and utility for the Envision model and its results.
85
Individual participation also “typically to substantially” (Vaske, 2008) positively
correlated with perception of feeling heard in the WW2100 research project (rs = .36, p < .001).
Relatedly, interviewees considered the stakeholder engagement process and incorporation of
stakeholder input successful overall. One stakeholder commented on the process’s iterative
nature as witness to how both groups listened to each other.
“I think the people that were gathered there were very interested and engaged.
My impression of how their input was received is that it was well-received I think.
The investigators were very interested in the input they were getting, sought to
clarify what they were hearing and, based on sort of the iterative manner of the
gatherings, it was clear that they took in what was said and kind of dialed those
changes or recommendations in to the best of their ability. So I think it was a high
energy environment for getting people together and with the output being paid
attention to.”
When participants felt heard, they also felt valued. “They seemed like they were accommodated
and got to express their opinions and felt like they were valued in the process.” When people
feel valued and heard, they are more likely to continue participating and the benefits of
participating in the researcher-stakeholder engagement process are extended.
Participating in WW2100 was also “typically to substantially” (Vaske, 2008) positively
correlated with expressing greater value of the stakeholder engagement process (rs = .39, p <
.001). Participants valued the researcher-stakeholder engagement process as a way to form or
strengthen relationships, to understand other water users, to share in a necessary conversation
about water with them, and to personally grow through learning. The process was valuable
because it allowed participants to explore current and future key water issues together.
Participants enjoyed interacting with people they do not normally interact with and considering
water in new ways. One interviewee stated that this aspect of the researcher-stakeholder
engagement process was the most valuable outcome.
“I would say that the discussions and the relationship-building have been more
beneficial to me than the actual nitty gritty numbers that it produces.”
Attending multiple events allowed participants to form and strengthen information and working
networks for collaborative natural resource adaptive management and policy.
In getting to know one another and sharing in conversations, participants arrived at a
better understanding of each other as Willamette basin water users. Stakeholders came to
understand each other’s water concerns better. An industry representative found common ground
with farmers over their economic dependency on water. Environmental flow and utility
86
managers better understood the tradeoffs associated with reservoir operations. Researchers came
to understand each other’s disciplines better and both research team members and stakeholders
developed a better sense of what each other’s priorities were. One researcher commented on the
benefit of engaging in conversations with the stakeholders.
“So it was interesting from a learning perspective to hear from them what the
management concerns are and then to think about the actual logistics of how we
would try to do that, some of them being really easy to evaluate and other ones
not really being feasible.”
Interviewees shared the value of this process as it led water users, research team members and
stakeholders, to better understand one another.
Such understanding would not have been possible without the process to facilitate a
necessary conversation about water in the Willamette basin. More than half of the interviewees
commented on the value of and success engaging diverse perspectives in a constructive dialogue
achieved through this process. They frequently commented on the success of the project in
getting diverse people to attend the events and how their presence enriched the event and the
project outcomes. One stakeholder recalled of the TAG meetings:
“I don’t know how many we had, 20-25 folks, all coming from different
perspectives and having…thoughtful conversation. It’s been a pleasure…working
with the group of folks as the stakeholders and knocking things around.”
Another interviewee commented on how valuable having the dialogue among multiple
perspectives was.
“I think just even having that dialogue amongst the users was probably one of the
most successful parts of the project. Even more I think than what the results will
be of the last two scenarios.”
As with relationships and understanding, individuals who participated more, considered the
process more valuable. More time in the process allowed participants to achieve the benefits that
came from sharing in constructive conversations with people of diverse perspectives.
Finally, all interviewees expressed that the researcher-stakeholder engagement process
was valuable as a learning opportunity. Engaging with the research team, participants were able
to learn about water modeling and technical aspects of different water use and management
sectors. Engaging with stakeholders led to learning about the concerns of each water sector,
leading to greater understanding of the big water picture. Through the researcher-stakeholder
87
engagement process, participants learned how to work together and how they might work
together in the future.
“I have no doubt that there [are] perspectives that people in the stakeholder
group hold now that they didn’t hold coming into this process; that they, in fact,
have learned from others in the group. So I think there’s been some success
there.”
One interviewee summed up the education benefit of participating in WW2100’s researcherstakeholder engagement process: “I’m learning …and that’s always helpful, both personally and
professionally.”
Interviews elucidated still more benefits of participating in the WW2100 researcherstakeholder engagement process. Engaging researchers with stakeholders improved both groups’
operations through researcher-stakeholder give and take. Stakeholders were able to inform the
model and provided necessary feedback to the researchers. One researcher spoke of the benefit
of including stakeholders in research: ‘I think it keeps us from just doing silly things. It keeps us
grounded in the real world.” Stakeholders provided information on their operations, estimates,
and perceptions of reality to inform the model. They also shared their research questions and data
observations with the research team. A research team member commented on the value of
including stakeholders in the research process:
“That one was useful to me because there were [stakeholders] there who asked
some good questions about some of the results that I was plotting, why they
looked different from what they were expecting. So it helped me think about the
problem a little more completely because they looked at the data and…asked
thoughtful questions.”
Stakeholder groups also benefitted from the researcher-stakeholder give and take. Some realized
how their operations could improve to support academic research and most were able to glean
useful information from the results and conversations.
Finally, the WW2100 researcher-stakeholder engagement project directly contributed to
the broader impacts of the research by building the model’s credibility, and training stakeholder
project ambassadors. These were not explicit goals of the process, but naturally resulted
according to the participants interviewed. The process “created a lot more buy-in…from all the
people at the table.” One stakeholder explained:
“you had stakeholders involved from the beginning that could see this process all
the way through. That’s good because otherwise we would’ve just gotten
88
something at the end and not had any idea how it was developed or who was
working on it or why they were asking these questions.”
Through collaboration, everyone involved had a better understanding of the research questions,
the reasons they were asked, the answers, and how the results were obtained. This made the
results more credible. When the results were more credible, those involved were more likely to
share them with others. Interviewees were already discussing the project and results with coworkers not involved in WW2100 at the time of this study. They also expressed an interest in
helping to disseminate the final results when the time came. One interviewee volunteered: “I
would be more than happy to help roll out whatever it is to the key people or help roll it out.”
Others posited that they could be “ambassadors” or “a champion” and “take the output back to
the groups they belong to.”
Analyses of interviews and survey responses yielded a codebook characterizing
challenges and successes of the WW2100 researcher-stakeholder engagement process (Table
3.4). A team dedicated to facilitating the interaction between both groups utilized various
formats to unite diverse actors in a shared conversation about current and future water resources
in the Willamette basin. This process met different challenges including a lack of a shared vision
for the process, interdisciplinary challenges, challenges associated with the complex research
question and approach, and project management challenges. The project also achieved great
successes including overcoming challenges and engaging many diverse people in a constructive
conversation leading to greater understanding of water and each other. Through a newly
developed network, all participants were able to learn and establish connections that may be
helpful in future projects. Finally, the process improved the science itself by integrating
stakeholder concerns and knowledge into the questions asked and the methods used to answer
them, and extended the project’s reach by training more people to represent the results in diverse
water contexts.
89
Table 3.4. Codebook summary of challenges and successes of WW2100 researcher-stakeholder
engagement process.
Challenges
Lack of shared vision
Interdisciplinary
Research complexity
Project management
Successes
Different research goals
Different stakeholder engagement plans
Unclear proposal
Discipline traditions
Lack of experience
Approach to research
Slow-moving
Different languages
Complex modeling
Large spatial scale
Long temporal scale
Interpersonal relationships
Participants are busy
Planning logistics
Overcoming challenges
Building model
Stakeholder participation
Inform scientific research
Learning
Forming and strengthening relationships
Constructive conversation
Build credibility
Train result ambassadors
Discussion
This study characterized the structure, challenges, and successes of one case of
researcher-stakeholder engagement in natural resource future modeling research. WW2100
utilized engagement process structure elements and encountered challenges that are documented
in transdisciplinary research cases. However, the unique combination of structural elements in
WW2100 produced outcomes for a stakeholder-engagement process previously undocumented.
These outcomes fall within the NSF’s five criteria broader impacts framework and suggest an
emerging sixth criterion.
Challenges of Transdisciplinary Projects
Many of the challenges observed in WW2100 are reviewed in similar cases studies (ex.
Lemos & Morehouse, 2006). Several of these case studies also provide suggestions for planning
90
to avoid or overcome such challenges should they arise (ex. Lang et al., 2012). WW2100 shared
its challenges with many studies before it who claim confronting a lack of a shared vision,
interdisciplinary challenges, and project management challenges.
The literature on collaboration and stakeholder engagement emphasizes, if not the
importance of establishing a common goal, the benefits of coming to a shared understanding of
the problem and solutions to it. Knowing the causes and the ways a lack of shared vision
manifested in this project can help other projects avoid it, or diagnose a problem once the
symptoms are recognized. Transdisciplinary natural resource studies need to develop a shared
frame for the issue being explored, question to answer, and how to answer it (Pahl-wostl, Craps,
et al., 2007). Without a shared frame, there will be different goals for a project. Stakeholders will
have different visions of what a plausible future may look like (Baker et al., 2004). Unaddressed
conflicting objectives can lead to distrust and resistance to new ideas in a project (Manring,
2014) and disparate methodological standards can lead to conflict (Lang et al., 2012). The lack of
a shared vision and approach to research was the greatest challenge to WW2100, leading to
disagreement over research questions and methods and stakeholder involvement in research, and
manifesting as conflict in the project. Researchers have disciplinary interests beyond those
defined by the interdisciplinary project (Lemos & Morehouse, 2006) and stakeholders may be
interested in solving a real-world problem. When diverse actors (multiple disciplines and nonacademic participants) work together on these projects it is necessary to make those different
interests transparent. The collaborative group must meet early in the process to develop the
shared goal and a shared understanding of each other (Tim Lynam et al., 2010; Star &
Griesemer, 1989). Without this meeting, the project must overcome the resulting conflict and
move more slowly as those involved arrive at shared understandings in a piecemeal fashion.
When interdisciplinary projects engage with stakeholders in research it is essential that
research team members share a vision for when and how the stakeholders will participate. As
with the lack of a shared research objective, without a shared approach to stakeholder
engagement, the WW2100 research team was challenged by the resulting disagreements over
when to solicit stakeholder input and how to use it. One prior case found that focusing on
objective fact in a study led to neglect of experiential practical knowledge which could have
contributed to innovation (Pahl-wostl, Craps, et al., 2007). In another study researchers were
defensive of their decisions to exclude stakeholder data saying that they were not “scientifically
91
sound” (Kloprogge & van der Sluijs, 2006), a sentiment that some members of WW2100
expressed. Knowledge external to academia must be seen as credible (Mackenzie et al., 2012)
and research team members must be flexible in their disciplines (Lemos & Morehouse, 2006) in
order for this challenge to be overcome.
Interdisciplinary challenges like those observed in WW2100 were also encountered in
previous studies. One study found that there was a greater difference in beliefs and perspectives
among scientists of different disciplines than between scientists and lay stakeholders (Fuller,
2011). WW2100 devoted much time to overcoming interdisciplinary language challenges,
arriving at a shared definition of key terms like “water scarcity” and “stakeholder engagement.”
A researcher’s disciplinary background can influence his or her concept of a common issue, the
meaning of which may not be self-evident to all involved and whose various meanings may not
be neutral (Dewulf et al., 2007). Even the concept of “science” and how it should be practiced is
understood differently among disciplines. This can make cooperation difficult (Fuller, 2011).
Scientists are often comfortable with ambiguity and uncertainty in their results; interdisciplinary
teams should grow more comfortable with ambiguity or uncertainty when it comes to
understanding each other’s methods to explore the multiple dimensions of a research problem
(Dewulf et al., 2007). A quantitative researcher may not understand the procedure for obtaining
qualitative results but that does not mean that they should be dismissed. Rather, they should be
welcomed as another piece of information.
Project management is an often overlooked transdisciplinary challenge. Large study areas
require the participation of many researchers as well as stakeholders to construct an accurate
model and to develop future scenarios (Mahmoud et al., 2009). Stakeholder engagement is time
and resource intensive (Baker et al., 2004; Mackenzie et al., 2012). The WW2100 Broader
Impacts Team encountered the logistical challenges of organizing a large number of participants
to attend many meetings and webinars and to respond to e-mails and phone calls over five years
to engage its stakeholders. If the goal is to produce research results that will contribute to
management and policy, it is important to remember that it takes time, and policy making has
deadlines. In one case, the resulting model was developed too late to directly contribute to the
policy questions (Tim Lynam et al., 2010). Managing project logistics becomes more
challenging as the study area, number of researchers, and number of stakeholders increase.
Resources devoted to the project should increase accordingly.
92
The challenges WW2100 encountered were also present in previous cases. From previous
literature and the reflections of researcher and stakeholder participants, recommendations for
future projects to avoid these challenges can be made.
Impacts of Stakeholder Engagement in Transdisciplinary Projects
Each case in the literature shares lessons learned and successes resulting from interacting
with stakeholders and incorporating stakeholder information, but does not reflect on the impacts
of the process on those involved. Stakeholder engagement (i.e. user involvement) and
information exchange facilitated by trusting relationships were key factors in research projects
able to affect change (Riley et al., 2011). Projects with these elements were able to improve
decision-making in a management community, as well as improve the quality of the research
conducted (Bonney et al., 2009). Stakeholder engagement is one of the proposed ways to
overcome management barriers and, when completed successfully, can lead to science-driven
decisions and action toward conservation, resilience, and sustainability (Shirk et al., 2012). But,
without an impending action decision, it is difficult to measure the impact of stakeholder
engagement in scientific research. This study places the successes reported by WW2100
interviewees and survey respondents in the context of the NSF Broader Impacts review criteria
as a framework for evaluating the impacts of stakeholder engagement.
WW2100 participants identified three kinds of success: research success, process success,
and personal success. Previous literature has emphasized the different ways in which different
participants measure success in transdisciplinary projects. Scientists celebrated publications and
contributing long lasting science as a success while industry members celebrated a study’s
contribution to management in a squid fishery collaboration (Johnson, 2011). Similarly
researchers valued work that allowed for an original contribution to a peer-reviewed journal, but
stakeholders considered a project successful when material improvements resulted from a water
modeling project in Australia (Mackenzie et al., 2012). Some researchers call for “alternative
measures of success” in transdisciplinary projects because the design experience may be more
important than the results themselves (Lautenbach et al., 2009). A project might be called
successful if it resulted in new relationships among participants, improved communication,
accessible knowledge, useful tools, behavior change, or specific societal outcomes (Baker et al.,
2004; Dilling & Lemos, 2011; Lienert et al., 2006). Rather than a dichotomy, where one type of
93
success is more important than another, project success can be considered in terms of NSF’s two
interdependent, and equally valuable, review criteria: intellectual merit and broader impacts
(National Science Board, 2011).
Alternative measures of success in the literature and WW2100’s three types of success
(process, research, personal) provide specific examples for achieving the NSF’s five broader
impacts review criteria and elucidate an emerging sixth criterion. As a result, the impact of
stakeholder engagement in natural resource research can be discussed in terms of each review
criterion: advancing scientific discovery and understanding, broadening participation of
underrepresented groups, enhancing infrastructure for research and education, broadly
disseminating results, and benefitting society. This study identifies specific ways that a
transdisciplinary research project may achieve broader impacts and suggests a sixth criterion:
enrich the research community (Table 3.5).
Stakeholder engagement processes, including that of WW2100, contributed to advancing
scientific discovery and understanding by directly influencing the research, facilitating learning,
and by pushing to achieve research deadlines. The clearest way in which stakeholder engagement
advances scientific discovery is by facilitating the incorporation of stakeholder knowledge and
questions into scientific studies in a way that research teams would not have done alone.
Stakeholders provided feedback on assumptions, output metrics, scenarios, interpretations, and
questions in WW2100. Similarly, stakeholders helped to define process components, methods,
design elements, output indicators, and outcome plausibility in the development of a decisionsupport tool (Holman et al., 2008; Mahmoud et al., 2009). Through WW2100 stakeholder
engagement participants learned about scientific modeling, the research process, each other, and
each other’s respective areas of expertise in a social learning process. Such understanding
extends beyond the individuals involved to a deeper understanding of a larger picture (Reed et
al., 2010). From WW2100, then, one way to advance discovery and understanding is to integrate
new and traditional sources of knowledge and perspectives. Finally, research products were
made for stakeholder engagement events that would not have been made otherwise (Pearson et
al., 1997). In this way, stakeholder engagement processes like WW2100 advance science at a
quicker pace than might otherwise have been achieved.
94
Table 3.5. Revised broader impact (BI) framework and examples from WW2100 outcomes.
BI review criteria
Advance discovery
and understanding
Specifications to criteria
Integrate new and traditional
sources of knowledge and
perspectives
Qualitative data
“Provided a feedback
mechanism on what we’re
doing and whether it’s
reasonable or not”
Quantitative data
Process utility
Diverse participants are
present
“I think the diversity is
remarkable and
respected”
Attendance record
Diverse participants feel
heard
“We’re being listened to”
Feeling heard
Enhance research infrastructure
Build new facilities and
instrumentation
“We’ve built a model”
Process utility
Develop research community
Form and strengthen research
partnerships and networks
“The most meaningful
product of projects like
this is the connections
between people”
Process utility
Broad dissemination
of results
Disseminate results broadly
Train stakeholder
ambassadors of science
“People take the output
back to the groups they
belong to”
Model
understanding
Benefit society
Benefit society
Science users believe results
will benefit society
“It just created a lot more
buy-in from the users”
Model utility
Broaden participation
of underrepresented
groups
Enhance research
infrastructure
Revised BI review criteria
Advance discovery and
understanding
Broaden participation of
underrepresented groups
Model
understanding
95
WW2100 stakeholder engagement broadened participation of underrepresented groups in
research. Survey respondents who participated more in the WW2100 engagement process
expressed a greater perception of feeling heard. Overall interviewees expressed gratitude for
their level of involvement and the level of respect given to participants. The process was also
able to include actors not often associated with scientific research projects like tribal and
agricultural representatives. Exchanging ideas with diverse groups to gain new insights and the
process that facilitates representation of diverse ideas is valuable (Bartels et al., 2013). Research
may broaden participation of underrepresented groups by inviting diverse participants to be
present but also by ensuring that their participants feel heard. This element is undocumented in
previous literature and should be explored more thoroughly in future studies.
Of the five NSF broader impacts review criteria, the one in which WW2100’s stakeholder
engagement process was most successful was enhancing research infrastructure. Interviewees
shared that they felt the most meaningful product of WW2100 was the network of users,
regulators, and researchers that participants were able to form and the discussion that followed
around water resources. A social network of stakeholders and researchers can be an essential
asset in adapting to future changes by increasing adaptive capacity (Pahl-wostl, 2007), social
capital (Leydesdorff & Ward, 2005), and learning (Manring, 2014). Networks achieve this by
forming the basis for necessary natural resource conversations. Participants are able to have an
open and ongoing dialogue of the tradeoffs (Hildén, 2011). As stakeholders join with researchers
to discuss technologies, policies, and worldviews, they build the bridge between various
approaches allowing for systems science to respond to wicked problems (Pahl-wostl, 2007; Rittel
& Webber, 1973). These process values are positively correlated with participation in
WW2100’s stakeholder engagement events.
Although forming partnerships fall within the NSF broader impacts criterion “enhance
research infrastructure,” the prominence strengthening a research network played in participant
evaluations of WW2100’s researcher-stakeholder engagement process leads this study to propose
a sixth criterion to NSF’s broader impact criteria: develop research community. Enhancing
research infrastructure remains an important criterion as it refers to building new facilities and
instrumentation. However, due to the strong emphasis on forming and forming and strengthening
relationships, a project may achieve broader impacts through developing the research community
within and beyond academia.
96
Stakeholder engagement processes also contribute to broadly disseminating research
results. By inviting stakeholders in to the research process, WW2100 researchers built credibility
for their tool. It is necessary to create buy-in for decision-support tools to do what they are
intended to do (Holman et al., 2008; Tim Lynam et al., 2010). This “live peer review” (Halofsky
et al., 2011) or “extended peer review” process “improve[s] the legitimacy, credibility, and
relevance of science, especially in the context…where facts are uncertain, values in dispute,
stakes high, and decisions urgent” (Johnson, 2011, p. 265). Once stakeholders view the research
results as credible, they are more likely to share them with co-workers and apply them in their
respective contexts. After one shared learning experience with researchers, stakeholders became
“climate change extension agents” (Cohen, 2010). Similarly, WW2100 stakeholders expressed
that they would share what they had learned with others beyond the research group. Training
stakeholder ambassadors of science will help a project to disseminate its results broadly.
The fifth broader impacts criterion, benefit society, is the broadest and perhaps most
difficult to define and measure. This study’s results suggest that one way to operationalize this
criterion is by ensuring that the users of science believe the research results will benefit society.
In WW2100, participant perceptions of the research results utility served as a proxy for how well
the research would benefit society. WW2100 interviewees identified how the research results
might be useful to them but they were also acutely aware of their limitations. With greater
understanding of the science comes greater understanding of its limitations. There may have
been unrealistic assumptions (Tim Lynam et al., 2010) or unrealistic demands on the research
(Dilling & Lemos, 2011) leading to perceptions of the model’s limitations. However, participants
valued the model as a useful way to understand the current processes (Santelmann et al. 2001)
and to provide a basis for a discussion on future water resources. As in all other categories,
participation in stakeholder engagement events was positively correlated with a perception of the
usefulness of the model and its results.
Throughout this discussion there is a link between individual participation in researcherstakeholder engagement events and contribution to achieving broader impact goals. Interaction
between researchers and science stakeholders can improve understanding of each other (Dilling
& Lemos, 2011) and create trust, commitment, arriving at a shared understanding of a problem
and its solutions (Sol et al., 2013), and network formation (Mader et al., 2013). One study found
that continued and intense interaction between researchers and stakeholders increased the
97
likelihood of research utilization (Landry, Amara, & Lamari, 2001). The degree to which
individuals are involved in scientific research ultimately influences project outcomes (Shirk et
al., 2012). Participation, then, is a key contributor to any stakeholder engagement-derived project
success. As explored above, there are at least five broad categories of broader impact success
which may depend on successful stakeholder engagement.
Conclusions
Researcher-stakeholder engagement processes are an effective method to achieve
research broader impacts and to answer wicked natural resource questions now and into the
future. This case study reports the perceptions of participants in a researcher-stakeholder
engagement process to explore its challenges and impacts as they relate to previous case studies
and the NSF broader impacts review criteria. There are a few recommendations for future
projects that this study would add to the list of previous lessons learned (Table 3.1).
•
Host conversations early in the process to arrive at consensus regarding the project’s
questions and methods to answer them.
•
Approach stakeholder engagement seeking to learn and with a willingness to tolerate
ambiguity.
•
Dedicate resources (time, money, personnel) to increasing participation in meaningful
stakeholder engagement events.
Researcher-stakeholder engagement processes in progress can use these recommendations to
guide formative evaluations of their process. Researchers may also utilize the broader impacts
review criteria to structure evaluations of their project outcomes.
98
References
Auerbach, C. F., & Silverstein, L. B. (2003). Qualitative data: An introduction to coding and
analysis. NYU Press.
Baker, J. P., Hulse, D. W., Gregory, S. V, White, D., Van Sickle, J., Berger, P. A., …
Schumaker, N. H. (2004). Alternative Futures for the Willamette River Basin , Oregon.
Ecological Applications, 14(2), 313–324.
Bartels, W. L., Furman, C. A., Diehl, D. C., Royce, F. S., Dourte, D. R., Ortiz, B. V., … Jones, J.
W. (2013). Warming up to climate change: a participatory approach to engaging with
agricultural stakeholders in the Southeast US. Regional Environmental Change, 13, S45–
S55. doi:10.1007/s10113-012-0371-9
Becu, N., Neef, A., Schreinemachers, P., & Sangkapitux, C. (2008). Participatory computer
simulation to support collective decision-making: Potential and limits of stakeholder
involvement. Land Use Policy, 25(4), 498–509. doi:10.1016/j.landusepol.2007.11.002
Berg, B. L. & Lune, H. 2012. Qualitative research methods for the social sciences. 8th ed. Upper
Saddle River, NJ: Pearson.
Brainard, R. E., Weijerman, M., Eakin, C. M., McElhany, P., Miller, M. W., Patterson, M., …
Birkeland, C. (2013). Incorporating climate and ocean change into extinction risk
assessments for 82 coral species. Conservation Biology : The Journal of the Society for
Conservation Biology, 27(6), 1169–78. doi:10.1111/cobi.12171
Callahan, B., Miles, E., & Fluharty, D. (2013). Policy implications of climate forecasts for water
resources management in the Pacific Northwest. Policy Sciences, 32(3), 269–293.
Cash, D. W., Clark, W. C., Alcock, F., Dickson, N. M., Eckley, N., Guston, D. H., … Mitchell,
R. B. (2003). Knowledge systems for sustainable development. Proceedings of the National
Academy of Sciences of the United States of America, 100(14), 8086–8091.
doi:10.1073/pnas.1231332100
Chang, H., Praskievicz, S., & Parandvash, H. (2014). Sensitivity of urban water consumption to
weather and climate variability at multiple temporal scales: The case of Portland, Oregon.
International Journal of Geospatial and Environmental Research, 1(1), Article 7.
Clarke, S., & Roome, N. (1999). Sustainable business: Learning-action networks as
organizational assets. Business Strategy and the Environment, 8, 296–310.
Cohen, S. J. (2010). From observer to extension agent—using research experiences to enable
proactive response to climate change. Climatic Change, 100(1), 131–135.
doi:10.1007/s10584-010-9811-z
99
Creswell, J. W. 2003. Research design: Qualitative, quantitative, and mixed-methods
approaches. Thousand Oaks, CA: Sage.
Cross, M. S., McCarthy, P. D., Garfin, G., Gori, D., & Enquist, C. (2013). Accelerating
adaptation of natural resource management to address climate change. Conservation
Biology : The Journal of the Society for Conservation Biology, 27(1), 4–13.
doi:10.1111/j.1523-1739.2012.01954.x
Daniell, K. A., White, I., Ferrand, N., Ribarova, I. S., Coad, P., Rougier, J. E., … Burn, S.
(2010). Co-engineering participatory water management processes: Theory and insights
from Australian and Bulgarian interventions. Ecology and Society, 15(4), 11. Retrieved
from http://www.ecologyandsociety.org/vol15/iss4/art11/
Dewulf, A., François, G., Pahl-wostl, C., & Taillieu, T. (2007). A framing approach to crossdisciplinary research collaboration: Experiences from a large-scale research project on
adaptive water management. Ecology and Society, 12(2), 14.
Dietz, T., Ostrom, E., & Stern, P. C. (2003). The struggle to govern the commons. Science (New
York, N.Y.), 302(5652), 1907–12. doi:10.1126/science.1091015
Dilling, L., & Lemos, M. C. (2011). Creating usable science: Opportunities and constraints for
climate knowledge use and their implications for science policy. Global Environmental
Change, 21(2), 680–689. doi:10.1016/j.gloenvcha.2010.11.006
Eccles, J. S., & Wigfield, A. (2002). Motivational beliefs, values, and goals. Annual Review of
Psychology, 53, 109 – 132.
Farkas, N. (1999). Dutch Science Shops: Matching community needs with university R & D.
Science Studies, 2, 33–47.
Ferguson, L. F. (2015, in preparation). Collaborative Science-Stakeholder Engagement.
Unpublished annotated bibliography.
Freitag, A. (2014). Naming, framing, and blaming: Exploring ways of knowing in the
deceptively simple question “What is water quality?” Human Ecology, 42, 325–337.
doi:10.1007/s10745-014-9649-5
Frodeman, R., Holbrook, J. B., Bourexis, P. S., Cook, S. B., Diederick, L., & Tankersley, R. A.
(2013). Broader Impacts 2.0: Seeing - and Seizing—the Opportunity. Bioscience, 63(3),
153–155. doi:10.1525/bio.2013.63.3.2
Fuller, B. (2011). Enabling problem-solving between science and politics in water conflicts:
impasses and breakthroughs in the Everglades, Florida, USA. Hydrological Sciences
Journal, 56(4), 576–587. doi:10.1080/02626667.2011.579075
100
Glaser, B. G. & Strauss, A. L. 2009. The discovery of grounded theory: Strategies for qualitative
research. Transaction Publishers.
Gregory, R., Arvai, J., & Gerber, L. R. (2013). Structuring decisions for managing threatened
and endangered species in a changing climate. Conservation Biology : The Journal of the
Society for Conservation Biology, 27(6), 1212–21. doi:10.1111/cobi.12165
Grin, J., & van de Graaf, H. (1996). Technology assessment as learning. Science, Technology &
Human Values, 21(1), 72–99.
Halofsky, J. E., Peterson, D. L., Furniss, M. J., Joyce, L. A., Millar, C. I., & Neilson, R. P.
(2011). Workshop approach for developing change adaptation strategies and actions for
natural resource management agencies in the United States. Journal of Forestry, (June),
219–225.
Hansen, J. A., & Lehmann, M. (2006). Agents of change: universities as development hubs.
Journal of Cleaner Production, 14(9-11), 820–829. doi:10.1016/j.jclepro.2005.11.048
Hildén, M. (2011). The evolution of climate policies – the role of learning and evaluations.
Journal of Cleaner Production, 19(16), 1798–1811. doi:10.1016/j.jclepro.2011.05.004
Holman, I. P., Rounsevell, M. D. A., Cojacaru, G., Shackley, S., McLachlan, C., Audsley, E., …
Richards, J. A. (2008). The concepts and development of a participatory regional integrated
assessment tool. Climatic Change, 90(1-2), 5–30. doi:10.1007/s10584-008-9453-6
Holzkämper, A., Kumar, V., Surridge, B. W. J., Paetzold, A., & Lerner, D. N. (2012). Bringing
diverse knowledge sources together- a meta-model for supporting integrated catchment
management. Journal of Environmental Management, 96(1), 116–27.
doi:10.1016/j.jenvman.2011.10.016
Huntington, H. P., Brown-schwalenberg, P. K., Frost, K. J., Fernandez-gimenez, M. E., Norton,
D. W., & Rosenberg, D. H. (2002). Observations on the workshop as a means of improving
communication between holders of traditional and scientific knowledge. Environmental
Management, 30(6), 778–792. doi:10.1007/s00267-002-2749-9
Johnson, T. R. (2011). Fishermen, scientists, and boundary spanners: Cooperative research in the
U.S. Illex squid fishery. Society & Natural Resources: An International Journal, 24(3),
242–255. doi:10.1080/08941920802545800
Kastenhofer, K., Bechtold, U., & Wilfing, H. (2011). Sustaining sustainability science: The role
of established inter-disciplines. Ecological Economics, 70(4), 835–843.
doi:10.1016/j.ecolecon.2010.12.008
Kearney, J., Berkes, F., Charles, A., & Wiber, M. (2007). The role of participatory governance
and community-based management in integrated coastal and ocean management in Canada.
Coastal Management, 35(1), 79–104. doi:10.1080/10.1080/08920750600970511
101
Kloprogge, P., & van der Sluijs, J. P. (2006). The inclusion of stakeholder knowledge and
perspectives in integrated assessment of climate change. Climatic Change, 75, 359–389.
doi:10.1007/s10584-006-0362-2
Landry, R., Amara, N., & Lamari, M. (2001). Utilization of social science research knowledge in
Canada. Research Policy, 30, 333–349.
Lang, D. J., Wiek, A., Bergmann, M., Stauffacher, M., Martens, P., Moll, P., … Thomas, C. J.
(2012). Transdisciplinary research in sustainability science: practice, principles, and
challenges. Sustainability Science, 7(Supplement 1), 25–43. doi:10.1007/s11625-011-0149x
Lautenbach, S., Berlekamp, J., Graf, N., Seppelt, R., & Matthies, M. (2009). Scenario analysis
and management options for sustainable river basin managementt: Application of the Elbe
DSS. Environmental Modelling & Software, 24, 26–43. doi:10.1016/j.envsoft.2008.05.001
Lawler, J. J., Tear, T. H., Pyke, C., Shaw, M. R., Gonzalez, P., Kareiva, P., … Pearsall, S.
(2010). Resource management in a changing and uncertain climate. Frontiers in Ecology
and the Environment, 8(1), 35–43. doi:10.1890/070146
Lemos, M. C., & Morehouse, B. J. (2006). The co-production of science and policy in integrated
climate assessments. Global Environmental Change, 15(2005), 57–68.
doi:10.1016/j.gloenvcha.2004.09.004
Lengwiler, M. (2008). Participatory approaches in science and technology: Historical origins and
current practices in critical perspective. Science, Technology & Human Values, 33(2), 186–
200.
Lester, S. E., McLeod, K. L., Tallis, H., Ruckelshaus, M., Halpern, B. S., Levin, P. S., …
Parrish, J. K. (2010). Science in support of ecosystem-based management for the US West
Coast and beyond. Biological Conservation, 143(3), 576–587.
doi:10.1016/j.biocon.2009.11.021
Leydesdorff, L., & Ward, J. (2005). Science shops: a kaleidoschope of science-society
collaborations in Europe. Public Understanding of Science.
doi:10.1177/0963662505056612
Li, L. C., Grimshaw, J. M., Nielsen, C., Judd, M., Coyte, P. C., & Graham, I. D. (2009). Use of
communities of practice in business and health care sectors: A systematic review.
Implementation Science, 4(27), 1–9. doi:10.1186/1748-5908-4-27
Lienert, J., Monstadt, J., & Truffer, B. (2006). Future scenarios for a sustainable water sector: A
case study from Switzerland. Environmental Science & Technology, 40(2), 436–442.
doi:10.1021/es0514139
102
Lynam, T., de Jong, W., Sheil, D., Kusumanto, T., & Evans, K. (2007). A review of tools for
incorporating community knowledge, preferences, and values into decision making in
natural resources management. Ecology and Society, 12(1).
Lynam, T., Drewry, J., Higham, W., & Mitchell, C. (2010). Adaptive modelling for adaptive
water quality management in the Great Barrier Reef region, Australia. Environmental
Modelling and Software, 25(11), 1291–1301. doi:10.1016/j.envsoft.2009.09.013
Mackenzie, J., Tan, P. L., Hoverman, S., & Baldwin, C. (2012). The value and limitations of
Participatory Action Research methodology. Journal of Hydrology, 474, 11–21.
doi:10.1016/j.jhydrol.2012.09.008
Mader, M., Mader, C., Zimmermann, F. M., Görsdorf-Lechevin, E., & Diethart, M. (2013).
Monitoring networking between higher education institutions and regional actors. Journal
of Cleaner Production, 49, 105–113. doi:10.1016/j.jclepro.2012.07.046
Mahmoud, M., Liu, Y., Hartmann, H., Stewart, S., Wagener, T., Semmens, D., … Winter, L.
(2009). A formal framework for scenario development in support of environmental
decision-making. Environmental Modelling and Software, 24(7), 798–808.
doi:10.1016/j.envsoft.2008.11.010
Manring, S. L. (2014). The role of universities in developing interdisciplinary action research
collaborations to understand and manage resilient social-ecological systems. Journal of
Cleaner Production, 64, 125–135. doi:10.1016/j.jclepro.2013.07.010
Martin-Sempere, M. J., Garzon-Garcia, B., & Rey-Rocha, J. (2008). Scientists’ motivation to
communicate science and technology to the public: surveying participants at the Madrid
Science Fair. Public Understanding of Science, 17(3), 349–367.
doi:10.1177/0963662506067660
Matso, K. E., & Becker, M. L. (2014). What can funders do to better link science with decisions?
Case studies of coastal communities and climate change. Environmental Management,
54(6), 1356–71. doi:10.1007/s00267-014-0347-2
McClellan, E., MacQueen, K. M., & Neidig, J. L. 2003. Beyond the qualitative interview: Data
preparation and transcription. Field Methods, 15(1): 63 - 84.
McClure, M. M., Alexander, M., Borggaard, D., Boughton, D., Crozier, L., Griffis, R., … Van
Houtan, K. (2013). Incorporating climate science in applications of the US endangered
species act for aquatic species. Conservation Biology : The Journal of the Society for
Conservation Biology, 27(6), 1222–33. doi:10.1111/cobi.12166
Miles, M. B., Huberman, A.M., & Saldana, J. (2014). Qualitative data analysis: A methods
sourcebook (3rd ed.). Thousand Oaks: Sage Publications.
103
Nadkarni, N. M., & Stasch, A. E. (2013). How broad are our broader impacts? An analysis of the
National Science Foundation’s Ecosystem Studies Program and the Broader Impacts.
Frontiers in Ecology and the Environment2, 11(1), 13–19. doi:10.1890/110106
National Science Board. (2011). Merit Review Criteria. Review and Revisions. Retrieved from
papers2://publication/uuid/910994F5-3EE1-4236-8346-5602486DA1D2
National Science Foundation. (2012). Proposal and award policies and procedures guide.
Retrieved from http://www.nsf.gov/pubs/policydocs/pappguide/nsf13001/nsf13_1.pdf
Pahl-wostl, C. (2007). Transitions towards adaptive management of water facing climate and
global change. Water Resources Management, 21, 49–62. doi:10.1007/s11269-006-9040-4
Pahl-wostl, C., Craps, M., Dewulf, A., Mostert, E., Tabara, D., & Taillieu, T. (2007). Social
learning and water resources management. Ecology and Society, 12(2), 5.
Pahl-wostl, C., Sendzimir, J., Jeffrey, P., Aerts, J., Berkamp, G., & Cross, K. (2007). Managing
change toward adaptive water management through social learning. Ecology and Society,
12(2), 30.
Patton, M. Q. (2002). Qualitative research and evaluation methods. 3rd edition. Thousand Oaks,
CA: Sage.
Pearson, G., Pearson, G., Pringle, S. M., Pringle, S. M., Thomas, J. N., & Thomas, J. N. (1997).
Scientists and the public understanding of science. Science, 6, 279–289.
Podolny, J. M., & Page, K. L. (1998). Network forms of organization. Annual Review of
Sociology, 24, 57–76.
Raddick, M. J., Bracey, G., Gay, P. L., Lintott, C. J., Murray, P., Schawinski, K., …
Vandenberg, J. (2010). Galaxy Zoo: Exploring the motivations of citizen science
volunteers. Astronomy Education Review, 9, 15. doi:10.3847/AER2009036
Rayner, S., Lach, D., & Ingram, H. (2005). Weather forecasts are for wimps*: Why water
resource managers do not use climate forecasts. Climatic Change, 69, 197–227.
Reed, M. S., Evely, A. C., Cundill, G., Fazey, I., Glass, J., & Laing, A. (2010). What is social
learning ? Ecology and Society.
Riley, C., Matso, K., Leonard, D., Stadler, J., Trueblood, D., & Langan, R. (2011). How research
funding organizations can increase application of science to decision-making. Coastal
Management, 39(3), 336–350. doi:10.1080/08920753.2011.566117
Rittel, H. W. J., & Webber, M. M. (1973). Dilemmas in a general theory of planning. Policy
Sciences, 4(2), 155–169.
104
Robinson, C. J., & Wallington, T. J. (2012). Boundary work : Engaging knowledge systems in
co-management of feral animals on indigenous lands. Ecology and Society, 17(2), 16.
Rotman, D., Preece, J., Hammock, J., Procita, K., Hanse, D., Parr, C., … Jacobs, D. (2012).
Dynamic changes in motivation in collaborative citizen-science projects. In Session: Civic
and Community Engagement (pp. 217–226). doi:10.1145/2145204.2145238
Ryan, G. W. & Bernard, H. R. (2003). Techniques to identify themes. Field Methods, 15(1): 85 109.
Santelmann, M., Freemark, K., White, D., Nassauer, J., Clark, M., Danielson, B., Eilers, J.,
Cruse, R.M., Galatowitsch, S., Polasky, S., Vache, K., and J. Wu. 2001. Applying
ecological principles to land-use decision making in agricultural watersheds. Pages 226252 in V.H. Dale and R. A. Haeuber, editors. Applying ecological principles to land
management. Springer-Verlag, New York, New York, USA.
Sheppard, S. R. J., Shaw, A., Flanders, D., Burch, S., Wiek, A., Carmichael, J., … Cohen, S.
(2011). Future visioning of local climate change: A framework for community engagement
and planning with scenarios and visualisation. Futures, 43(4), 400–412.
doi:10.1016/j.futures.2011.01.009
Shirk, J. L., Ballard, H. L., Wilderman, C. C., Phillips, T., Wiggins, A., Jordan, R., … Bonney,
R. (2012). Public participation in scientific research: A framework for deliberate design.
Ecology and Society, 17(2), 29. doi:10.5751/ES-04705-170229
Smith, J. B., Strzepek, K., Rozaklis, L., Ellinghouse, C., & Hallett, K. (2009). The Potential
Consequences of Climate Change for Boulder Colorado’s Water Supplies.
Snover, A. K., Mantua, N. J., Littell, J. S., Alexander, M. A., McClure, M. M., & Nye, J. (2013).
Choosing and using climate-change scenarios for ecological-impact assessments and
conservation decisions. Conservation Biology: The Journal of the Society for Conservation
Biology, 27(6), 1147–57. doi:10.1111/cobi.12163
Sol, J., Beers, P. J., & Wals, A. E. J. (2013). Social learning in regional innovation networks:
trust, commitment and reframing as emergent properties of interaction. Journal of Cleaner
Production, 49, 35–43. doi:10.1016/j.jclepro.2012.07.041
Spiegel, J. M., Breilh, J., Beltran, E., Parra, J., Solis, F., Yassi, A., … Parkes, M. (2011).
Establishing a community of practice of researchers, practitioners, policy-makers and
communities to sustainably manage environmental health risks in Ecuador. BMC
International Health and Human Rights, 11 Suppl 2(Suppl 2), S5. doi:10.1186/1472-698X11-S2-S5
Star, S. L., & Griesemer, J. R. (1989). Institutional ecology, 'translations’ and boundary objects:
Amateurs and professionals in Berkeley’s Museum of Vertebrate Zoology, 1907-39. Social
Studies of Science, 19(3), 387–420.
105
Stubbs, M., & Lemon, M. (2001). Learning to network and networking to learn: Facilitating the
process of adaptive management in a local response to the UK’s National Air Quality
Strategy. Environmental Management, 27(8), 321–334. doi:10.1007/s002670010152
Swart, R. J., Raskin, P., & Robinson, J. (2004). The problem of the future: sustainability science
and scenario analysis. Global Environmental Change, 14(2), 137–146.
doi:10.1016/j.gloenvcha.2003.10.002
Tuler, S. (1998). Learning through participation. Human Ecology Review, 5(1), 58–60.
Tullos, D., Brown, P. H., Kibler, K., Magee, D., Tilt, B., & Wolf, A. T. (2010). Perspectives on
the salience and magnitude of dam impacts for hydro development scenarios in China.
Water Alternatives, 3(2), 71–90.
Turnhout, E., Stuiver, M., Klostermann, J., Harms, B., & Leeuwis, C. (2013). New roles of
science in society: Different repertoires of knowledge brokering. Science and Public Policy,
40, 354–365. doi:10.1093/scipol/scs114
Van Herk, S., Zevenbergen, C., Ashley, R., & Rijke, J. (2011). Learning and Action Alliances
for the integration of flood risk management into urban planning: a new framework from
empirical evidence from The Netherlands. Environmental Science & Policy, 14(5), 543–
554. doi:10.1016/j.envsci.2011.04.006
Vaske, J. V. (2008). Surveey research and analysis: Applications in parks, recreation and human
dimensions. State College: PA: Venture.
Voinov, A., & Bousquet, F. (2010). Modelling with stakeholders. Environmental Modelling &
Software, 25(11), 1268–1281. doi:10.1016/j.envsoft.2010.03.007
Webler, T. (1998). Beyond science: Deliberation and analysis in public decision making. Human
Ecology Review, 5(1), 61–62.
Weible, C. M., & Sabatier, P. A. (2009). Coalitions, science, and belief change: Comparing
adversarial and collaborative policy subsystems. Policy Studies Journal, 37(2), 195–212.
doi:10.1111/j.1541-0072.2009.00310.x
Wolf, B., Lindenthal, T., Szerencsits, M., Holbrook, J. B., & Heb, J. (2013). Evaluating research
beyond scientific impact: How to include criteria for productive interactions and impact on
practice and society. GAIA, 22(2), 104–114.
Wright, M., Hulse, D., Chan, S., & Ferguson, L. B. (2015). Section 3e. Methods - Stakeholder
involvement. In: Willamette Water 2100: Project Report. In preparation. Corvallis, Oregon:
Institute for Water and Watersheds, Oregon State University.
106
Yang, L., Wu, J., & Shen, P. (2013). Roles of science in institutional changes: The case of
desertification control in China. Environmental Science & Policy, 27(37), 32–54.
doi:10.1016/j.envsci.2012.10.017
107
CHAPTER 4: RESULTS, DISCUSSION & CONCLUSIONS
There is increasing emphasis on transdisciplinary research with stakeholder engagement
processes in natural resource research. Multiple case studies incorporate interdisciplinary studies
with an element of stakeholder engagement for sustainability (Swart et al., 2004), climate change
adaptive management (Cross et al., 2013), decision-support tool development (Holzkämper et al.,
2012), and alternative future exploration (Mahmoud et al., 2009). For as many cases as exist to
represent the scientific perspective and results, there are less which thoroughly discuss a
project’s stakeholder engagement process structure (Lemos & Morehouse, 2006), stakeholder
perspective of the process (Kloprogge & van der Sluijs, 2006), or the impacts of collaboration
between academic research teams and science stakeholders (Lang et al., 2012; Lautenbach et al.,
2009). It is necessary to fill these gaps to improve the researcher-stakeholder interaction in
transdisciplinary research projects (Baker et al., 2004).
The goals of this study were to characterize a transdisciplinary researcher-stakeholder
engagement process to address the gaps in previous reports of stakeholder engagement in natural
resource research by asking:
1) Who is participating in the researcher-stakeholder engagement process?
2) What are their motivations and expectations for participating?
3) What are their perceptions of the process?
Willamette Water 2100 (WW2100), the chosen case, reflected elements of all previously
documented cases of collaborative natural resource research and included a stakeholder
engagement process throughout the five year research project. Analyzing this case in this way,
this study demonstrates that participants in a researcher-stakeholder engagement process are
diverse, representing many organizations and interests. Participants enter the researcherstakeholder engagement process because of different social, knowledge, and tool-seeking
motivations and different expectations for it and for its results. As a result, there are different
perceptions of the process, different challenges, successes, and impacts reported. Future
researcher-stakeholder engagement processes can learn from the experience of WW2100 and
develop a process in which boundaries between participants are managed over multiple
108
interactions to facilitate iterative research development and achieve the potential diverse myriad
impacts.
Who is participating in the researcher-stakeholder engagement process?
Participants in the WW2100 researcher-stakeholder engagement processes were
numerous and diverse. Depending on the event, 25 to 74 people attended. The composition of
participants in any one engagement event was different from that of another engagement event
for the same research project. Representation refers to the number of individuals at any one event
and to the continuous participation of any one individual. As a university-led researcher project,
universities were the most-represented organization, with the most individuals at any given event
and continuously attending. Among stakeholders, the most represented groups were government
agencies (state, county, federal, and city) and the least represented were the tribes, farmers,
watershed councils, and non-profit organizations.
Participation is influenced by organizational support. Individuals from organizations
which typically support researcher-stakeholder collaboration were able to attend, and continue
attending, WW2100 engagement events. One researcher talked about how they could justify
attending events “because it could be something they could put on their resume as having done
this outreach-y thing.” Similarly, a government agency representative commented: “I do this for
a living, right? I go to meetings all day long…We do all this stuff and so we work on these
interdisciplinary teams.” University researchers and government agency employees were the
most participatory representatives. Institutional support for researcher-stakeholder engagement
can lead to greater participation in such events.
However, not all participants are supported in this way to attend events and their
participation is related to their free time and personal motivation to participate. Unequal support
for individuals to attend meetings leads to imbalanced stakeholder participation at events. As one
private citizen stakeholder remarked of the stakeholder events:
“it’s stacked with people being paid to be there. They’re government employees.
How is that representative? [One man] is the farmer and there is no timber
person from the private sector. So real stakeholders were not in the room…So for
a bunch of them it’s a day off. For [us], it’s a chunk out of our day that we gave to
the cause.”
109
Tribal and private industry interests were among the least represented organizations and they
receive the least institutional support to attend WW2100 researcher-stakeholder engagement
events.
One emergent question, then, is: how can fair and equal representation be achieved in a
transdisciplinary research project? Some discussions suggest offering economic incentives to
those who are not paid for their time in engagement events. However, this raises still more
questions and debate over who deserves compensation, how much, and what is the message to
those the project does not pay? Does a project value government agency participation less
because these representatives are not compensated by the project? Should participants traveling
greater distances to attend receive greater compensation?
Rather than reconcile these questions, it is better to focus on what draws participants to a
project beyond economic incentive. Not all WW2100 participants were paid to be there, and
even those that were expressed motivations to participate beyond the basic work day. What about
this project attracted individuals to participate in the first place and then, what led them to
continue participating? Future projects can look to WW2100 participant motivations to
encourage and maintain participation throughout a project for diverse and representative
stakeholder and researcher participation.
What are participants’ motivations and expectations in participating?
Just as participants represented diverse interests, they were motivated to participate for
diverse reasons and held distinct expectations for their roles, the process, and the project’s
results. In the survey of all WW2100 participants, concern for future water availability,
professional relevance, and the search for new tools to address water scarcity were the most
highly rated reasons participants attended events. However, the highest rated reason in the survey
was “other.” Respondents who marked this could then clarify what exactly motivated them to
become involved. Coupled with the semi-structured interview data, these open-ended responses
fall into three motivation categories: seeking knowledge, the promise of the research products or
for social reasons. These categories are not necessarily distinct from the suggested motivations in
the survey. Concern for future water availability and professional relevance can be connected to
the motivation of seeking knowledge, and new tools are linked to the promised research products
category. The social motivation options were not present in the survey instrument and so do not
110
have a survey corollary, but they include: collaboration and invitation, knowledge about the topic
of interest or to inform other projects, and for reaching broader audiences with the alternative
futures modeling tool produced by the project.
The three motivation categories identified in this study fit well within three of four
motivation categories identified by Rotman et al. (2012) in a study of online citizen science
engagement participants. Seeking knowledge and research products falls within the “egoism”
category where an individual is motivated to participate because it will be beneficial to that
individual. The social motivations identified in this study fit well within the collectivism and
altruism motivations identified by Rotman et al. (2012). Scientists and stakeholders are
motivated to engage with each other in research because they see a mutual benefit
(collaboration/collectivism) and/or because they believe they can help the other
(invitation/altruism). Participants were motivated to participate in the project by what they could
gain personally and professionally. Participants were also motivated to participate in the project
by what they could give personally and professionally.
The collectivistic and altruistic motivations of WW2100 participants are exemplified in
the expectations participants held for the process and their own roles within it. Participants
expected that the WW2100 researcher-stakeholder engagement process would provide an
opportunity to share what they know, to work with others in and outside of their fields, and to
learn. All participants expected that they would play a role to contribute to the research in some
way. Stakeholders were expected to provide a “boots-on-the-ground” perspective and evaluate
scenario assumptions while research team members were expected to interpret model outputs and
develop pieces of the model. In general, these contributory expectations for individuals were met
to varying degrees throughout the process.
The egoistic motivations of WW2100 participants were also exemplified in their
expectations for the research process and its results. Survey respondents reported that they
expected the process to be an opportunity to satisfy their curiosity, to build an integrated model
of water in the Willamette Valley, and to produce results which are relevant to their jobs and
which contribute to science. As with the collectivistic and altruistic-driven expectations, these
egoistic-driven motivations were met to varying degrees by the WW2100 research process.
WW2100 participants were not a homogenous group of individuals with identical
expectations for each other’s roles. There was a significant difference between the roles research
111
team members were expected to fulfill and the roles stakeholders were expected to fulfill. As
leaders of the project, research team members were expected to fulfill all suggested roles except
for providing a “boots-on-the-ground” perspective. Stakeholders, on the other hand, were
expected to provide feedback and communicate results rather than participate in the tasks
directly associated with the research process. In similar projects stakeholders were commonly
expected to contribute knowledge (Johnson, 2011) and fill data gaps (Sheppard et al., 2011) in
what can be called a “contributory” rather than a “collaborative” process (Shirk et al., 2012).
Exceeding their expectations, participants reported that stakeholders in fact developed pieces of
the model and wrote reports, shifting the project from the expected “contributory” process and
into the realm of “collaboration” or “co-creation” (Shirk et al., 2012).
Not all WW2100 expectations were exceeded or met. One expectation, to have frequent
interaction with stakeholders, was not met by the WW2100 researcher-stakeholder engagement
process. This could be indicative of very high expectations for interaction with stakeholders or it
could indicate that the WW2100 researcher-stakeholder engagement process could have
coordinated more interactive events and facilitated them in a way to promote greater researcherstakeholder interaction.
Participant expectations for the resulting WW2100 model reflect the typical researcherstakeholder engagement goals: outcomes for research, outcomes for individuals, and outcomes
for social-ecological systems (i.e. influencing policies) (Shirk et al., 2012). Participants expected
that the resulting model would be an accurate representation of water in the Willamette Valley,
that the model would contribute to science, and that it would provide results useful to their jobs.
All participants agreed that the model met these expectations but to a degree significantly less
than was expected.
As the leaders of the project inviting the stakeholders in (Lang et al., 2012), research
team members generally held higher expectations for the process, their own role within it, and
the results than the stakeholders. When the different expected roles blur, as they did in WW2100,
and when the same project elements are expected but to different degrees, conflict may arise as a
result of the differing methodological and quality standards (Daniell et al., 2010; Lang et al.,
2012). Stakeholders may develop ideas for what an engagement process can do for them (Daniell
et al., 2010) which may influence their perceptions of the resulting model (Becu et al., 2008).
Potential conflicts can be avoided by hosting early engagement meetings which focus on
112
developing shared expectations for the process, its results, and each other’s roles within it (Tim
Lynam et al., 2010).
Understanding motivations and expectations for participation in researcher-stakeholder
engagement projects is important not only to avoid conflict but also to improve the results
produced and the likelihood for current and future participation. Universities are viewed
increasingly as development hubs (Hansen & Lehmann, 2006). Although researchers tend to
emphasize the contributions of their research to science through publications, stakeholders
require increased applied research in natural resource management (Johnson, 2011). In order to
address the needs and expectations of participating groups, it must first understand them. When a
project allows participants to meet their needs and expectations, they are more likely to
participate again (Eccles & Wigfield, 2002).
What are participants’ perceptions of the process?
Semi-structured interviews and an online survey documented WW2100 participants’ “en
route” reflection (Daniell et al., 2010) on the challenges of the project, the impact it has had, and
how the researcher-stakeholder engagement process structure can contribute to a successful
experience.
Challenges
This study identified four overarching challenges to the WW2100 transdisciplinary
research project including the lack of a shared vision, interdisciplinary challenges, research
complexity, and project management challenges. Examples of these types of challenges are
characterized in similar case studies (ex. Lemos & Morehouse, 2006) which offer suggestions to
avoid or overcome these challenges in “lessons learned” sections (ex. Lang et al., 2012).
The most prevalent challenge to WW2100 and frequently mentioned in the literature is
the challenge a project faces when it does not have a shared vision. The participation of diverse
actors with diverse experiences can lead to a project with diverse visions for what the project will
achieve and the method to achieve it. Transdisciplinary research requires a common goal and
plan (Pahl-wostl, Craps, et al., 2007); without these elements, there will be conflict (Lang et al.,
2012), distrust, and resistance to new ideas (Manring, 2014). In WW2100, the lack of a shared
113
vision was caused by different experiences, research and stakeholder engagement philosophies,
and an unclear research proposal. When participants sought to realize their different visions, they
came into conflict with each other over whether the research should produce publications or an
accessible tool, at the sub-basin or regional scale, and if the model was producing projections or
exploring different planning scenarios. Research team members conflicted over when and how to
involve stakeholders based on their philosophies, previous experience, and research goals.
Interviewees detailed these conflicts and suggested that if there had been an initial meeting or
series of meetings to clarify goals and arrive at a consensus over a shared vision, there would
have been much less confusion and conflict. Researchers in previous studies also came to this
conclusion as one of their lessons learned (Dewulf et al., 2007; Fuller, 2011; Halofsky et al.,
2011; Kearney et al., 2007; Lang et al., 2012; Lautenbach et al., 2009; Lemos & Morehouse,
2006; Mackenzie et al., 2012; Matso & Becker, 2014; Sol et al., 2013). This study offers the
novel finding that when there is a lack of shared vision among research team members, it is
reflected in the stakeholder engagement process. Stakeholder participants are either confused
about the project goals and feel that they missed something, perceive the resulting conflict
among research team members, or find themselves in conflict with other participants as well.
Therefore, not only should there be a meeting among research team members to establish goals
and methods, but there should be a meeting among all participants to confirm the vision for the
project.
Arriving at a shared goal is one challenge that can be categorized among many
interdisciplinary challenges that a transdisciplinary project may encounter. WW2100 participants
also shared that communication across disciplines and the time it required was a challenge. A
study on water and agricultural collaboration in the Everglades found that there was a greater
difference in perspectives among scientists of different disciplines than between scientists and
lay stakeholders (Fuller, 2011). The meaning of a common issue or concept may not be selfevident (Dewulf et al., 2007) and is thus determined by an individual’s training and experience,
which in interdisciplinary research, varies. WW2100 devoted a half-day workshop, multiple
meetings, and a publication to define its key term “water scarcity.” Less time was dedicated to
define what “stakeholder engagement” or “future water modeling” meant. Different
understandings of concepts can make cooperation difficult (Fuller, 2011). Just as with arriving at
a shared vision, arriving at a shared language requires time, patience, and a willingness to learn.
114
One challenge unique to WW2100 was the overwhelming complexity of the
transdisciplinary research task. According to the project website, the research team objectives
were to: identify and quantify the linkages and feedbacks among human, hydrologic, and
ecologic dimensions of the water system, make projections about where and when human
activities and climate change will impact future water scarcities, evaluate how alternative
scenarios affect future water scarcities, and develop transferable tools and methods for projecting
water scarcities and modeling policy alternatives (OSU Institute for Water and Watersheds,
2015). WW2100 was a project with broad goals to accurately and precisely characterize a widely
used natural resource over a vast region and across a long expanse of time. Interviewees shared
that it was challenging to characterize such a broad region and to imagine what the climate,
political and otherwise, would look like in 85 years. Projects with smaller aims (geographically,
temporally, disciplinarily) have encountered research challenges which would only be amplified
in the upscaled process (ex. Hulse & Gregory, 2001; Mackenzie et al., 2012; Santelmann et al.,
2001).
A complex research topic is accompanied by the challenge of complex project
management. The large study area and the need to represent many hydrological, ecological,
economical, and political processes require the participation of many researchers and
stakeholders (Mahmoud et al., 2009). With a high number of participants, the probability for
personality differences increases. The amount of time and resources required to coordinate
researcher-stakeholder engagement events also increases (Baker et al., 2004; Mackenzie et al.,
2012). WW2100 participants were often unable to attend events or respond to e-mails due to
their busy schedules. Engagement event dynamics shifted as the composition of participants
shifted which made project continuity a challenge. Managing project logistics becomes more
challenging as the study area, number of researchers, and number of stakeholders increase. The
resources devoted to the project should increase accordingly.
Successes and Impacts
WW2100 participants identified three types of success which resulted from the WW2100
researcher-stakeholder engagement process: research success, process success, and personal
success. Previous work has identified a dichotomy between what can be considered traditional
measures of research success and “alternative measures of success” in transdisciplinary projects.
115
Scientists in collaborative projects may focus on publications and contributing to long-lasting
science while industry members may celebrate a study’s contribution to management (Johnson,
2011; Mackenzie et al., 2012). Others claim that the collaborative design experience itself may
be more important than the research result and so is an alternative measure of success
(Lautenbach et al., 2009). A project resulting in new relationships among participants, improved
communication, accessible knowledge, useful tools, behavior change, or specific societal
outcomes can also be considered successful (Baker et al., 2004; Dilling & Lemos, 2011; Lienert
et al., 2006). The traditional/alternative success dichotomy was only partially present in
WW2100 as represented by the research and process success categories. Both research team
members and stakeholders celebrated the project’s peer-reviewed publications and practical
applications. Both groups expressed the success of engaging in a project with each other,
developing relationships and participating in a rich discussion about a shared resource.
When asked to speak on the successes of the project, every individual interviewed
emphasized that he or she had learned something. Learning is an important element of the other
two forms of success and can be a success in and of itself. Learning may also be a necessary
intermediary between direct effects of a project and its broader impacts. In WW2100, social
learning took place as individuals experienced a change in understanding within and beyond the
individual through an interactive process (Reed et al., 2010). Social learning can provide impacts
beyond the process which led to it. Social learning supports resilience and adaptive capacity
(Tim Lynam et al., 2010; Manring, 2014) and is “inextricably bound to action” (Stubbs &
Lemon, 2001,p. 333). In addition to the research success and process success, this third category
of success can extend a project’s impact.
By creating successes like those above, stakeholder engagement in scientific research can
overcome science integration barriers and lead to science-driven decisions and action toward
conservation, resilience, and sustainability (Shirk et al., 2012). Each case in the literature reflects
on potential impacts of its project (Baker et al., 2004; Cash et al., 2003; Holman et al., 2008;
Holzkämper et al., 2012; Tim Lynam et al., 2010), however, it is difficult to measure the impact
of stakeholder engagement in scientific research without an impending action decision. This
study places the outcomes reported by WW2100 interviewees and survey respondents in the
context of the NSF Broader Impact as a framework for evaluating the impacts of stakeholder
engagement.
116
The impacts of the WW2100 researcher-stakeholder engagement process provide specific
examples for achieving NSF’s five broader impacts review criteria and suggest the creation of an
emerging sixth criterion (Table 4.1). As a result of its stakeholder engagement process, WW2100
met each of the original five review criteria to some degree. Stakeholder engagement in
WW2100 contributed to advancing scientific discovery and understanding by directly
influencing the research, facilitating learning, and by pushing to achieve research deadlines.
Stakeholders provided researchers feedback on questions, model assumptions, output metrics,
scenarios, and interpretations in a way that would not have been achieved otherwise. In other
cases stakeholders have helped to define process components, methods, design elements, output
indicators, and outcome plausibility (Holman et al., 2008; Mahmoud et al., 2009). Learning is of
key importance in meeting this criterion as well, and, as was discussed previously, every
interviewee expressed that they had learned something. It is impossible to advance science and
understanding without learning in the process. Finally, in preparation for stakeholder
engagement events, research products were made that would not have been produced otherwise
(Pearson et al., 1997) and at a faster pace. WW2100 research-stakeholder engagement
contributed to advancing scientific discovery an understanding by providing an opportunity to
integrate new and traditional sources of knowledge and perspectives.
The WW2100 stakeholder engagement process also contributed to enhancing research
infrastructure. Interviewees shared that the most meaningful product for them was the network of
users, regulators, and researchers that formed and the water resources discussion that took place
within it. Networks, which provide a platform for necessary natural resource conversations, are
essential for adapting to future changes (Pahl-wostl, Craps, et al., 2007). They increase learning
(Manring, 2014), social capital (Leydesdorff & Ward, 2005), and adaptive capacity (Pahl-wostl,
2007) by allowing for an ongoing transparent dialogue of resource management tradeoffs
(Hildén, 2011). Building and strengthening a network of users, regulators, and researchers was
positively correlated with participation for researchers and stakeholders engaging in WW2100.
As researchers and stakeholders join together to discuss technologies, policies, and beliefs, they
build the infrastructure required for systems science to respond to wicked problems (Pahl-wostl,
2007; Rittel & Webber, 1973).
Although forming partnerships fall within the NSF broader impact criterion “enhance
research infrastructure,” the prominence strengthening a research network played in participant
117
evaluations of WW2100’s researcher-stakeholder engagement process leads this study to propose
a sixth criterion to NSF’s broader impact criteria: develop research community. Enhancing
research infrastructure remains an important criterion as it refers to building new facilities and
instrumentation. However, due to the strong emphasis on forming and forming and strengthening
relationships, a project may achieve broader impacts through developing the research community
within and beyond academia.
The WW2100 stakeholder engagement process contributed to broadly disseminating its
research results. By inviting stakeholders into the research process, WW2100 researchers built
credibility for their tool. Before stakeholders accept or share research findings with others, they
must believe that they are legitimate, credible, and relevant to their work (Cash et al., 2003;
Johnson, 2011). Researcher-stakeholder engagement can be a “live peer review” (Halofsky et al.,
2011) which can lead to improved credibility and legitimacy for decision-support tools (Holman
et al., 2008; Tim Lynam et al., 2010). When participants perceive research results as credible,
they are more likely to share them with co-workers and apply them in their respective contexts.
WW2100 interviewees were sharing what they were learning with their co-workers throughout
the project and stated that they would be “ambassadors” for the project when it produced final
results. Training stakeholder ambassadors of science will help a project to disseminate its results
broadly.
Broader participation of underrepresented groups in scientific research is an NSF review
criterion not documented in previous cases of stakeholder engagement. The WW2100
stakeholder engagement process was able to include participants not often associated with
scientific research projects like tribal and agricultural representatives. One agricultural
interviewee remarked: “we’ve had many researchers through the years come out to look at
different things and never been questioned about environmental things… the ag community,
we’re a pretty small number of people.” Interviewees also applauded the process for achieving
the participation of busy political representatives. As the participation data show, however, the
WW2100’s representation of underrepresented groups was imperfect. Some perspectives were
missing, others were underrepresented, and others were simply less salient. Still, representing
diverse perspectives on an issue is valuable in natural resource research and can lead to new
insights (Bartels et al., 2013). This broader impact element should continue to be explored.
118
Research, then, may broaden participation of underrepresented groups by inviting diverse
participants to attend but also by ensuring that participants feel heard.
The broadest, and perhaps most difficult to measure, broader impacts criterion is whether
or not the research benefits society. This study’s results suggest that one way to operationalize
this criterion is by ensuring that the users of science believe the research results will benefit
society. In WW2100, the perceived model utility and its results can indicate the benefit of this
research to society. First, through the researcher-stakeholder engagement process, stakeholders
were able to direct researchers towards results that they would find useful. Stakeholders defined
scenarios to answer questions they found relevant. Through the process, participants gained a
greater understanding of the model and its limitations. There may have been unrealistic
assumptions going into the model (Tim Lynam et al., 2010) or unrealistic demands on the
research (Dilling & Lemos, 2011) which contributed to perceptions of the model’s limitations.
Still, WW2100 interviewees identified ways in which the research results could be useful to
understand the current processes (Santelmann et al., 2001) and to provide a basis for future water
resources discussion. Survey respondents agreed that the model was useful for resource
managers, policy makers, water users and researchers. Defined in this way, the WW2100
researcher-stakeholder engagement process contributed to scientific research which benefits
society.
There was a significant correlation between an individual’s participation in researcherstakeholder engagement events and the degree to which that individual perceived the project’s
impacts. A high level of participation may be key to achieving broader impacts through
researcher-stakeholder engagement. Alternatively, individuals who participated more may have
done so because they already valued the process and its outcomes. It is impossible to identify the
causative agent from this data – whether greater participation led to a greater perceived degree of
WW2100 impacts or if valuing the project to a great degree led participants to increase
participation. Still, that participation and perceived project impacts correlate and interact is
noteworthy for future researcher-stakeholder engagement efforts. Interaction between
researchers and science stakeholders can improve understanding of one another (Dilling &
Lemos, 2011), create trust, commitment, and shared understanding of a problem and its solutions
(Sol et al., 2013). The degree to which individuals are involved in scientific research ultimately
influences a project’s outcomes (Shirk et al., 2012).
119
Table 4.1. Revised broader impact (BI) framework and examples from WW2100 outcomes.
BI review criteria
Advance discovery
and understanding
Specifications to criteria
Integrate new and traditional
sources of knowledge and
perspectives
Qualitative data
“Provided a feedback
mechanism on what we’re
doing and whether it’s
reasonable or not”
Quantitative data
Process utility
Diverse participants are
present
“I think the diversity is
remarkable and
respected”
Attendance record
Diverse participants feel
heard
“We’re being listened to”
Feeling heard
Enhance research infrastructure
Build new facilities and
instrumentation
“We’ve built a model”
Model utility
Develop research community
Form and strengthen research
partnerships and networks
“The most meaningful
product of projects like
this is the connections
between people”
Process utility
Broad dissemination
of results
Disseminate results broadly
Train stakeholder
ambassadors of science
“People take the output
back to the groups they
belong to”
Model
understanding
Benefit society
Benefit society
Science users believe results
will benefit society
“It just created a lot more
buy-in from the users”
Model utility
Broaden participation
of underrepresented
groups
Enhance research
infrastructure
Revised BI review criteria
Advance discovery and
understanding
Broaden participation of
underrepresented groups
Model
understanding
120
“Mere involvement of outside actors does not yet stipulate a system of equal or adequate
representation” (Lengwiler, 2008, p. 16)
Engagement Process Structure to achieve impacts
Participation in the WW2100 researcher-stakeholder engagement process was important
for achieving scientific broader impacts. WW2100 utilized multiple engagement formats over
five years organized by a team dedicated to facilitating researcher-stakeholder interaction. Given
its successful outcomes, the WW2100 process may serve as a model for how to structure future
engagement processes.
WW2100 provided many opportunities for researcher-stakeholder interaction by
coordinating multiple events of various formats continuously over the five year study period.
One study found that continued and intense interaction between researchers and stakeholders
increased the likelihood of research utilization (Landry et al., 2001). In WW2100 there were
formal interactions in the form of first year field trips, annual large group workshops, and
monthly small group workshops in the last project year. Interviewees identified two or three
phases of the stakeholder engagement process based on their perceptions of what the goals of
these processes were. One stakeholder summarized and reflected on the phases in this way:
“they all three deliver different things. I think the middle phase was
informational. They were just letting us know what they’d been working on. The
beginning phase was just really getting started building the relationships, getting
to know the whole concept, why are we doing this and what’s important to think
about. And then by the end, the third phase was much more specific about
these…are the actual numbers or the actual things that we’re putting into the
model and from my perspective, to make it as realistic as possible, the third phase
was the most important. And I would say honestly, I think there could have been
more of that directly facilitated scientists and stakeholders and what should go
into the model runs discussion earlier.”
Each type of interaction was valuable for the goal it intended to achieve. Field trips served to
connect participants to each other through the shared experience of exploring the study area.
Annual large group meetings in years 2-4 informed stakeholders of the progress the research
team was making and allowed for stakeholder feedback on the modeling. In year 5, a small group
met monthly to develop stakeholder scenarios and to hold more in-depth discussions about the
details of the model that had been developed.
121
As the forum for interaction varied, so too did the interactive activities within them. Each
event was between a half-day and a full day long and typically consisted of plenary presentations
from research team leaders, short oral presentations or poster presentations from research team
members, and opportunities for stakeholder feedback through facilitated discussions, interactive
posters, and informal conversations. Of one plenary presentation in a large group meeting one
stakeholder said: “It was more of a presentation style in an auditorium so it wasn't designed
where they were requesting a lot of dialogue.” During the presentation portions, there was no
real opportunity to interact and connect with each other. Research members spoke more
favorably of the poster sessions and focus groups:
“We had poster sessions and presented our preliminary results to the
stakeholders and answered questions. And that one was useful to me because
there were folks…there who asked some good questions.”
“I think the most effective one was when we had the facilitated sessions.”
Huntington et al. (2002) found that presentations and project graphics, as in WW2100 plenary
presentations and poster sessions, contribute to a sense of formality and inhibit discussion.
Whereas breakout sessions, like the WW2100 facilitated sessions, promote participant
interaction and generate ideas (Halofsky et al., 2011).
WW2100 also utilized less formal outreach and engagement methods throughout the
research process including newsletters, a project website, and personal communication. As
members of the project mailing list, participants in WW2100 received e-mails from the project
team regarding updates and news stories on the topic of interest. On the project website,
researchers and stakeholders can access an overview description of the project and get to know
the research team members and the water model. They may also access educational webinars,
slides from previous workshop presentations, newsletters and research publications resulting
from research, and a “frequently asked questions” page developed as a result of one workshop’s
feedback. There were also numerous personal correspondences among participants. Research
team members may have solicited specific information from one known source to inform the
model or stakeholders may have wanted to provide extensive feedback without sidetracking a
large group meeting. These conversations occurred on the telephone, over e-mail, and over
meals.
When asked how important the various WW2100 activities were to engaging
stakeholders in scientific research, survey respondents considered small group workshops,
122
personal communication, and field trips to be the most important. Interviewees agreed,
commenting more positively on the field trips and small group workshops than the large group
workshops (Figure 4.1). These formats more effectively facilitated the researcher-stakeholder
interaction by providing, as one stakeholder put it, “face time not meaning necessarily person to
person but more interaction among people that were working on individual pieces of the project
with stakeholders.” Workshops improve communication and collaboration between holders of
different kinds of knowledge (Huntington et al., 2002) but only if they are small enough.
Kloprogge & van der Sluijs (2006) found that 25-30 is the optimal number of people to get the
maximum number of ideas and that was the number of participants in WW2100’s small group
workshops.
Contrarily, the least important activities for engaging stakeholders in science according to
survey respondents were large group workshops, webinars, and newsletters. These activities
more often reflect one-way communication. In large groups, it was difficult to have the
conversations stakeholders wanted due to the prevalence of plenary talks and the auditoriumstyle setting. Cross et al. (2013) suggest that no more than one quarter of an event’s schedule
should be designated to one-way presentations like these. Similarly, webinars and newsletters
share important information but do not provide the give and take that is needed for stakeholder
engagement. These elements, however, should not be discarded entirely from a process.
Interviewees shared that these activities were valuable to gather important information and
proceed in the project.
“It was great, really informative to me to know what the overall inputs to the
model were going to be, what the expected outputs were going to be.”
“And if anything it helped to say, hey, maybe we need to have that follow-up
conversation or we need to meet separately or something.”
123
Figure 4.1. Timeline of stakeholder engagement events and positive and negative perceptions of these
events. Stakeholder engagement events are represented by blue boxes and positive and negative
perceptions by green and red boxes, respectively. Location on the y-axis was determined by calculating
the difference between positive and negative perception ratios drawn from interviews. A ratio/location
nearer to 1 indicates that event was spoken of more positively than negatively by interviewed subjects.
What unites the three most important activities is the potential they raised for the
WW2100 research process to be iterative. Among the lessons learned from several sustainability,
climate change adaptive management, decision-support tool construction, and alternative future
exploration studies is the need for an iterative research process (Dilling & Lemos, 2011;
Halofsky et al., 2011; Holman et al., 2008; Lang et al., 2012; Swart et al., 2004; Voinov &
Bousquet, 2010). Iterative processes can create relationships between scientific and decisionmaking processes (Lemos & Morehouse, 2006) and then customize knowledge to meet
stakeholder needs and uncover new uses for science (Dilling & Lemos, 2011). However, the
engagement process structure can only open the door to iterativity, for it also depends on the
nature of the problem being addressed and the ability and willingness of participants to engage
with each other. It also depends on individual participants’ disciplinary and personal flexibility
124
and a project’s resource availability and allocation (Lemos & Morehouse, 2006). For example,
iterative processes are time-intensive, suggesting that a project wishing to engage stakeholders in
an iterative process should begin early and host events often. Small group workshops and
personal communication between researchers and stakeholders are opportunities for an iterative
research process to unfold.
Such opportunities were the product of extensive planning and coordination performed
by the WW2100 Broader Impacts Team. Stakeholder engagement in WW2100 was more
effective because of the efforts of what can be considered a boundary spanning team (Turnhout,
Stuiver, Klostermann, Harms, & Leeuwis, 2013). The Broader Impacts Team (BIT), as part of
the research team, fulfilled the role of facilitator and worked towards smooth boundary crossings
between the disciplinary science cultures and the diverse stakeholder cultures (Aikenhead, 2001).
They coordinated and orchestrated all of the engagement events and crafted written
communications between groups. The BIT often held meetings with researchers and stakeholder
and encouraged both sides to be more transparent in their language and objectives. As one
stakeholder noted: “If [the BIT] hadn’t been involved, there would have been no public input at
all. I’m convinced of that.” As boundary spanners, the BIT was integral to WW2100’s
stakeholder engagement process and contributed greatly to its success.
The concept of a boundary spanner is present in much of the collaborative management,
transdisciplinary research, and participatory governance literature. Any time there are
representatives from distinct groups coming together, there is need in a project for a boundary
spanner. Boundary spanners facilitate the integration of different knowledge types (Robinson &
Wallington, 2012), reduce conflict, and create buy-in (Johnson, 2011). Boundary spanners can
fulfill this role informally as a participant (Freitag, 2014) or formally on the project team to serve
this function (Matso & Becker, 2014). One research team member described a boundary spanner
saying, “And you need someone comfortable moving across fields and has experience with that,
definitely helps. It couldn't hurt.” The BIT was formally responsible for facilitating the
researcher-stakeholder interaction in WW2100.
A dedicated boundary spanning team facilitated engaging interaction between researchers
and stakeholders through multiple methods and frequent events over the five year project period.
Small group workshops, personal communication, and field trips were considered the most
important engagement activities by participants while large group meetings, webinars, and
125
newsletters were the least important. The Broader Impacts Team, with project support, facilitated
iterative, cross-cultural interactions which contributed to stakeholder engagement success.
Limitations
There are limitations and emergent areas for future research that result from any study.
One of the major limitations of this study is the potential for bias. Efforts were made to reduce
bias through triangulation. Multiple methods were employed for methodological triangulation, a
variety of data sources were used for data triangulation, and several researchers analyzed the
same data for investigator triangulation (Creswell, 2003). Combining objective attendance data
and quantitative survey results with the qualitative semi-structured interview codes allowed for
robust and illuminating results. For example, subjects in the semi-structured interview may have
been biased to express what they perceived to be the socially desirable response to questions
(Vaske, 2008). This social desirability bias was reduced by using an anonymous online survey
for methodological triangulation. Similarly, survey respondents may have recall bias (Patton,
2002) when asked to consider their expectations prior to participating in the process five years
ago. Through semi-structured interviews, the researcher could gauge the certainty with which
subjects expressed their expectations which aided data interpretation. Finally, the objective
measures of participation were, in fact, biased to measure only participation in formal events.
Through the survey, the researcher was able to obtain a self-reported measure of participation in
less formal engagement activities such as e-mail and phone conversations.
Of course, to provide a fuller characterization of the WW2100 researcher-stakeholder
engagement project, it would have been better to survey and/or interview all participants. This
study was limited by time, resources, and subjects’ willingness to participate. Not all WW2100
participants wanted to be interviewed or completed the survey. Although the results of this study
may not be generalizable to all transdisciplinary studies, it is generalizable to the participant
experience in WW2100. Furthermore, lessons learned from the WW2100 case can be transferred
to projects with similar goals and challenges.
The freedom of the exploratory research design allowed the researcher to focus this
report on the issues that were salient to the broader groups of participants. However, some
themes emerged from the semi-structured interviews which future studies may be able to explore
further. Many participants compared their experience in WW2100 to previous experience that
126
may or may not have been similar to the project of interest. How does previous experience with
engagement impact future engagement projects? Interviewees also discussed how they preferred
to receive scientific results and emphasized the need for strong leadership in transdisciplinary
projects. Similarly, several elements of the survey instrument remain unexplored. Future studies
can continue to compare researcher-stakeholder responses to various items and relate
participation in specific events to the level of importance ascribed to each event. Finally, this
report provides an overview of the challenges and successes of one case. Future studies could
take a closer look at each one for a deeper understanding of these phenomena across cases.
Recommendations
Based on the experiences of researchers and stakeholders in the WW2100 engagement
process, this study offers some recommendations for researchers and stakeholders who engage
with one another in future projects.
•
Explore and understand the motivations and expectations of participants early or even
prior to engagement to ensure that their expectations can be met or at least managed
before the hopes get too high.
•
Hold early meetings to establish shared goals, methods, and understandings of key
concepts.
•
Explore a geographic and temporal range that suits the shared research question and
interested stakeholders. No need to build a regional model if the questions are at the subbasin level.
•
Stakeholder engagement is time and resource intensive. Plan accordingly.
•
Include small group workshops and field trips in the formal stakeholder engagement
structure.
•
Encourage professional researcher-stakeholder relationships. Bartels et al. (2013)
recommend pairing scientists and stakeholders by expertise.
127
Conclusions
This research characterizes one case of a transdisciplinary research project. Who is
participating, their motivations for being there, and their expectations contribute to the
challenges and successes of the process. This research also provides a framework for evaluating
future engagement process impacts.
Three main conclusions are drawn from this study.
1. Processes can be improved by understanding diverse participant motivations and
expectations early in the project so that work can begin to arrive at a common vision for
how the project will proceed.
2. A formal team to facilitate stakeholder engagement process makes a more effective
stakeholder engagement process.
3. Stakeholder engagement is an effective process to achieve NSF broader impact goals.
128
Literature Cited
Aikenhead, G. A. (2001). Science communication with the public: A cross-cultural event. In C.
Bryant, M. Gore, & S. Stocklymayer, Science Communication in Theory and Practice. The
Netherlands: Kluwer Academic Publishers
Baker, J. P., Hulse, D. W., Gregory, S. V, White, D., Van Sickle, J., Berger, P. A., …
Schumaker, N. H. (2004). Alternative Futures for the Willamette River Basin, Oregon.
Ecological Applications, 14(2), 313–324.
Bartels, W. L., Furman, C. A., Diehl, D. C., Royce, F. S., Dourte, D. R., Ortiz, B. V., … Jones, J.
W. (2013). Warming up to climate change: a participatory approach to engaging with
agricultural stakeholders in the Southeast US. Regional Environmental Change, 13, S45–
S55. doi:10.1007/s10113-012-0371-9
Becu, N., Neef, A., Schreinemachers, P., & Sangkapitux, C. (2008). Participatory computer
simulation to support collective decision-making: Potential and limits of stakeholder
involvement. Land Use Policy, 25(4), 498–509. doi:10.1016/j.landusepol.2007.11.002
Brainard, R. E., Weijerman, M., Eakin, C. M., McElhany, P., Miller, M. W., Patterson, M., …
Birkeland, C. (2013). Incorporating climate and ocean change into extinction risk
assessments for 82 coral species. Conservation Biology : The Journal of the Society for
Conservation Biology, 27(6), 1169–78. doi:10.1111/cobi.12171
Callahan, B., Miles, E., & Fluharty, D. (2013). Policy Implications of climate forecasts for water
resources management in the Pacific Northwest. Policy Sciences, 32(3), 269–293.
Cash, D. W., Clark, W. C., Alcock, F., Dickson, N. M., Eckley, N., Guston, D. H., … Mitchell,
R. B. (2003). Knowledge systems for sustainable development. Proceedings of the National
Academy of Sciences of the United States of America, 100(14), 8086–8091.
doi:10.1073/pnas.1231332100
Chang, H., Praskievicz, S., & Parandvash, H. (2014). Sensitivity of urban water consumption to
weather and climate variability at multiple temporal scales: The case of Portland, Oregon.
International Journal of Geospatial and Environmental Research, 1(1), Article 7.
Clarke, S., & Roome, N. (1999). Sustainable business: Learning-action networks as
organizational assets. Business Strategy and the Environment, 8, 296–310.
Cohen, S. J. (2010). From observer to extension agent—using research experiences to enable
proactive response to climate change. Climatic Change, 100(1), 131–135.
doi:10.1007/s10584-010-9811-z
Creswell, J. W. 2003. Research design: Qualitative, quantitative, and mixed-methods
approaches. Thousand Oaks, CA: Sage.
129
Cross, M. S., McCarthy, P. D., Garfin, G., Gori, D., & Enquist, C. (2013). Accelerating
adaptation of natural resource management to address climate change. Conservation
Biology : The Journal of the Society for Conservation Biology, 27(1), 4–13.
doi:10.1111/j.1523-1739.2012.01954.x
Daniell, K. A., White, I., Ferrand, N., Ribarova, I. S., Coad, P., Rougier, J. E., … Burn, S.
(2010). Co-engineering participatory water management processes: Theory and insights
from Australian and Bulgarian interventions. Ecology and Society, 15(4), 11. Retrieved
from http://www.ecologyandsociety.org/vol15/iss4/art11/
Dewulf, A., François, G., Pahl-wostl, C., & Taillieu, T. (2007). A framing approach to crossdisciplinary research collaboration: Experiences from a large-scale research project on
adaptive water management. Ecology and Society, 12(2), 14.
Dietz, T., Ostrom, E., & Stern, P. C. (2003). The struggle to govern the commons. Science (New
York, N.Y.), 302(5652), 1907–12. doi:10.1126/science.1091015
Dilling, L., & Lemos, M. C. (2011). Creating usable science: Opportunities and constraints for
climate knowledge use and their implications for science policy. Global Environmental
Change, 21(2), 680–689. doi:10.1016/j.gloenvcha.2010.11.006
Eccles, J. S., & Wigfield, A. (2002). Motivational beliefs, values, and goals. Annual Review of
Psychology, 53, 109 – 132.
Farkas, N. (1999). Dutch Science Shops: Matching community needs with university R & D.
Science Studies, 2, 33–47.
Freitag, A. (2014). Naming, framing, and blaming: Exploring ways of knowing in the
deceptively simple question “What is water quality?” Human Ecology, 42, 325–337.
doi:10.1007/s10745-014-9649-5
Frodeman, R., Holbrook, J. B., Bourexis, P. S., Cook, S. B., Diederick, L., & Tankersley, R. A.
(2013). Broader Impacts 2.0: Seeing - and Seizing—the Opportunity. Bioscience, 63(3),
153–155. doi:10.1525/bio.2013.63.3.2
Fuller, B. (2011). Enabling problem-solving between science and politics in water conflicts:
impasses and breakthroughs in the Everglades, Florida, USA. Hydrological Sciences
Journal, 56(4), 576–587. doi:10.1080/02626667.2011.579075
Gregory, R., Arvai, J., & Gerber, L. R. (2013). Structuring decisions for managing threatened
and endangered species in a changing climate. Conservation Biology : The Journal of the
Society for Conservation Biology, 27(6), 1212–21. doi:10.1111/cobi.12165
Grin, J., & van de Graaf, H. (1996). Technology assessment as learning. Science, Technology &
Human Values, 21(1), 72–99.
130
Halofsky, J. E., Peterson, D. L., Furniss, M. J., Joyce, L. A., Millar, C. I., & Neilson, R. P.
(2011). Workshop approach for developing change adaptation strategies and actions for
natural resource management agencies in the United States. Journal of Forestry, (June),
219–225.
Hansen, J. A., & Lehmann, M. (2006). Agents of change: universities as development hubs.
Journal of Cleaner Production, 14(9-11), 820–829. doi:10.1016/j.jclepro.2005.11.048
Hildén, M. (2011). The evolution of climate policies – the role of learning and evaluations.
Journal of Cleaner Production, 19(16), 1798–1811. doi:10.1016/j.jclepro.2011.05.004
Holman, I. P., Rounsevell, M. D. A., Cojacaru, G., Shackley, S., McLachlan, C., Audsley, E., …
Richards, J. A. (2008). The concepts and development of a participatory regional integrated
assessment tool. Climatic Change, 90(1-2), 5–30. doi:10.1007/s10584-008-9453-6
Holzkämper, A., Kumar, V., Surridge, B. W. J., Paetzold, A., & Lerner, D. N. (2012). Bringing
diverse knowledge sources together - a meta-model for supporting integrated catchment
management. Journal of Environmental Management, 96(1), 116–27.
doi:10.1016/j.jenvman.2011.10.016
Hulse, D. W. and Gregory, S. V. 2001. Alternative futures as an integrative frameworks for
riparian restoration of large rivers. Pages 194-212 in V. H. Dale and R. A. Haeuber,
editors. Applying ecological principle to land management. Springer-Verlag, New York,
New York, USA.
Huntington, H. P., Brown-schwalenberg, P. K., Frost, K. J., Fernandez-gimenez, M. E., Norton,
D. W., & Rosenberg, D. H. (2002). Observations on the workshop as a means of improving
communication between holders of traditional and scientific knowledge. Environmental
Management, 30(6), 778–792. doi:10.1007/s00267-002-2749-9
Johnson, T. R. (2011). Fishermen, scientists, and boundary spanners: Cooperative research in the
U.S. Illex squid fishery. Society & Natural Resources: An International Journal, 24(3),
242–255. doi:10.1080/08941920802545800
Kastenhofer, K., Bechtold, U., & Wilfing, H. (2011). Sustaining sustainability science: The role
of established inter-disciplines. Ecological Economics, 70(4), 835–843.
doi:10.1016/j.ecolecon.2010.12.008
Kearney, J., Berkes, F., Charles, A., & Wiber, M. (2007). The role of participatory governance
and community-based management in integrated coastal and ocean management in Canada.
Coastal Management, 35(1), 79–104. doi:10.1080/10.1080/08920750600970511
Kloprogge, P., & van der Sluijs, J. P. (2006). The inclusion of stakeholder knowledge and
perspectives in integrated assessment of climate change. Climatic Change, 75, 359–389.
doi:10.1007/s10584-006-0362-2
131
Landry, R., Amara, N., & Lamari, M. (2001). Utilization of social science research knowledge in
Canada. Research Policy, 30, 333–349.
Lang, D. J., Wiek, A., Bergmann, M., Stauffacher, M., Martens, P., Moll, P., … Thomas, C. J.
(2012). Transdisciplinary research in sustainability science: practice, principles, and
challenges. Sustainability Science, 7(Supplement 1), 25–43. doi:10.1007/s11625-011-0149x
Lautenbach, S., Berlekamp, J., Graf, N., Seppelt, R., & Matthies, M. (2009). Scenario analysis
and management options for sustainable river basin managementt: Application of the Elbe
DSS. Environmental Modelling & Software, 24, 26–43. doi:10.1016/j.envsoft.2008.05.001
Lawler, J. J., Tear, T. H., Pyke, C., Shaw, M. R., Gonzalez, P., Kareiva, P., … Pearsall, S.
(2010). Resource management in a changing and uncertain climate. Frontiers in Ecology
and the Environment, 8(1), 35–43. doi:10.1890/070146
Lemos, M. C., & Morehouse, B. J. (2006). The co-production of science and policy in integrated
climate assessments. Global Environmental Change, 15(2005), 57–68.
doi:10.1016/j.gloenvcha.2004.09.004
Lengwiler, M. (2008). Participatory approaches in science and technology: Historical origins and
current practices in critical perspective. Science, Technology & Human Values, 33(2), 186–
200.
Lester, S. E., McLeod, K. L., Tallis, H., Ruckelshaus, M., Halpern, B. S., Levin, P. S., …
Parrish, J. K. (2010). Science in support of ecosystem-based management for the US West
Coast and beyond. Biological Conservation, 143(3), 576–587.
doi:10.1016/j.biocon.2009.11.021
Leydesdorff, L., & Ward, J. (2005). Science shops: a kaleidoschope of science-society
collaborations in Europe. Public Understanding of Science.
doi:10.1177/0963662505056612
Li, L. C., Grimshaw, J. M., Nielsen, C., Judd, M., Coyte, P. C., & Graham, I. D. (2009). Use of
communities of practice in business and health care sectors: A systematic review.
Implementation Science, 4(27), 1–9. doi:10.1186/1748-5908-4-27
Lienert, J., Monstadt, J., & Truffer, B. (2006). Future scenarios for a sustainable water sector: A
case study from Switzerland. Environmental Science & Technology, 40(2), 436–442.
doi:10.1021/es0514139
Lynam, T., de Jong, W., Sheil, D., Kusumanto, T., & Evans, K. (2007). A review of tools for
incorporating community knowledge, preferences, and values into decision making in
natural resources management. Ecology and Society, 12(1).
132
Lynam, T., Drewry, J., Higham, W., & Mitchell, C. (2010). Adaptive modelling for adaptive
water quality management in the Great Barrier Reef region, Australia. Environmental
Modelling and Software, 25(11), 1291–1301. doi:10.1016/j.envsoft.2009.09.013
Mackenzie, J., Tan, P. L., Hoverman, S., & Baldwin, C. (2012). The value and limitations of
Participatory Action Research methodology. Journal of Hydrology, 474, 11–21.
doi:10.1016/j.jhydrol.2012.09.008
Mader, M., Mader, C., Zimmermann, F. M., Görsdorf-Lechevin, E., & Diethart, M. (2013).
Monitoring networking between higher education institutions and regional actors. Journal
of Cleaner Production, 49, 105–113. doi:10.1016/j.jclepro.2012.07.046
Mahmoud, M., Liu, Y., Hartmann, H., Stewart, S., Wagener, T., Semmens, D., … Winter, L.
(2009). A formal framework for scenario development in support of environmental
decision-making. Environmental Modelling and Software, 24(7), 798–808.
doi:10.1016/j.envsoft.2008.11.010
Manring, S. L. (2014). The role of universities in developing interdisciplinary action research
collaborations to understand and manage resilient social-ecological systems. Journal of
Cleaner Production, 64, 125–135. doi:10.1016/j.jclepro.2013.07.010
Martin-Sempere, M. J., Garzon-Garcia, B., & Rey-Rocha, J. (2008). Scientists’ motivation to
communicate science and technology to the public: surveying participants at the Madrid
Science Fair. Public Understanding of Science, 17(3), 349–367.
doi:10.1177/0963662506067660
Matso, K. E., & Becker, M. L. (2014). What can funders do to better link science with decisions?
Case studies of coastal communities and climate change. Environmental Management,
54(6), 1356–71. doi:10.1007/s00267-014-0347-2
McClure, M. M., Alexander, M., Borggaard, D., Boughton, D., Crozier, L., Griffis, R., … Van
Houtan, K. (2013). Incorporating climate science in applications of the US endangered
species act for aquatic species. Conservation Biology : The Journal of the Society for
Conservation Biology, 27(6), 1222–33. doi:10.1111/cobi.12166
Nadkarni, N. M., & Stasch, A. E. (2013). How broad are our broader impacts? An analysis of the
National Science Foundation’s Ecosystem Studies Program and the Broader Impacts.
Frontiers in Ecology and the Environment2, 11(1), 13–19. doi:10.1890/110106
National Science Board. (2011). Merit Review Criteria. Review and Revisions. Retrieved from
papers2://publication/uuid/910994F5-3EE1-4236-8346-5602486DA1D2
National Science Foundation. (2012). Proposal and award policies and procedures guide.
Retrieved from http://www.nsf.gov/pubs/policydocs/pappguide/nsf13001/nsf13_1.pdf
133
Pahl-wostl, C. (2007). Transitions towards adaptive management of water facing climate and
global change. Water Resources Management, 21, 49–62. doi:10.1007/s11269-006-9040-4
Pahl-wostl, C., Craps, M., Dewulf, A., Mostert, E., Tabara, D., & Taillieu, T. (2007). Social
learning and water resources management. Ecology and Society, 12(2), 5.
Pahl-wostl, C., Sendzimir, J., Jeffrey, P., Aerts, J., Berkamp, G., & Cross, K. (2007). Managing
change toward adaptive water management through social learning. Ecology and Society,
12(2), 30.
Patton, M. Q. (2002). Qualitative research and evaluation methods. 3rd edition. Thousand Oaks,
CA: Sage.
Pearson, G., Pearson, G., Pringle, S. M., Pringle, S. M., Thomas, J. N., & Thomas, J. N. (1997).
Scientists and the public understanding of science. Science, 6, 279–289.
Podolny, J. M., & Page, K. L. (1998). Network forms of organization. Annual Review of
Sociology, 24, 57–76.
Raddick, M. J., Bracey, G., Gay, P. L., Lintott, C. J., Murray, P., Schawinski, K., …
Vandenberg, J. (2010). Galaxy Zoo: Exploring the motivations of citizen science
volunteers. Astronomy Education Review, 9, 15. doi:10.3847/AER2009036
Rayner, S., Lach, D., & Ingram, H. (2005). Weather forecasts are for wimps*: Why water
resource managers do not use climate forecasts. Climatic Change, 69, 197–227.
Reed, M. S., Evely, A. C., Cundill, G., Fazey, I., Glass, J., & Laing, A. (2010). What is social
learning ? Ecology and Society.
Riley, C., Matso, K., Leonard, D., Stadler, J., Trueblood, D., & Langan, R. (2011). How research
funding organizations can increase application of science to decision-making. Coastal
Management, 39(3), 336–350. doi:10.1080/08920753.2011.566117
Rittel, H. W. J., & Webber, M. M. (1973). Dilemmas in a general theory of planning. Policy
Sciences, 4(2), 155–169.
Robinson, C. J., & Wallington, T. J. (2012). Boundary work: Engaging knowledge systems in
co-management of feral animals on indigenous lands. Ecology and Society, 17(2), 16.
Rotman, D., Preece, J., Hammock, J., Procita, K., Hanse, D., Parr, C., … Jacobs, D. (2012).
Dynamic changes in motivation in collaborative citizen-science projects. In Session: Civic
and Community Engagement (pp. 217–226). doi:10.1145/2145204.2145238
Santelmann, M., Freemark, K., White, D., Nassauer, J., Clark, M., Danielson, B., Eilers, J.,
Cruse, R.M., Galatowitsch, S., Polasky, S., Vache, K., and J. Wu. 2001. Applying
ecological principles to land-use decision making in agricultural watersheds. Pages 226-
134
252 in V.H. Dale and R. A. Haeuber, editors. Applying ecological principles to land
management. Springer-Verlag, New York, New York, USA.
Sheppard, S. R. J., Shaw, A., Flanders, D., Burch, S., Wiek, A., Carmichael, J., … Cohen, S.
(2011). Future visioning of local climate change: A framework for community engagement
and planning with scenarios and visualisation. Futures, 43(4), 400–412.
doi:10.1016/j.futures.2011.01.009
Shirk, J. L., Ballard, H. L., Wilderman, C. C., Phillips, T., Wiggins, A., Jordan, R., … Bonney,
R. (2012). Public participation in scientific research: A framework for deliberate design.
Ecology and Society, 17(2), 29. doi:10.5751/ES-04705-170229
Smith, J. B., Strzepek, K., Rozaklis, L., Ellinghouse, C., & Hallett, K. (2009). The Potential
Consequences of Climate Change for Boulder Colorado’s Water Supplies.
Snover, A. K., Mantua, N. J., Littell, J. S., Alexander, M. A., McClure, M. M., & Nye, J. (2013).
Choosing and using climate-change scenarios for ecological-impact assessments and
conservation decisions. Conservation Biology: The Journal of the Society for Conservation
Biology, 27(6), 1147–57. doi:10.1111/cobi.12163
Sol, J., Beers, P. J., & Wals, A. E. J. (2013). Social learning in regional innovation networks:
trust, commitment and reframing as emergent properties of interaction. Journal of Cleaner
Production, 49, 35–43. doi:10.1016/j.jclepro.2012.07.041
Spiegel, J. M., Breilh, J., Beltran, E., Parra, J., Solis, F., Yassi, A., … Parkes, M. (2011).
Establishing a community of practice of researchers, practitioners, policy-makers and
communities to sustainably manage environmental health risks in Ecuador. BMC
International Health and Human Rights, 11 Suppl 2(Suppl 2), S5. doi:10.1186/1472-698X11-S2-S5
Star, S. L., & Griesemer, J. R. (1989). Institutional ecology, 'translations’ and boundary objects:
Amateurs andprofessionals in Berkeley’s Museum of Vertebrate Zoology, 1907-39. Social
Studies of Science, 19(3), 387–420.
Stubbs, M., & Lemon, M. (2001). Learning to network and networking to learn: Facilitating the
process of adaptive management in a local response to the UK’s National Air Quality
Strategy. Environmental Management, 27(8), 321–334. doi:10.1007/s002670010152
Swart, R. J., Raskin, P., & Robinson, J. (2004). The problem of the future: sustainability science
and scenario analysis. Global Environmental Change, 14(2), 137–146.
doi:10.1016/j.gloenvcha.2003.10.002
Tuler, S. (1998). Learning through participation. Human Ecology Review, 5(1), 58–60.
135
Tullos, D., Brown, P. H., Kibler, K., Magee, D., Tilt, B., & Wolf, A. T. (2010). Perspectives on
the salience and magnitude of dam impacts for hydro development scenarios in China.
Water Alternatives, 3(2), 71–90.
Turnhout, E., Stuiver, M., Klostermann, J., Harms, B., & Leeuwis, C. (2013). New roles of
science in society: Different repertoires of knowledge brokering. Science and Public Policy,
40, 354–365. doi:10.1093/scipol/scs114
Van Herk, S., Zevenbergen, C., Ashley, R., & Rijke, J. (2011). Learning and Action Alliances
for the integration of flood risk management into urban planning: a new framework from
empirical evidence from The Netherlands. Environmental Science & Policy, 14(5), 543–
554. doi:10.1016/j.envsci.2011.04.006
Vaske, J. V. (2008). Surveey research and analysis: Applications in parks, recreation and human
dimensions. State College: PA: Venture.
Voinov, A., & Bousquet, F. (2010). Modelling with stakeholders. Environmental Modelling &
Software, 25(11), 1268–1281. doi:10.1016/j.envsoft.2010.03.007
Webler, T. (1998). Beyond science: Deliberation and analysis in public decision making. Human
Ecology Review, 5(1), 61–62.
Weible, C. M., & Sabatier, P. A. (2009). Coalitions, science, and belief change: Comparing
adversarial and collaborative policy subsystems. Policy Studies Journal, 37(2), 195–212.
doi:10.1111/j.1541-0072.2009.00310.x
Wolf, B., Lindenthal, T., Szerencsits, M., Holbrook, J. B., & Heb, J. (2013). Evaluating research
beyond scientific impact: How to include criteria for productive interactions and impact on
practice and society. GAIA, 22(2), 104–114.
Yang, L., Wu, J., & Shen, P. (2013). Roles of science in institutional changes: The case of
desertification control in China. Environmental Science & Policy, 27(37), 32–54.
doi:10.1016/j.envsci.2012.10.017
136
APPENDICES
137
Appendix A. Semi-Structured Interview Guide
1. What has been your involvement with the science-stakeholder engagement process?
a. How have you participated in the LAN?
b. Why have you continued to participate or stopped participating?
c. What do you see your role as in this project?
d. How do you contribute to the study?
2. Why are you involved professionally involved in WW2100?
3. What were your expectations coming in to the project?
a. Have they been met?
b. What do you think has helped or hindered the meeting of your expectations?
4. What has been the biggest challenge and/or success you have observed during this
process?
a. To what do you attribute it?
b. How was success achieved or a challenge overcome in your opinion?
5. Are you satisfied with the results of the project as you see them?
a. How?
b. What are situations where you may already be using the results?
6. How might your participation in WW2100 have impacted the way you perform your job?
a. Has your professional network changed as a result of participation?
7. What do you think about this project’s engagement process in comparison to other
scientific research you have been a part of or utilized?
138
Appendix B. Survey Instrument and Results
Thank you for participating in the Willamette Water 2100 (WW2100) Learning and Action
Network (LAN) researcher-stakeholder engagement survey. Your responses will help identify
future pathways and barriers to successful collaborative research and its applications. Your
responses will be kept confidential and there are no risks to you in participating in this
survey. Your name will not be associated with your responses and your privacy will be protected
to the maximum extent allowable by law. Your anonymous response to this survey and any of
the questions is completely voluntary. You indicate your voluntary agreement to participate by
completing and returning the survey. Some parts of this questionnaire may look familiar as the
questions come from surveys conducted early in the WW2100 project. This is part of the
research design and we ask that you answer these questions again. If you have any questions
about this project, please feel free to e-mail Laura Ferguson at fergusla@onid.oregonstate.edu.
The questionnaire should take approximately 30 minutes to complete. Please do so at your
earliest convenience. It is only with your generous help that our research can be successful.
Thank you for your time and consideration.
1. In the Willamette Water 2100 Project, I was a... (Please select one)
#
1
2
3
4
5
6
Answer
Research Team Member (Principal
Investigator)
Research Team Member
(Contracted)
Student
Stakeholder (Technical Advisory
Group Member)
Stakeholder (General)
Other (Please specify)
Total
Response
%
17
13%
13
10%
9
7%
28
22%
40
23
130
31%
18%
100%
2. In what professional capacity are you acting in the Willamette Water 2100 project?
(Please select one)
#
1
2
3
4
5
6
7
8
9
10
11
12
13
Answer
Water resources planner
Water resources engineer
Water resources regulator
Water resource policy staff
Irrigation district manager
Policy maker
Fisheries biologist
Hydrologist
Water utilities manager
Educator
Environmental NGO staff
Agriculture
Other
Total
Response
15
7
6
3
0
13
3
10
8
14
5
8
34
126
%
12%
6%
5%
2%
0%
10%
2%
8%
6%
11%
4%
6%
27%
100%
139
3. I work in the ..... area of the Willamette Basin. (Please select one)
#
1
2
3
0
Answer
Lower (Clackamas, Tualatin,
Portland Metro)
Middle (Corvallis, Albany, Salem)
Upper (Eugene and surrounding
tributaries)
I do not work in the Willamette
Valley
Total
Response
%
33
27%
55
44%
28
23%
8
6%
124
100%
4. Use the slider to indicate the percentage of the available water each sector currently uses
in your opinion. (Select a percentage for each item)
#
Answer
1
2
3
4
Municipal
Agriculture
Industry
Recreation
Energy
Production
Fish and
Wildlife
5
6
Average
Value
Standard
Deviation
Responses
14.82
21.33
12.92
17.64
118
118
117
105
90.00
16.66
17.68
110
100.00
25.24
26.58
112
Min Value
Max Value
2.00
1.00
1.00
1.00
85.00
86.00
73.00
100.00
0.00
0.00
19.57
50.19
14.39
15.50
140
5. To what extent do you disagree or agree with each of the following statements? (Check
one item for each statement)
#
1
2
3
4
Question
Currently,
the
Willamette
Valley has
enough
water for
human and
ecological
needs.
In 10
years, the
Willamette
Valley
will have
enough
water for
human and
ecological
needs.
In 50
years, the
Willamette
Valley
will have
enough
water for
human and
ecological
needs.
In 100
years, the
Willamette
Valley
will have
enough
water for
human and
ecological
needs.
Disagree
Neither
Agree
nor
Disagree
Agree
Strongly
Agree
Total
Responses
Mean
3
25
11
63
16
118
3.54
8
35
20
48
7
118
3.09
29
36
32
19
3
119
2.42
39
34
30
14
1
118
2.19
Strongly
Disagree
141
6. Currently, how much risk does each of the following pose to Oregon's water quantity?
(Check one item for each statement)
#
1
2
3
4
5
6
7
8
9
10
11
Question
Agricultural
practices (e.g.
irrigation)
Forestry
practices (e.g.
timber
harvest)
Hydroelectric dams
Drought
Conditions
Climate
Change
Population
Growth
Water
privatization
Industry
Private Wells
Historical
appropriation
of water (e.g.
water rights)
Appropriation
towards
habitat and
ecological
needs
No Risk
Minor
Risk
Moderate
Risk
High Risk
Total
Responses
Mean
6
24
56
31
117
2.96
15
59
37
6
117
2.29
26
57
24
8
115
2.12
0
7
40
70
117
3.54
0
13
46
58
117
3.38
1
23
46
46
116
3.18
17
45
36
16
114
2.45
12
16
67
60
29
33
6
6
114
115
2.25
2.25
9
38
48
22
117
2.71
26
52
32
7
117
2.17
142
7. Currently, how much risk does each of the following pose to Oregon's water quality?
(Check one item for each statement)
#
1
2
3
4
5
6
7
8
9
10
11
Question
Agricultural
practices (e.g.
irrigation)
Forestry
practices (e.g.
timber
harvest)
Hydroelectric dams
Drought
Conditions
Climate
Changes
Population
Growth
Water
privatization
Industry
Private Wells
Historical
appropriation
of water (e.g.
water rights)
Appropriation
towards
habitat and
ecological
needs
No Risk
Minor
Risk
Moderate
Risk
High Risk
Total
Responses
Mean
3
11
45
53
112
3.32
4
19
53
36
112
3.08
15
49
41
7
112
2.36
2
22
48
40
112
3.13
2
19
56
35
112
3.11
2
17
50
43
112
3.20
23
62
16
10
111
2.12
5
23
30
71
55
14
22
4
112
112
2.84
1.99
24
48
31
8
111
2.21
73
25
11
4
113
1.52
143
8. Besides employment, list up to five characteristics of the Willamette Valley that you
value, and make it a place that you want to live. (Type your answer below)
Text Response
access to natural areas, clean air, clean water, access to rural areas, access to mountains, access to coast
Relatively unpolluted environment, access to the outdoors, recreation opportunities, wildlife watching, good food.
Nature, opportunities
Mixed land uses Open spaces Good quality of life The river
Proximity to ocean. Proximity to mountains. Moderate to progressive political climate. Climate Population's overall
respect to others and the environment
Ecological quality, moderate climate, available water, positive citizenship
Swimmable, fishable, canoeable/kayakable
Temperate climate, proximity to outdoor recreation areas, open spaces,
home to listed species, beautiful, productive land, cheap power, flood control provided
Recreation, wildlife, fisheries, aesthetics, agriculture, forestry,
Managed urban growth (could be better/stronger)
Recreation and outdoors, seasonality, sustainability ethic
great climate, scenic beauty, amazing supply of fresh food, outstanding growing conditions, fairly well-educated and
enlightened populations (compared with much of the rest of the world,
Scenery, diverse agriculture, locally grown produce, wineries, access to recreational opportunities.
scenic beauty, healthy food sources, access to fresh waterways, access to Pacific Ocean, native cultural heritage
Proximity to resource areas, mtn,ocean, ect. Climate available resources recreational opportunities
Clean air and water access to outdoor activities the people
It's historic land use laws that have kept agriculture a prominent part of the landscape and economy. It isn't too
crowded (yet). The mild weather (excluding Nov - Apr).
Growing conditions, local food, beauty, educated populace, water availability
Diversity of land uses. Proximity to urban and rural area Scenic Beauty Recreation opportunities Mild weather
scenic beauty, diverse landscapes, temperate climate, charming communities, recreational opportunities (inc. farmto-table activities)
Friendly Community Educational Opportunities Friends Outdoor activities Diverse Environment
long growing season, good recreational opportunities, scenic, environmental ethic (for the most part), moderate
weather
temperate rainforest, unique bio-region, recreation, access to natural spaces, integration of water into urban
environments
temperate climate, diverse economic activity, excellence in educational opportunity, world-class outdoor and
recreational activity, healthy population
natural setting, people, habitat, beauty
Abundant water, low conflict, moderate population, high level of education
safety, beauty, access to nature and urban areas
Landscape (eg rivers mts forests,fauna,etc). I used to be proud of the cutting edge/progressive stands the state and
local govts took. I am hopeful we will have a resurgence. I love the short distance to the Cascades and the Pacific.
I appreciate the relatively small population we have and consequently the easy access to most everything in the
valley.
It is a beautiful place with access to hiking in mountains and foothills, near the coast. There are good schools and a
culture that values education Great opportunities to get fresh local food, especially produce and good beer and wines
mild winters, open space, easy access to forests and oceans, great birdwatching
the climate, the landscape, the mix of places, the urban growth boundaries, the history
Agricultural, Natural, Beauty
outdoor recreation, good food/beer, no traffic, farmers markets
Mountains and forests, snow, culture, variety of food grown, and wildlife
Water Quality, Fisheries Resources, Viewshed, Water Supply, Recreation
Adequate resources for healthy living, climate that sustains my desire to be a gardener, access to recreational
regions for fishing, hiking, biking, and camping, the Valley's lush vegetation speaks to my soul, and I ahve an
amazing job as a public school teacher in an area that values education.
Climate, scenery, outdoor recreation, Oregon's land use system (close in agriculture and open space), connectivity
(Amtrak Cascades)
144
River and tribs support native salmon and steelhead; provide clean water from tribs for drinking; adequate water
instream and riparian habitat maintains (barely) habitat functions for fish and wildlife; great place for recreation
1 four seasons. 2 its green 3 moderate weather 4 outdoor recreation 5 not to crowded yet
Quality of life, functional ecosystems, low population density
Relatively natural environment, low population density, relatively functioning stram systems, high quality
localfood, relatively educated population
I do not live in the Willamette Valley - but its scenery, clean air and water, recreational opportunities and
universities are attractive
recreation, people
relatively few people relative to US; food production; cultural amenities; access to coast, recreational amenities
such as hiking/biking trails, rafting & fishing opportunities.
Environment, close to natural features (forests, nature), diverse land use(combination of nature, ag and urban), land
use planning, quality of life (access to the built and natural environment), most people seem to value a balance
between urban, ag, forestry and natural. close to
Wildlife and/or natural resources in general Scenic value Food availability Educational and social opportunities
Clean water Climate amenable to producing fresh food Compact urban development (with transit &other urban
amenities inside UGBs) and rural outside UGBs Healthy wildlife and adequate habitat Natural beauty
(1) the beauty (natural and human-mediated); (2) the people (culture, attitude, values, spirit); (3) proximity to other
cultural, recreational, intellectual; educational and other opportunities;
Weather, liberal politics, proximity to mountains and ocean, high education rates, balance of environmental
protection and industrial growth
The people and moderate political attitudes; Moderate climate; Urban planning that reduces sprawl; access to cities,
mountains and ocean
Access to Ocean and Mountains; Mild Winters; Cultural Activities, music art; Progressive mentality; Plane, Bus and
Rail service
healthy vegetatin and wildlife, healthy creeks and rivers, universities and towns with cultural offerings, incredible
drinking water, passive nature-immersion recreation opportunities close-by
access to recreation and open space, relatively clean water resources, access to urban amenities, opportunity for
regional food system
1. Dynamic landscapes 2. Thriving fish and fauna 3. Rich in Culture 4. Open spaces 5. Economic stability
hosts a university good hospitals good public services healthy environment/green spaces/access to nature
natural beauty, fresh water, progressive social culture, good food, access to recreation
Rain; trees; mountains;
access to clean water, water recreation opporutnities, resilience to climate change, local food production, climate
Publicly accessible water resources Fishing opportunities Hunting opportunities Protection of open space
Consolidation of development
recreation, climate, good local government, lots of public lands, limited congestion
rural lands recreation wildlife agriculture livable cities
Recreation access Wildlife Clean water
moderate climate, ecological value, proximity to recreation, general political/social climate
diversity of recreation opportunities, local agriculture, scenic beauty, good schools/universities, nice cultural
opportunities
Diversity of the landscape and agriculture Progressive Access to water Quality of life cultural tolerance
Ease of access to natural environment/wilderness/protected areas, locally grown organic food, PNW cultural values
Nice weather, rural landscape, proximity to natural areas, interesting culture, nice people and friends
aesthetics, recreation, fishing
climate, politics, agriculture,recreation, population
access to wilderness, clean air and water, low population density, ease of transportation
climate
Topography, wealth of environmental resources, temperate climate, lack of large cities, UGB
Opportunity for recreation (camping, river access, hiking), moderate climate, progressive communities, functional
institutions, opportunities for eduction
Clean, adequate flow, fish abundance, healthy, responsive
natural beauty, land use planning, agriculture, forestry, universities
Empowered citizens; access to recreation; healthcare availability; clean water; clean air
Environment, education,
145
Open space, smaller cities, access to other parts of the state, climate, affordable
Open space, wetlands, agriculture, access to the river, biological diversity
climate, agricultural uses, small communities, greenways/ vegetation, ease of working with policy makers
Livability Commerce Location Education Home
Environmental quality, general livability, recreational opportunities,
Natural beauty, outdoor recreation, abundant resources, progressive values.
sufficient water supply for cities and farms; fish, wildlife and habitat protection; preservation of open space; mild
climate
close proximity to beaches, lakes, mountains, fishing, Portland. Moderate climate Good soils Diversity of plant &
animal species Small towns
Consolidated development Agricultural values Natural beauty Abundance of rivers and streams Open space access
Ecological functions- fish, wildlife and natural areas; adequate water resources compared to other areas in the
country; advanced land use practices compared to other areas in the country; access to federal lands; comparatively
clean air and water compared to other areas in the country and world; access to native american culture and practices
Recreation, climate, community, schools, family
Climate, scenery, towns, waterways
Agricultural, forestry, open space, wildlife, clean water
climate, forested landscape, access to parks, agriculture, biodiversity
Farm and nursery operations, recreation, clean water
good environmental conditions, air and water quality good educational resources and access to healthcare
recreational opportunities (outdoors, sports etc) lack of crowding adequate infrastructure - roads, electricity, sewer,
etc
Recreation availability, clean air and water, not overpopulated, scenic
4 distinct seasons, plenty of water, low population generally, diverse landscapes, I grew up there.
vegetation (greenness), temperate climate, low population
ecological diversity, abundant produce, recreational opportunities
Proximity to Cascade Mountains, proximity to coast, excellent food production potential, clean abundant water,
forward thinking people/leaders including two universities
Access to natural spaces and intact ecosystems, clean drinking watner from the tap, vibrant healthy culture and
community, educated public, access to educational resources and programs,
Landscape (Mountains, Rivers, Ocean). Community in general values the environment. Water quality, Wildlife
diversity, Weather
146
9. List up to five concerns that you have about quality of life in the Willamette Valley in
the future. (Type your answer below)
Text Response
extent of sand and gravel extraction, extent of clear-cutting, population growth, loss of wildlife habitat, public
school quality
Poorly managed population growth, environmental damage from droughts and climate change
Lack of intelligent land use planning and transportation corridors, lack of public green spaces that are
interconnected, sprawling urban growth
Population growth, climate change, changes to land use laws, rising home prices
Population growth as it relates to cost/quality of living. Population growth as is relates to property cost. Population
growth as it relates to environmental degredation. Lack of snow (climate change). Job prospects (lack of
employment). And population growth again.
Over population, over consumption, inadequate policy development via political processes
Algae growth in pool above WIllamette Falls
populaton increase, increased traffic, sprawl,
Too populated,
population growth, economy, urban growth, loss of sustainable agriculture,
urban growth onto farmland, traffic, water quality
Impacts on natural resources due to climate change impacts -wildfire, low summer streamflows, lower snowpack
huge population growth, due largely to climate refugees from other regions of the country
Population growth and associated land use issues, increased freight traffic on I-5, decreasing water quality,
decreases in locally produced food due to water shortages
water scarcity, overpopulation, habitat loss, climate change, invasive species
Urbanization of farm lands; population density; overallocated resources (water); Emergency response availability;
Short sighted use of resources
increasing pressure between ag an urban areas, having a big enough tax base to support state and local govt
activities
Suburban/urban sprawl.
Population growth, water abuse, land use abuse, drought, overpopulation(cannot be stated enough times)
Traffic congestion
population growth, increased traffic, rural-urban conflict, growing imbalance between rooted long-time residents
and oblivious new arrivals
Increasing population growth, increase in social issues (ex. wealth gap, education, health, labor rights...), Inequity
(ex. water), climate change, urbanization
increasing population and development, water availability, water quality, climate change
population growth, growing corporate timber and ag practices, GMO, water rights, decisive political & social
environment
population growth, climate change, disparity of economic wealth, sad state of public education - inaccessibility of
public higher education
population growth,
Population growth, bad zoning, declining snow
population growth, climate change
Population explosion. Out-of-state Need for water. Lack of transportation investment.
Absence of "crises" re
climate change preparation.
Population growth and changes in state planning goals that might allow extensive development and urban and
suburban sprawl
population growth, climate change, potential for economic growth with minimal ecological impacts, degree of
homelessness, sustainable agriculture
sprawl, lack of transportation alternatives to driving, cost of living
Contamination by Ag, people, industry
too far removed from what urban life entails, too much advocacy
population growth and climate change effects
Resource Depletion, Population Increases, Polution, Climate Change,
Increasing population brings me worry. Increased pollution (mainly the garbage I see on the ground). Temperatures
are increasing, making my need to irrigate my extensive gardens a higher and more frequent priority. Crime rates
147
appear to be on the rise, but this may be due to increased media access. Traffic is increasing.
Lack of intrastructure investment, lack of significant mass transit, loss of close-in agriculture, air quality
deterioration, wildfire.
too many people, dense development and overconsumption of resources, land use that allows owners to destroy
riparian habitat
1 over population 2 overzealyrous government policies 3
Degredation of quality of life
increasing population density, rapid development, loss of green space/natural areas, increasingly urbanized
population, loss of ecological diversity
agricultural pollution, rapid population growth
Loss of access to snow for recreation; air quality for increasing fire risk, increasing populations, loss of open space
& farmland; getting too hot in summer
Assumtion that the qualities folks like will always be there with out making the effort to assuer they are, Continue to
assume there is enought water but not do the things necessary to make sure the water is there, not enought enphises
on water conservation, population growth, migration from Calif as the drought worsens
Water quantity and quality, population growth and associated consumption patterns, lack of cultural
diversity/understanding, municipal infrastructure and financing
Increasing traffic, threats to clean water and wildlife habitat, possibility of rapid in-migration & inability to plan in
time to accomodate, potential increase in pollution affecting health in future
(1) water quantity and quality; (2) population growth; (3) corporate farming dominance; (4) environmental quality;
(5) potential impacts that climate variability and change, and what we (ie, society ) does in advance to limit impacts,
mitigate adapt
climate change, inability to modify policies to adapting climate
Political polarization from urban/rural differences and in-migration,
Over Crowding and all associated issues: Air Quality declines; Crowded Rivers and trails; water resources conflicts,
shortages
urbanization (paved-over soil and low density development limit future options), scattered development at urban
edge/rural areas cutting into contiguous habitat and creating multiple pollution sources, pollution by toxics
(pesticides/heavy metals), lack of safety for non-car transportation (bikes sharing roads with cars), lack of
regulation and meaningful fines/enforcement for polluters
population growth, increasingly industrialized agriculture, impacts of agriculture on water quality and quantity,
failure to invest in education, racial and economic disparities
1. Development encroachment ; 2. Water Quantity; 3. Increasing cost of living 4; increase population 5;
government land management
population growth, reduction in values associated with a healthy environment, reduction in social services because
of budgets, loss of green space,
increased population density, impacts of wildfires, impact of earthquake, economic stress due to climate change and
rising energy prices
Population growth
impact of drought cycles, environmental degredation, loss of recreational opportunity, stagnant economy, increased
poverty
Population growth, cost of living, impacts to natural resources, loss of rural character
increased population, increased traffic congestion, agricultural practices like application of herbicides, increased
homelessness, wildfire
unmanaged growth; lack of resources; congestion; degrated environment;
Population growth. Contaminated water.
population growth, job competition, natural resource quality (e.g., ecological) and quantity (e.g., water)
urban sprawl, traffic
Stable funding for public programs . Not in my back yard -NIMBY. Fires/smoke. Invasive species, Aging public
infrastructure. Emerging contaminates
Increasing population and development, increasing water scarcity
Population pressures, economic stagnation and social strains, wildfires and resulting poor air quality, higher
temperatures, drought
growing population, lack of coordination across the whole basin, climate change
incresed population, change in climate, urban sprawl, deforestation
population growth and demographics changes - increased traffic, increased density of housing and people, reduced
148
focus on environmental sustainability
Climate change and resource availability, Cascadia subduction zone, fair tax structure/funding for services,
education
population growth leading to high cost of housing, sprawl and traffic, reduced access to recreation (crowding)
robust natural systems, strong economy, healthy envirnoment, clean water
water quantity, water quality, push back against land use planning, loss of agriculture, loss of forestry
Less winter snowpack at mid- to high elevations; population growth; lack of agreement regarding forest
management;
Housing, jobs, culture
population growth, in migration, ignorant voters, unneccessary new laws, failure to enforce current laws
impacts of growth, water quality and quantity impacts, air quality, reduction in farmland, sprawl
high density urbanization, removal of trees/ vegetation especially in the urban areas, traffic gridlock, excessive
regulatory atmosphere
None
Growing traffic congestion/commuting times, lack of a cohesive strategy for managing natural resources, inability to
bring a sense of priority, proportion and watershed context to our management of the watershed.
Overpopulation, housing prices, economic inequality, corrupt democratic process..
vast urban expansion due to population growth; water quality concerns due to higher temperatures (i.e. algal
blooms); late summer water supply
threats to farmland development from small, cumulative projects; excess water use by nurseries and unwise water
use (e.g. irrigating grass seed); lack of protection of drinking water sources & cooperative protections; impacts from
storage projects
Population growth and resulting urbanization
great increases in population growth, climate change vulnerabilities, degradation of ecological functions and values
that make the Valley unique,
Climate migrants, crowding, traffic,
Population growth, urban sprawl, pollution
Social pressure to change agricultural and forestry and push them out of western Oregon
drought, flooding, over population, habitat change, floodplain encroachment
Economic viablility of agriculture; water supply for municipalities, conservation and agriculture
population growth, economic opportunities that provide employment with living wages, resiliency of ecosystems
Drought, population growth/urban sprawl, severity and frequency of wildfires
Concern about California migration, climate change (decreasing snowpack), extreme environmental activism that
create restrictive policy
assholes, corruption of Oregon land use and water appropriation laws, GM crops, corporate takeover
human population growth, urbanization, habitat destruction
increased development pressures along rivers (loss of riparian area, revetment, septic systems), increasing frequency
of harmful algal blooms, increasing population growth, loss of snow pack
loss of ecosystem function and services, degraded water quality, population growth
Drought, climate change, fish and wildlife species not being able to adapt, population growth
Aerial pesticide spraying, population growth, climate change, habitat destruction
unplanned growth (urban sprawl), climate change, public transportation
149
10. When considering your motivations to participate in WW2100, to what extent do you
disagree or agree with the following? (Check one for each item)I am participating in the
WW2100 project because....
#
1
2
3
5
6
7
8
9
Question
It is
relevant to
my job
I want to
know more
about water
in the
Willamette
Valley
I am
concerned
for water in
the future
Other
(Please
specify)
I want new
tools to
address
water issues
It is focused
on the
Willamette
Valley
It has the
potential to
shape the
future
I am the
representing
a larger
group
whose
voice needs
to be heard
Disagree
Neither
Agree
nor
Disagree
Agree
Strongly
Agree
Total
Responses
Mean
1
3
8
53
46
111
4.26
4
1
12
56
36
109
4.09
3
1
8
46
51
109
4.29
0
0
3
2
9
14
4.43
2
0
14
61
31
108
4.10
4
7
17
54
26
108
3.84
3
6
9
61
31
110
4.01
6
21
31
33
17
108
3.31
Strongly
Disagree
150
11. Active involvement indicates interacting with the project in any way, including reading
newsletters or webpages as well as attending events. Please mark the years in which you
have been actively involved in the WW2100 researcher-stakeholder engagement process.
(Select all that apply)
#
1
2
3
4
5
Answer
2011
2012
2013
2014
2015
Response
60
67
74
83
68
%
58%
64%
71%
80%
65%
12. In which year would you say you were most actively involved in the WW2100
researcher-stakeholder engagement process?
#
1
2
3
4
5
15
Answer
2011
2012
2013
2014
2015
I was equally involved throughout
all years
Total
Response
%
13
13
14
34
10
12%
12%
13%
31%
9%
24
22%
108
100%
151
13. On average how often did you engage in the following Willamette Water 2100 activities
during the year of your greatest participation in WW2100? (Check one item for each
statement)
# Question
1
2
3
4
5
6
E-mailed
a research
team
member
E-mailed
a
stakehold
er
Spoke
with a
research
team
member
Spoke
with a
stakehold
er
Read a
WW2100
Newslette
r
Visited
the
WW2100
website
Neve
r
Onc
ea
Yea
r
2-3
Time
sa
Year
Once
a
Mont
h
2-3
Time
sa
Mont
h
Onc
ea
Wee
k
2-3
Time
sa
Wee
k
Dail
y
Unsur
e
Total
Respons
es
Mea
n
18
16
30
7
9
6
12
4
7
109
3.35
30
22
27
9
8
3
2
0
6
107
2.46
6
16
40
13
6
6
9
10
3
109
3.84
14
17
40
21
4
2
5
2
5
110
3.05
18
14
41
19
3
2
0
0
11
108
2.52
13
25
36
15
7
3
5
0
3
107
2.98
152
14. Please check all of the WW2100 stakeholder engagement events that you have attended
since 2011. (Select all that apply)
#
1
2
3
4
5
6
7
8
9
0
Answer
Summer 2011 Learning and Action
Network Willamette Basin Field
Trip - Upper, Middle, and Lower
Basin Sites
May 2012 Learning and Action
Network Workshop - Chemeketa
Center for Business and Industry,
Salem
February 2013 Learning and Action
Network Workshop - Chemeketa
Center for Business and Industry,
Salem
March 2014 Learning and Action
Network Workshop - Eola Hills
Chemeketa Events Center, Salem
September 2014 Technical Advisory
Workshop - Eola Hills Chemeketa
Events Center, Salem
October 2014 Technical Advisory
Workshop - Eola Hills Chemeketa
Events Center, Salem
November 2014 Technical Advisory
Workshop - Eola Hills Chemeketa
Events Center, Salem
December 2014 Technical Advisory
Workshop - Eola Hills Chemeketa
Events Center, Salem
March 2015 Technical Advisory
Workshop - Eola Hills Chemeketa
Events Center, Salem
I have not attended any stakeholder
engagement events.
Response
%
30
29%
39
38%
35
34%
39
38%
28
27%
26
25%
20
19%
22
21%
20
19%
26
25%
153
15. Please check all of the WW2100 webinars and seminars you have accessed or attended.
(Select all that apply)
#
1
2
3
4
5
6
7
8
9
10
11
0
Answer
January 2013 - Mountain Snowpack
and Vegetation: Implications of
Disturbance - Anne Nolin
January 2013 - Development of
Regional Climate Scenarios and their
Application to WW2100 - Phil Mote
February 2013 - Land-Use Models
for WW2100 - Andrew Plantinga
March 2013 - Flood Frequency and
Water Scarcity in the Santiam Basin
in a Changing Climate - Desiree
Tullos
March 2013 - The Implications of
Climate Change for Reservoir
Operations at Oregon's Cougar Dam
- Allison Danner
April 2013 - Modeling
Ecohydrologic Processes in
Mountain Watersheds - Naomi
(Christina) Tague and Elizabeth
Garcia
May 2013 - Willamette River Basin
Hydrodynamic and Temperature
Modeling - Scott Wells
December 2013 - Potential
Responses of Native and Non-native
Fish Communities to Thermal
Changes in the Willamette River Stan Gregory
May 2014 - The 2014 US National
Climate Assessment Report - Phil
Mote
October 2014 - Modeling the Human
Side of Water Scarcity in the
Willamette Basin - William Jaeger
November 2014 - Climate Change
and Upland Forest Dynamics in the
Willamette River Basin - David
Turner
I have not accessed or attended any
webinars or seminars.
Response
%
33
32%
41
40%
28
27%
25
25%
24
24%
12
12%
19
19%
32
31%
24
24%
39
38%
24
24%
25
25%
16. Laura Ferguson is the graduate student studying the WW2100 researcher-stakeholder
engagement process. Were you formally interviewed by her in 2015?
#
1
0
7
Answer
Yes
No
Unsure
Total
Response
16
85
11
112
%
14%
76%
10%
100%
154
17. In your opinion, how important were the following activities in engaging stakeholders
in WW2100 scientific research? (Check one item for each statement)
#
Question
1
Field Trips
Large Group
Workshops
(~80 people)
Small Group
Workshops
(~30 people)
Webinars
Newsletters
Personal
communication
with research
team members
Other (Please
specify)
2
3
4
5
6
7
Not at all
Important
Slightly
Important
Moderately
Important
Extremely
Important
Total
Responses
Mean
84
3.00
3
19
41
22
85
2.96
1
6
50
30
87
3.25
2
6
27
31
50
47
11
4
90
88
2.78
2.56
1
13
42
37
93
3.24
1
1
2
2
6
2.83
4
17
38
25
155
18. To what extent do you disagree or agree that you expected the following to result from
your participation BEFORE participating in WW2100? (Check one for each item)
#
1
2
3
4
5
6
7
8
9
10
Question
An
integrated
model of
water in
the
Willamette
Valley
Satisfy my
curiosity
Career
Experience
Model
results that
would
contribute
to science
Model
results that
I could use
in my job
An
opportunity
to share
what I
know
An
opportunity
to learn
An
opportunity
to work
with others
in my field
An
opportunity
to work
with others
outside of
my field
An
opportunity
to monitor
the type of
research
being done
at OSU
Disagree
Neither
Agree
nor
Disagree
Agree
Strongly
Agree
I do not
remember
Total
Responses
Mean
1
2
13
30
53
2
101
4.39
1
4
22
43
27
2
99
4.00
5
16
32
28
15
2
98
3.41
1
2
10
37
49
2
101
4.38
3
4
26
36
28
3
100
3.94
1
12
28
44
14
2
101
3.65
0
0
5
53
42
1
101
4.40
1
5
21
45
27
2
101
3.99
2
2
20
44
32
1
101
4.05
1
13
34
31
19
2
100
3.62
Strongly
Disagree
156
19. To what extent do you disagree or agree that you expected the following during your
participation BEFORE participating in the WW2100 researcher-stakeholder engagement
process? (Check one for each item)
#
1
2
3
4
5
6
7
8
Question
Transparency
on the
project's
progress
Frequent
interaction
with
researcher
team
members
Frequent
interaction
with
stakeholders
Research
team
members to
make
attempts to
understand
my concerns
for the
project
Stakeholders
to make
attempts to
understand
my concerns
for the
project
Some of my
assumptions
to change as
the project
progressed
To use what
we learn to
improve the
model
To be kept
up to date as
the model
evolved
Disagree
Neither
Agree
nor
Disagree
Agree
Strongly
Agree
I do not
remember
Total
Responses
Mean
1
4
11
57
21
2
96
4.05
3
15
33
34
9
2
96
3.41
4
12
32
40
5
2
95
3.40
4
8
20
47
15
2
96
3.72
7
5
33
42
6
2
95
3.45
0
1
23
48
18
6
96
4.11
1
4
11
54
22
3
95
4.09
1
5
13
53
20
3
95
4.03
Strongly
Disagree
157
20. To what extent do you disagree or agree that you expected stakeholders to fulfill the
following roles for WW2100 BEFORE participating in the WW2100 researcherstakeholder engagement process? (Check one for each item)
#
1
2
3
4
5
6
7
8
9
10
Question
Guide
research
questions
Develop
pieces of the
model
Write
scenario
assumptions
Evaluate
scenario
assumptions
Interpret
model
outputs
Write reports
Provide a
"boots-onthe-ground"
perspective
Provide a
scientific
perspective
Communicate
with
stakeholders
Communicate
research
findings to
stakeholders
who are not
active
WW2100
participants
Disagree
Neither
Agree
nor
Disagree
Agree
Strongly
Agree
I do not
remember
Total
Responses
Mean
3
11
19
45
15
5
98
3.80
14
39
19
16
5
5
98
2.79
6
24
21
28
13
6
98
3.43
2
10
10
47
24
5
98
4.03
5
16
24
36
11
6
98
3.57
16
48
19
9
2
4
98
2.48
1
4
9
48
33
3
98
4.22
4
16
33
35
7
3
98
3.38
2
7
17
50
18
3
97
3.90
2
6
19
47
20
3
97
3.92
Strongly
Disagree
158
21. To what extent do you disagree or agree that you expected research team members to
fulfill the following roles for WW2100 BEFORE participating in the WW2100 researcherstakeholder engagement process? (Check one for each item)
#
1
2
3
4
5
6
7
8
9
10
Question
Guide
research
questions
Develop
pieces of the
model
Write
scenario
assumptions
Evaluate
scenario
assumptions
Interpret
model
outputs
Write reports
Provide a
"boots-onthe-ground"
perspective
Provide a
scientific
perspective
Communicate
with research
team
members
Communicate
research
findings to
stakeholders
who are not
active
WW2100
participants
Disagree
Neither
Agree
nor
Disagree
Agree
Strongly
Agree
I do not
remember
Total
Responses
Mean
0
1
4
33
53
2
93
4.57
0
2
2
22
65
2
93
4.70
0
2
5
33
50
3
93
4.54
0
1
3
33
52
3
92
4.61
0
0
2
26
63
2
93
4.72
0
2
4
24
59
2
91
4.63
5
20
31
28
6
3
93
3.24
0
0
5
26
60
2
93
4.66
0
0
5
31
54
3
93
4.62
0
1
13
41
36
2
93
4.29
Strongly
Disagree
159
22. To what extent do you disagree or agree that the following resulted from your
participation AFTER participating in WW2100? (Check one for each item)
#
1
2
3
4
5
6
7
8
9
10
Question
An
integrated
model of
water in
the
Willamette
Valley
Satisfy my
curiosity
Career
Experience
Model
results that
would
contribute
to science
Model
results that
I could use
in my job
An
opportunity
to share
what I
know
An
opportunity
to learn
An
opportunity
to work
with others
in my field
An
opportunity
to work
with others
outside of
my field
An
opportunity
to monitor
the type of
research
being done
at OSU
Disagree
Neither
Agree
nor
Disagree
Agree
Strongly
Agree
Total
Responses
Mean
3
4
26
50
8
91
3.62
2
5
28
46
7
88
3.58
3
10
40
32
4
89
3.27
2
4
23
54
8
91
3.68
3
12
33
37
5
90
3.32
5
5
17
52
11
90
3.66
2
3
9
57
19
90
3.98
2
6
18
51
12
89
3.73
2
7
17
49
15
90
3.76
2
11
29
37
10
89
3.47
Strongly
Disagree
160
23. To what extent do you disagree or agree that you experienced the following AFTER
participating in the WW2100 researcher-stakeholder engagement process? (Check one for
each item)
#
1
2
3
4
5
6
7
8
Question
Transparency
on the
project's
progress
Frequent
interaction
with research
team
members
Frequent
interactions
with
stakeholders
Research
team
members
made
attempts to
understand
my concerns
for the
project
Stakeholders
made
attempts to
understand
my concerns
for the
project
Some of my
assumptions
changed as
the project
progressed
Use of what
we learned to
improve the
model
Updates as
the model
evolved
Disagree
Neither
Agree
nor
Disagree
Agree
Strongly
Agree
Total
Responses
Mean
4
10
26
44
5
89
3.40
5
19
30
31
2
87
3.07
6
22
35
24
0
87
2.89
6
13
24
41
3
87
3.25
4
7
37
37
2
87
3.30
3
6
21
50
6
86
3.58
4
10
30
37
6
87
3.36
4
10
23
44
5
86
3.42
Strongly
Disagree
161
24. To what extent do you disagree or agree that stakeholders fulfilled the following roles
for WW2100 AFTER participating in the WW2100 researcher-stakeholder engagement
process? (Check one for each item)
#
1
2
3
4
5
6
7
8
9
10
Question
Guide
research
questions
Develop
model plugins
Write
scenario
assumptions
Evaluate
scenario
assumptions
Interpret
model
outputs
Write reports
Provide a
"boots-onthe-ground"
perspective
Provide a
scientific
perspective
Communicate
with research
team
members
Communicate
research
findings to
stakeholders
who are not
active
WW2100
participants
Disagree
Neither
Agree
nor
Disagree
Agree
Strongly
Agree
Total
Responses
Mean
0
8
31
42
3
84
3.48
5
22
40
16
1
84
2.83
0
19
34
29
2
84
3.17
0
8
32
40
4
84
3.48
0
15
44
24
1
84
3.13
8
34
35
7
0
84
2.49
0
6
23
45
11
85
3.72
2
14
42
23
3
84
3.13
1
5
35
40
4
85
3.48
0
9
39
34
3
85
3.36
Strongly
Disagree
162
25. To what extent do you disagree or agree that research team members fulfilled the
following roles for WW2100 AFTER participating in the WW2100 researcher-stakeholder
engagement process? (Check one for each item)
#
1
2
3
4
5
6
7
8
9
10
Question
Guide
research
questions
Develop
model plugins
Write
scenario
assumptions
Evaluate
scenario
assumptions
Interpret
model
outputs
Write reports
Provide a
"boots-onthe-ground"
perspective
Provide a
scientific
perspective
Communicate
with
stakeholders
Communicate
research
findings to
stakeholders
who are not
active
WW2100
participants
Disagree
Neither
Agree
nor
Disagree
Agree
Strongly
Agree
Total
Responses
Mean
0
1
12
54
16
83
4.02
0
2
18
41
23
84
4.01
0
1
15
55
13
84
3.95
1
1
16
49
16
83
3.94
0
2
14
45
23
84
4.06
1
3
21
41
18
84
3.86
6
25
37
14
2
84
2.77
0
2
14
41
26
83
4.10
2
5
25
41
9
82
3.61
5
9
38
26
6
84
3.23
Strongly
Disagree
163
26. Regarding your contribution to WW2100, to what extent do you disagree or agree with
the following? (Check one for each item)
#
1
2
3
4
5
Question
Research
team
members
respected
my opinions
during
researcherstakeholder
engagement
events
Stakeholders
respected
my opinions
during
researchstakeholder
engagement
events
Research
team
members
learned from
me
Stakeholders
learned from
me
My
knowledge
was
incorporated
in to the
Envision
modeling
tool
Disagree
Neither
Agree
nor
Disagree
Agree
Strongly
Agree
Total
Responses
Mean
1
3
20
53
13
90
3.82
1
1
24
56
9
91
3.78
4
8
35
39
6
92
3.38
2
6
45
35
4
92
3.36
4
18
44
24
2
92
3.02
Strongly
Disagree
164
27. Through participating in the WW2100 researcher-stakeholder engagement process, to
what extent do you disagree or agree with the following? (Check one for each item)
#
1
2
3
4
5
Question
I gained a
broader
view of
water in the
Willamette
Valley
I
understand
the
perspectives
of diverse
water users
in the
Willamette
Valley
I
understand
what the
Envision
model can
do
I
understand
the
Envision
model's
limitations
I
understand
the reasons
for the
Envision
model's
limitations
Disagree
Neither
Agree
nor
Disagree
Agree
Strongly
Agree
Total
Responses
Mean
2
2
13
51
23
91
4.00
1
3
15
57
15
91
3.90
1
11
25
47
7
91
3.53
2
13
31
40
5
91
3.36
1
17
29
38
5
90
3.32
Strongly
Disagree
165
28. To what extent do you disagree or agree that the WW2100 Envision model... (Check
one for each item)
#
Question
Contributes
to
scientific
knowledge
Informs
resource
managers
Informs
policy
makers
Informs
water users
Adequately
depicts
water use
and
scarcity in
the
Willamette
Valley
1
2
3
4
5
Disagree
Neither
Agree
nor
Disagree
Agree
Strongly
Agree
Total
Responses
Mean
2
2
18
57
12
91
3.82
2
4
33
47
5
91
3.54
2
4
31
48
5
90
3.56
3
4
32
47
5
91
3.52
2
9
43
35
2
91
3.29
Strongly
Disagree
29. How do you intend to use the project results? (Select all that apply)
#
1
2
3
4
5
7
8
Answer
Report the model scenario results to
my peers
Inform upcoming water use
decisions
Inform upcoming water regulatory
decisions
Base future research on the model
results
I do not intend to use the results
Unsure
Other (Please explain)
Response
%
50
52%
32
33%
22
23%
28
29%
8
33
10
8%
34%
10%
166
30. Regarding how you were personally impacted by participating in Willamette Water
2100, to what extent do you disagree or agree with the following? (Check one for each item)
#
Question
I learned
from
research
team
members
I learned
from
stakeholders
I formed or
strengthened
relationships
with
research
team
members
I formed or
strengthened
relationships
with
stakeholders
I shared in a
necessary
discussion
on water in
the
Willamette
Valley
1
2
3
4
5
Disagree
Neither
Agree
nor
Disagree
Agree
Strongly
Agree
Total
Responses
Mean
2
3
9
56
25
95
4.04
0
6
20
46
22
94
3.89
3
9
34
40
9
95
3.45
1
12
42
37
3
95
3.31
4
8
13
47
23
95
3.81
Strongly
Disagree
31. Would you participate in a researcher-stakeholder engagement process again? (Select
one. If you would like to explain your response, please do so in the text box provided)
#
1
0
7
Answer
Yes
No
Unsure
Total
Response
75
6
16
97
%
77%
6%
16%
100%
167
32. Would you help to fund (i.e. fund yourself or seek funding for) future projects similar
to the Willamette Water 2100, including the researcher-stakeholder engagement process?
(Select one; please remember that your answers are anonymous and will have no future
repercussions for you)
#
1
2
3
Answer
Yes
No
Unsure
Total
Response
39
23
35
97
33. If you have any final thoughts, please type them in the space below.
Text Response
%
40%
24%
36%
100%
168
Appendix C. Verbal Consent Guide
Be certain to mention the following details while verbally recruiting subjects:
Title of Study: Characterizing and Assessing the Willamette Water 2100 Science ResearchStakeholder Engagement Process
Name of Principle Investigator: Sam Chan
This is graduate student thesis research.
Contact information for interested individuals: Please contact Laura Ferguson at
fergusla@onid.orst.edu if you are interested.
Purpose of research: Identify who is participating, their reasons for participating, and voice their
opinions on how the collaborative research went. This is to inform future collaborative research
projects by identifying potential barriers and pathways to success.
Primary criteria to determine eligibility: We are looking for participants with greater experience
with WW2100 (one year or more).
Time commitment of subjects: 1-1.5 hours to be scheduled at your schedule and location
convenience.
169
Appendix D. Survey Letter of Invitation
Dear WW2100 LAN member:
At the last LAN meeting, we notified you about an upcoming survey regarding your participation
in and experiences with the Willamette Water 2100 project as part of the study “Characterizing
and Assessing the Willamette Water 2100 Science Research-Stakeholder Engagement Process.”
I am now writing to ask for your help in this study. The goals of this questionnaire are to better
understand the opinions of LAN participants regarding water use and distribution, priorities for
use, and concerns about future water availability as well as opinions regarding the experience
working in collaboration with the WW2100 research team/stakeholders, preferred
communication techniques, and individual goals for the project. The information will be used to
inform future collaboration efforts by identifying pathways and barriers to success.
We are contacting all LAN members who attended the most recent WW2100 workshop. We are
interested in the wide range of opinions that exists from the diverse population of Willamette
Water stakeholders involved in the project. There are no right or wrong answers. Your opinions
are very important to us and will make a difference for future collaborative processes.
This study is not designed to benefit you directly, and there are no risks. Your responses will be
kept private to the maximum extent allowable by law. The survey does not ask for any
identifying information. Your responses will be combined with others in a database that does not
contain identifying information and will be reported as part of a larger group. Your response to
this survey and any of the questions is completely voluntary and it should take about 30 minutes
to complete. You indicate your voluntary agreement to participate by completing and returning
this survey. Please complete this questionnaire at your earliest convenience. You may access the
survey by following this link: XX.
If you have any questions about this project now or after you access the survey, please feel free
to call me at (847) 732-2374. If you have questions or concerns regarding your rights as a study
participant, or are dissatisfied at any time with any aspect of this study, you may contact the
Oregon State University Institutional Review Board, by phone: (541) 737-8008, fax: (541) 7373093, or e-mail: irb@oregonstate.edu.
If is only with your generous help that our research can be successful. Thank you in advance for
your time and consideration.
Sincerely,
Laura Ferguson, Graduate Student Investigator
Sam Chan, Principal Investigator
170
Appendix E. Participating Research Team University Departments and Stakeholder
Organizations
Table E.1. Participating research team university departments and stakeholder organizations.
Research team university departments
Oregon State University (OSU) – OCCRI
OSU – Environmental Policy
OSU – Institute for Water and Watersheds
OSU – Oregon Sea Grant
OSU – Hydrogeology
OSU – Natural Resource Management
OSU – Geography
OSU – Applied Economics
OSU – Biological and Ecological Engineering
OSU – Forest Ecosystems and Society
OSU – Environmental Sciences
OSU – Water Resources
OSU – Marine Resource Management
OSU – Hydroclimatology
OSU – Fish and Wildlife
OSU – Hydrology
OSU – Hydrogeomorphology
OSU – Forest Engineering
OSU – Ecohydrology
OSU – Ecology
OSU – Institute for Natural Resources
OSU – Extension
OSU – STEPS
Portland State University (PSU) – Center for
Global Leadership in Sustainability
PSU – Dynamic Ecosystems and Landscapes
PSU – Hatfield School of Government
PSU – Hydrology
PSU – Geography
PSU – Climate science
University of Santa Barbara (UCSB) –
Ecohydrology
UCSB – Economics
University of Oregon (UO) – Landscape
Architecture
UO – School of Law
Stakeholder organizations
1000 Friends of Oregon
Agriculture Drainage Inc.
Association of Oregon Counties
Barney and Worth Business Consultants
Beaverton Schools
Benton County
Benton Soil and Water Conservation District
Bonneville Power Administration
City of Albany
City of Beaverton
City of Corvallis
City of Eugene
City of Hillsboro
City of Salem
City of Springfield
City of Tigard
Clackamas Water Providers
Clackamas County
Clean Water Services
Coast Fork Willamette Watershed Council
Coca-Cola Refreshments
Columbia County
Columbia River Inter-Tribal Fish Commission
Corvallis Public Schools
Deschutes County
Douglas County
Environmental Protection Agency
Eugene School District
Eugene Water and Electric Board
Farmer - private
Forest Fractal LLC
Freshwater Trust
Geosyntec
Greenberry Irrigation District
GSI Water Solutions, Inc.
Hillsboro School District
House Subcommittee on Water
Intel
Johnson Creek Watershed Council
Lacomb Irrigation and Hydro District
Lane County Community College
Lane County
171
Linn County
Long Tom Watershed Council
Marion County
Marys River Watershed Council
McKenzie ClearWater Coalition
McKenzie River Trust
Metro Councilor
Metro Wastewater Management Commission
Meyer Memorial Trust
Middle Fork Willamette Watershed Council
Multnomah County
Network of Oregon Watershed Councils
National Marine Fisheries Service (NMFS)
National Oceanic and Atmospheric Administration
(NOAA)
North Santiam Watershed Council
Northwest Power and Conservation Council
North Pacific Landscape Conservation Cooperative
Natural Resources Conservation Service
Oregon Department of Agriculture
Oregon Department of Environmental Quality
Oregon Department of Fish and Wildlife
City of Portland
Oregon state representatives
Oregon Association of Clean Water Agencies
Oregon Association of Nurseries
Oregon Cascades West Council of Governments
Oregon Farm Bureau
Oregon Governor’s Office
Oregon Water Resources Congress
Oregonians for Food and Shelter
Oregon Water and Electric Board
Oregon Water Resources Department
Polk County
Portland Public Schools
Portland Water Bureau
Pringle Creek Watershed Council of Salem
Pudding River Watershed Council
Santiam Water Control District
Sidney Irrigation Cooperative
South Lane School District
Nike Inc.
Tillamook County
Triangle Associates
Tualatin Valley Water District
Tualatin Water River Council
US Army Corps of Engineers
US Department of Agriculture
US Forest Service
US Fish and Wildlife Service
US Geological Survey
172
Vitality Farms
Washington County
WaterWatch of Oregon
Western States Water Council
Willamette Partnership
Willamette River Keepers
Willamette River Water Coalition
Writer – private
Yamhill County
173
Appendix F. Supplemental Survey Results Tables
Table F.1. Motivations of survey respondents. To what extent do you disagree or agree that you
are participating in WW2100 because…
It is relevant to my job
I want to know more about water
in the Willamette Valley
I am concerned for water in the
future
I want new tools to address water
issues
It is focused on the Willamette
Valley
It has the potential to shape the
future
I am representing a larger group
whose voice needs to be heard
Other
1
Mean1
4.26
4.09
Std. Dev
.78
.90
4.29
.86
4.10
.76
3.84
.99
4.01
.91
3.31
1.12
4.43
.85
Means on a five point scale 1 strongly disagree and 5 strongly agree.
Table F.2. Expectations for research team member and stakeholder roles.
Expectations for
Guide research questions
Develop pieces of the model
Write scenario assumptions
Evaluate scenario assumptions
Interpret model outputs
Write reports
Provide a “boots-on-the-ground”
perspective
Provide a scientific perspectives
Communicate with stakeholders
Communicate with stakeholders who are
not active WW2100 participants
1
Stakeholders
Research
Team
Members
Z – value
pvalue
3.64
2.54
3.22
3.92
3.36
2.26
4.51
4.64
4.44
4.52
4.66
4.57
5.29
7.44
5.75
4.40
6.81
7.78
<.001
<.001
<.001
<.001
<.001
<.001
Effect
Size
Cohen’s
d
1.03
2.30
1.25
.73
1.53
2.81
4.19
3.11
5.72
<.001
1.20
3.27
3.84
4.60
4.53
11.01
6.82
<.001
<.001
1.66
.91
3.84
4.23
3.37
.001
.48
Cell values are means of reported expectations on a 5-point scale from 1 “strongly disagree” to 5 “strongly agree.
Effect size is considered substantial at d > .80, typical at d > .50 and minimal at d > .20.
174
Table F.3. Expectations for stakeholder roles and whether they were met.
Guide research questions
Develop pieces of the model
Write scenario assumptions
Evaluate scenario assumptions
Interpret model outputs
Write reports
Provide a “boots-on-the-ground”
perspective
Provide a scientific perspectives
Communicate with stakeholders
Communicate with stakeholders who are
not active WW2100 participants
Expected
Met
Z-value
pvalue
3.60
2.55
3.17
3.88
3.32
2.27
3.48
3.83
3.17
3.48
3.13
3.48
1.01
2.43
.112
3.12
1.30
1.99
.312
.015
.911
.002
.195
.047
Effect
Size
Cohen’s
d
.14
.29
0.00
.46
.21
.25
4.15
3.73
4.18
<.001
.53
3.22
3.75
3.13
3.49
.88
1.95
.377
.051
.10
.31
3.80
3.36
3.31
.001
.53
1
Cell values are means of reported expectations on a 5-point scale from 1 “strongly disagree” to 5 “strongly agree.
Effect size is considered substantial at d > .80, typical at d > .50 and minimal at d > .20.
Table F.4. Expectations for research team member roles and whether they were met.
Guide research questions
Develop pieces of the model
Write scenario assumptions
Evaluate scenario assumptions
Interpret model outputs
Write reports
Provide a “boots-on-the-ground”
perspective
Provide a scientific perspectives
Communicate with stakeholders
Communicate with stakeholders who are
not active WW2100 participants
1
Expected
Met
Z-value
pvalue
4.51
4.65
4.43
4.51
4.67
4.58
4.06
4.05
4.00
3.99
4.10
3.91
4.47
5.29
4.07
4.34
5.46
5.55
<.001
<.001
<.001
<.001
<.001
<.001
Effect
Size
Cohen’s
d
.73
.86
.66
.77
.93
.89
3.10
2.78
3.43
.001
.33
4.59
4.52
4.14
3.51
4.60
5.99
<.001
<.001
.68
1.51
4.21
3.24
5.93
<.001
1.13
Cell values are means of reported expectations on a 5-point scale from 1 “strongly disagree” to 5 “strongly agree.
Effect size is considered substantial at d > .80, typical at d > .50 and minimal at d > .20.
175
Appendix G. Exploratory Factor Analysis of Researcher-Stakeholder Engagement Process
and Model Expectations
Table G.1. Exploratory factor analysis of researcher-stakeholder engagement process and model
expectations.
Factor 1/
Progress
To use what we learn to improve the
model
Transparency on the project’s progress
To be kept up to date as the model
evolved
Some of my assumptions to change as
the project progressed
I expected to gain career experience
An opportunity to work with others in
my field
An opportunity to share what I know
An opportunity to learn
An opportunity to work with others
outside of my field
Satisfy my curiosity
Frequent interaction with stakeholders
Stakeholders to make attempts to
understand my concerns for the project
Research team members to make
attempts to understand my concerns
for the project
Frequent interaction with research
team members
Model results that I could use in my
job
An integrated model of water in the
Willamette Valley
Model results that would contribute to
science
An opportunity to monitor the type of
research being done at OSU
Eigenvalue
Percent (%) of total variance
explained2
1
Factor Loadings1
Factor 2/
Factor 3/
Factor 4/
Factor
Opportunity Interaction Applicability 5/
Monitor
.80
.78
.79
.65
.79
.76
.71
.59
.57
.43
.49
.86
.82
.44
.70
.45
.70
.82
.43
.67
.51
.64
.92
3.26
3.17
3.07
2.06
18.12
17.60
17.03
11.03
1.33
7.36
Principal component factor analysis with Varimax rotation. Only factors with eigenvalues greater than 1 and items
with factor loadings greater than .40 were retained in the final factor structure (Tabachnick and Fidell 1996). Items
coded on a 5-point scale from 1 “Strongly disagree” to 5 “Strongly agree.”
2
Total cumulative percent (%) variance explained = 74.
176
Appendix H. Cronbach Reliability Analyses Index Analyses
Table H.1. Cronbach alpha reliability analyses for participation indices.
Participation Communicationa
Emailed a research team member
Emailed a stakeholder
Spoke to a research team member
Spoke to a stakeholder
Visited the WW2100 website
Participation overallb
Years of participation
Event participation
Seminar participation
Participation communication
1
Mean
(M)1
Std. dev.
(SD)1
Item Total
Correlation
Alpha (α)
if deleted
3.58
2.53
4.00
3.08
3.13
2.14
1.45
2.07
1.48
1.51
.74
.60
.73
.40
.71
.76
.80
.76
.85
.77
0.04
.10
.08
.12
.98
1.01
1.02
.97
.34
.44
.53
.44
.65
.59
.52
.59
Cronbach
alpha (α)
.83
.66
Means and standard deviations are measured on scales as indicated by the corresponding letter superscripts.
Means and standard deviations are measured on an 8-point frequency scale from 1 “Never” to 8 “Daily”. The
mean values center around 3 “2-3 Times a year”.
b
Means and standard deviations are standardized z scores.
a
177
Table H.2. Cronbach alpha reliability analyses for model utility, process utility, feeling heard,
and model understanding indices
Perceived model utility
The Envision model informs resource
managers
The Envision model informs policy makers
The Envision model informs water users
The Envision model adequately depicts
water use and scarcity in the Willamette
Valley
The Envision model contributes to
scientific knowledge
Gained a broader view of water in the
Willamette Valley
Perceived process utility
I understand the perspectives of diverse
water users in the Willamette Valley
I learned from research team members
I learned from stakeholders
I formed or strengthened relationships with
research team members
I formed or strengthened relationships with
stakeholders
I shared in a necessary discussion on water
in the Willamette Valley
Understanding of the Envision model
I understand the Envision model’s
limitations
I understand the reasons for the Envision
model’s limitations
I understand what the Envision model can
do
Participants felt heard
Research team members respected my
opinions during researcher-stakeholder
engagement events
Stakeholders respected my opinions during
researcher-stakeholder engagement events
Research team members learned from me
Stakeholders learned from me
My knowledge was incorporated into the
Envision modeling tool
1
Mean
(M)1
Std. dev.
(SD)1
Item Total
Correlation
Alpha (α)
if deleted
3.54
.74
.84
.88
3.56
3.52
.74
.79
.82
.77
.89
.89
3.29
.76
.72
.90
3.82
.77
.71
.90
3.99
.84
.65
.91
Cronbach
alpha (α)
.91
.88
3.90
.75
.68
.86
4.03
3.92
.84
.82
.75
.73
.85
.85
3.44
.91
.65
.87
3.31
.77
.62
.87
3.82
1.03
.74
.86
.90
3.37
.88
.86
.82
3.32
.89
.82
.86
3.52
.85
.76
.91
.82
3.82
.76
.61
.79
3.78
.68
.64
.79
3.36
3.34
.89
.77
.75
.62
.75
.79
3.00
.85
.49
.83
Means and standard deviations are measured on a 5-point scale from 1 “Strongly Disagree” to 5 “Strongly Agree.”