Collaborative Research Scoring Rubric
for Projects with Indigenous Peoples
Research and Presentation By:
Dr. Ronald L. Trosper with
Katherine Brooks
Elizabeth Eklund
Matthew Schwoebel
Kestrel Smith
Juanita Francis
Lisa Palacios
Peter Dubois
American Indigenous Research
Association Meeting
Pablo, Montana
October 11, 2014
Introduction: Why a rubric is useful
• A rubric is useful for assessing the amount of collaboration and joint benefit that
occurs in research projects with Indigenous peoples or communities.
• The rubric has four general areas of criteria, and under each of the criterion, it has
levels of achievement described in observable terms with assigned numerical
values for achievement. The four areas are;
1) Community Centered Control-Joint Planning,
2) Joint Methodology Implementation,
3) Community Benefit-Outcomes, and
4) Researcher Benefit-Outcomes
• The contribution of the rubric is to provide a helpful worksheet for the design of
research projects, and to provide a potential set of standards for ranking different
projects that are seeking funding. The rubric initially developed as a class project for
a course on collaborative research and learning with Indigenous communities.
Background on Collaborative Research
• What has come to be known as collaborative research developed recently,
between 1950 and 1970, after western scholars had been studying and working
with Indigenous communities for more than a hundred years.
• Collaborative research is defined as a process in which researchers, program
developers, and community members are networked to do research for joint
problem solving and positive social change (Van Willigen 2002: 101).
• The goal of collaborative research is to blend theory into practice, heading away
from conceptual frameworks only applicable within a university setting, moving
toward the application of action within communities.
Literature Review
To frame the rubric and its ideas, an analysis of theoretical
publications and examples of other attempts to create
models or guidelines for effective collaborative research is
necessary. Key pieces include:
• Wulfhorst et al. (2008) “Core Criteria and Assessment of
Participatory Research”
• Fortmann (2008) “Participatory Research in Conservation
and Rural Livelihoods: Doing Science Together”
• Lambert (2013) “Spider Web Conceptual Model”
• Arnstein (1969) “A Ladder Of Citizen Participation”
Lambert’s Spider Web Conceptual Model (2013)
• The majority of past
research regarding
Indigenous peoples
has been conducted
on them and not for
them.
• Lambert’s Spider
Web Conceptual
Model (2013)
accounts for
elements that are
important to
Indigenous peoples
not only when
conducting research,
but central to their
daily lives.
Arnstein’s Ladder
• Despite the early date of Arnstein’s 1969
article the concepts presented prove
relevant and directly transferable to the
discussion of research collaboration with
Indigenous peoples.
• Arnstein’s Ladder was deemed essential
for Community-Centered Control of
research, as well as projects involving
Joint Planning, Joint Methodology
Implementation, and Joint BenefitsOutcomes.
• Arnstein’s concepts of the identification
and organization of vital aspects of
community participation provided key
elements in the development of the
collaborative research assessment rubric
that the current study has created.
Image from Arnstein 1969
Description of the Rubric
The four categories of the rubric include:
1) Community Centered Control-Joint Planning,
2) Joint Methodology Implementation,
3) Community Benefit-Outcomes, and
4) Researcher Benefit-Outcomes
•
Presented under these four larger categories are a series of more specific criteria,
with each item assisting in the assessment of the design, implementation, final
product, and dissemination of collaborative research projects.
•
Each set of criterion is assigned a point value, on a scale of zero to five, with five
representing the highest score based on the fulfillment of the criteria at an above
average level, and zero serving as the lowest possible score, indicative of the absence
of collaboration within a given criterion.
•
The rubric provides a five to zero point scale for each set of criteria within each
category. Twenty-one total criteria are present within the rubric, allowing for a
maximum score of 105 or lower.
Category
1) Community-Centered control –Joint Planning
Criteria
Scoring: Evidence that the criteria have been met
1.A) Community
working with or
conducting its own
research.
__ (5) Community plays an active role in the study by working with
their own community and conducting research.
__ (3) Community plays a role in the study beyond research
participant (serving as a researcher; interviewing, consulting, etc.)
__ (0-1) Community plays no role beyond research/ study
participant.
__ (5) Community members are able to decide or change the
1.B) Community plays study outcome, goals, and priorities before the researcher visits.
__ (3) Community members know about and agree with
prominent role in
determining outcomes researchers’ pre-determined study goals, outcomes, and priorities.
and goals of research. __ (0-1) Community members find researchers’ study goals,
outcomes, and priorities to be of minimal interest/ not important/
or not necessary.
__ (5) A formal agreement with informed consent exists between
1.C) Community has
the community and researcher
provided free, prior
__ (3) Some type of agreement exists between the community and
and informed consent researcher (IRB; Tribal Permission; Interview Consent, etc.)
for the project
__ (0-1) No agreement exists between the community and
researcher
Category Criteria
1) Community-Centered control –Joint
Planning
Scoring: Evidence that the criteria have been met
__ (5) Outside researchers are aware of and discuss the
political and legal context with the community’s
1.D) Political and Legal representatives, and changes plans as required.
context understood.
__ (3) Outside researchers demonstrate awareness of the
political and legal context.
__ (0-1) The political and legal context is not discussed.
1.E) Breadth of
Community
Participation
(Participation of
marginalized people
i.e., gender, age,
income, class, etc.)
__ (5) Community participation is broad and inclusive of
all members. Dialogue with all parties continues
throughout the research project
__ (3) Community participation is diverse in some aspects
(gender, age range, income level, etc.)
__ (0-1) Community participation is limited to one
category (gender only; age only; etc.)
Total Points: _________
2) Joint methodology implementation
Category Criteria
2.A) Shared
Control of
Information or
data gathered
2.B) Community
members as
researchers:
Data gathering
Scoring: Evidence that the criteria have been met
___(5) Control over the data/information is balanced. The
researcher and community equally manage data and develop
protocol for use, storage, processing, disposal, etc.
___(3) Community has some control over study related data/
information. Researcher follows community’s standard research
protocol, but community input is not unique to the study.
___(0-1) No shared control over data/ information or protocol
for management.
___(5) Community members help design data gathering
templates, questionnaires, and other tools. Community
members also participate in data acquisition.
___(3)Community members gather data following procedures
that are determined by the external research team.
___(0-1) No community members are hired to gather data.
Category Criteria
2) Joint methodology implementation
2.C) Community
members as
researchers:
analysis
2.D)
Communication
Scoring: Evidence that the criteria have been met
___(5) Community members participate in discovering the significant results of
the research, including any findings that are surprises.
___(3) Community members help interpret results. Researcher may compute
statistical tests, but the results of the tests are discussed with community
members in the process of determining what the data show.
___(0-1) Researchers do all of the analysis.
___(5) Communication between researcher and community continues
regularly throughout duration of project. (Quarterly reports, meetings, town
hall, etc).
___(3) Communication occurs only at major project milestones,: start, end,
___(0-1)
___(5) Researcher is confident in the abilities and knowledge of the
2.E) Mutual trust community. Community is comfortable with the researcher/or institution, who
between
respects the needs, views, and morals of the community.
researcher and ___(3) Community and researcher/institution appear to have some degree of
community
mutual trust (measurable through previous collaboration; community has
extensive relationship w/ institution, i.e. University students work on projects
every summer, etc.)
___(0-1) Researcher is legally permitted to work with community, extent of
mutual trust is unclear.
Total _____
Category Criteria
Scoring: Evidence that the criteria have been met
Community Benefit
3) Outcomes-Community Benefit
3.A) Material
Benefit
3.B) Control
over
publication
__(5) Food security enhanced; land tenure increased; new economic
opportunities developed
__(3)Positive material benefit, but small scale (Grants, Return of cultural
materials, etc.)
__(0-1) No material benefit was attained by the community
__(5) Full veto power; Co-authorship
__(3) Complete review of drafts
__(0-1) Informed of the publication; Received a copy after published
__(5) Membership on examination committee; changed procedures by
government agencies which move the community to higher levels of
3.C)
Arnstein’s ladder
Empowerment __(3) Community experienced financial, educational, etc. empowerment
__(0-1) Study did not empower community
__(5) The community creates a research office or museum which stores all
data collected and is capable of negotiating new projects with new
3.D) Capacity researchers.
Building
__(3) Capacity for community members to work/hold authoritative
positions
__(0-1) Community capacity building was not encouraged/ No capacity
building took place
Category
Criteria
Scoring: Evidence that the criteria have been met
Community Benefit
3) Outcomes-Community Benefit
3.E) Research
questions
answered
__(5) Were the questions that the community developed answered;
and there is one or more surprising finding from the community’s
viewpoint.
__(3) Findings were something previously unknown or surprising.
__(0-1) Findings were not new or surprising; Community questions
were answered.
3.F) Sustainability __(5) Outcomes of the study demonstrate long term effectiveness
of outcomes
through implementation of policies, programs, etc.
__(3) Outcome offers assistance in attaining sustainability, but does
not implement a solution
__(0-1) Outcomes of the study are not sustainable
__(5) Community houses recordings, images, records in their own
3.G) Control of
facility; Full control in physicality and legal rights
Information
__(3) Legal rights to information shared during a project/ ability to
and gathered
manage access
data
__(0-1) No control beyond signing a consent form
Total: ______
4)Outcomes- Researcher benefits
Category
Criteria
Scoring:
Evidence that the criteria have been met
Researcher Benefit
4)Outcomesbenefits
4.A) Publications
Researcher
4.B) Degree
Attainment
__(5) Journal accessible to non-scholarly readers; is it
widely distributed; or does the publication fall into gray
literature such as technical reports, etc?
__(3) Publication is scholarly and has academic audience
primarily
__(0-1) Non-publishable
__(5) MA, MSc, or Ph.D. awarded to the researcher if a
student.
__(3) BA awarded
__(0-1) Degree sought but not awarded based on
shortcomings of study.
Category
Criteria
Scoring:
Evidence that the criteria have been met
Researcher Benefit
4)OutcomesResearcher benefits
__(5) Community and researcher successfully collaborate
on study, move forward in working together again and
again over an extended period of time. Relationship does
4.C) Long term
not end upon finalization of current project.
relationships
__(3) Community and researcher maintain contact after
study, possibility of future work but no solidified project
known.
__(0-1)Community and researcher end relationship upon
finishing current project.
__(5) Triple loop: fundamental, unrecognized
4.D) Number of loops assumptions are recognized, causing a revision of
in the learning
worldview.
__(3) Double loop The structure of an existing model is
changed; new equations used with the same set of
variables.
__(0-1) Single loop: parameters in an existing model are
changed (improved) based on more data.
Total Score:
The Rubric Applied
The Collaborative Research Scoring Rubric was applied to three articles. We
have scored these pieces as both text based articles, and then a score for the
project as a whole was assessed.
• Lloyd KWSS-PSBLCB. 2012. Reframing development through collaboration:
towards a relational ontology of connection in Bawaka, North East Arnhem
Land. Third World Quarterly 33(6).
• Sherwood J, and Kendall S. 2013. Reframing spaces by building
relationships: Community collaborative participatory action research with
Aboriginal mothers in prison. Contemporary Nurse 46(1):83-94.
• Stoffle, Richard and Richard Arnold. 2003. Confronting the Angry Rock:
American Indians’ Situated Risks from Radioactivity. Ethnos 68.2: 230-248.
Scoring Comparison
30
25
23
20 21
20
22
25
24
20
17
16
16
15
14
11
10
5
0
Lloyd et al. 2012
Community
Centered
Control-Joint
Planning
(Max=25)
Joint
Methodology
Implementation
(Max=30)
Community
Benefit
Outcomes
(Max=35)
Outcome
Researcher
Benefits
(Max=20)
Sherwood & Kendall
2013
Stoffle & Arnold (2003)
Scoring Comparison – Articles vs.
Projects
The Rubric Applied
97
100
93
89
86
90
80
72
71
70
60
50
40
30
20
10
0
Lloyd et al. Project
Lloyd et al. Article
Sherwood &
Kendall Project
Sherwood&
Kendall Article
Stoffle & Arnold
Project
Stoffle& Arnold
Article
References
Arnstein SR. 1969. A Ladder Of Citizen Participation. Journal of the American Institute of Planners 35(4):216-224.
Barsh RL, and Trosper RL. 1975. Title I of the Indian Self-Determination and Education Assistance Act of 1975. American Indian Law Review 3(2):361395.
Canby WC. 2009. American Indian law in a nutshell, fifth edition. St. Paul, MN: Thomson/West.
Fortmann L. 2008. Participatory research in conservation and rural livelihoods : Doing science together. Hoboken, NJ: Wiley-Blackwell.
Kishchuk N, Torrie J, Horn-Miller K, Légaré-Dionne C, and Gagnon C. 2008. Stepping Into A Flowing River: Evaluation of the Social Sciences and
Humanities Research Council's Aboriginal Research Pilot Program. Final Evaluation Report. Ottawa, Ontario: Social Sciences and Humanities Research
Council of Canada.
Lambert L. 2013. Spider Conceptual Framework.
Lloyd KWSS-PSBLCB. 2012. Reframing development through collaboration: towards a relational ontology of connection in Bawaka, North East Arnhem
Land. Third World Quarterly 33(6).
Meeds L. 1976. The Indian Policy Review Commission. lawcontprob Law and Contemporary Problems 40(1):9-11.
National Congress of American Indians. 2014. American Indian and Alaska Native Genetics Resource Center. Havasupai Tribe and the Lawsuit
Settlement Aftermath.
Schensul and LeCompte. 1999. The Ethnographer’s Toolkit. 7 volumes. Book (190 - 250 pages each): Introduction to Ethnographic Research Methods
(1); Essential Ethnographic Methods (2); Enhanced Ethnographic Methods (3); Researching Social Networks, Spatial Data and Hidden Populations (4) ;
Analysis and Interpretation of Ethnographic Data (Book 5); Researcher’s Role and Research Partnerships (Book 6); Disseminating Ethnographic Data
(7). Seven Oaks Innovation. CA: Altamira Press,
Schensul, J. 1982. Issues in Applied Anthropology Training Programs. Practicing Anthropology, 4.
Sherwood J, and Kendall S. 2013. Reframing spaces by building relationships: Community collaborative participatory action research with Aboriginal
mothers in prison. Contemporary Nurse 46(1):83-94.
Shirk JL, Ballard HL, Wilderman CC, Phillips T, Wiggins A, Jordan R, McCallie E, Minarchek M, Lewenstein BV, Krasny ME et al. . 2012. Public
Participation in Scientific Research: A Framework for Deliberate Design. Ecology and Society 17(2):207-227.
Sobeck J, Elizabeth C, and Charles F. 2003. Conducting research with American Indians: A case study of motives, methods, and results. Journal of
Ethnic and Cultural Diversity in Social Work 12(1):69-84.
Stoffle, Richard and Richard Arnold. 2003. Confronting the Angry Rock: American
Indians’ Situated Risks from Radioactivity. Ethnos 68.2: 230-248.
Van Willigen, John. 2002. Applied Anthropology: An Introduction, 3rd Edition. Westport, CT.: Bergin and Garvey.
Wulfhorst J.D, with Eisenhauer, B.W, Gripne, S.L, and Ward, J.M. 2008. Core Criteria and Assessment of Participatory Research. In Wilmsen C,
Elmendorf W, Fisher L, Ross J, Sarathy B, and Wells G, editors. Partnerships for empowerment : Participatory research for community-based natural
resource management. London: Earthscan.