Collaborative Research Scoring Rubric for Projects with Indigenous Peoples Research and Presentation By: Dr. Ronald L. Trosper with Katherine Brooks Elizabeth Eklund Matthew Schwoebel Kestrel Smith Juanita Francis Lisa Palacios Peter Dubois American Indigenous Research Association Meeting Pablo, Montana October 11, 2014 Introduction: Why a rubric is useful • A rubric is useful for assessing the amount of collaboration and joint benefit that occurs in research projects with Indigenous peoples or communities. • The rubric has four general areas of criteria, and under each of the criterion, it has levels of achievement described in observable terms with assigned numerical values for achievement. The four areas are; 1) Community Centered Control-Joint Planning, 2) Joint Methodology Implementation, 3) Community Benefit-Outcomes, and 4) Researcher Benefit-Outcomes • The contribution of the rubric is to provide a helpful worksheet for the design of research projects, and to provide a potential set of standards for ranking different projects that are seeking funding. The rubric initially developed as a class project for a course on collaborative research and learning with Indigenous communities. Background on Collaborative Research • What has come to be known as collaborative research developed recently, between 1950 and 1970, after western scholars had been studying and working with Indigenous communities for more than a hundred years. • Collaborative research is defined as a process in which researchers, program developers, and community members are networked to do research for joint problem solving and positive social change (Van Willigen 2002: 101). • The goal of collaborative research is to blend theory into practice, heading away from conceptual frameworks only applicable within a university setting, moving toward the application of action within communities. Literature Review To frame the rubric and its ideas, an analysis of theoretical publications and examples of other attempts to create models or guidelines for effective collaborative research is necessary. Key pieces include: • Wulfhorst et al. (2008) “Core Criteria and Assessment of Participatory Research” • Fortmann (2008) “Participatory Research in Conservation and Rural Livelihoods: Doing Science Together” • Lambert (2013) “Spider Web Conceptual Model” • Arnstein (1969) “A Ladder Of Citizen Participation” Lambert’s Spider Web Conceptual Model (2013) • The majority of past research regarding Indigenous peoples has been conducted on them and not for them. • Lambert’s Spider Web Conceptual Model (2013) accounts for elements that are important to Indigenous peoples not only when conducting research, but central to their daily lives. Arnstein’s Ladder • Despite the early date of Arnstein’s 1969 article the concepts presented prove relevant and directly transferable to the discussion of research collaboration with Indigenous peoples. • Arnstein’s Ladder was deemed essential for Community-Centered Control of research, as well as projects involving Joint Planning, Joint Methodology Implementation, and Joint BenefitsOutcomes. • Arnstein’s concepts of the identification and organization of vital aspects of community participation provided key elements in the development of the collaborative research assessment rubric that the current study has created. Image from Arnstein 1969 Description of the Rubric The four categories of the rubric include: 1) Community Centered Control-Joint Planning, 2) Joint Methodology Implementation, 3) Community Benefit-Outcomes, and 4) Researcher Benefit-Outcomes • Presented under these four larger categories are a series of more specific criteria, with each item assisting in the assessment of the design, implementation, final product, and dissemination of collaborative research projects. • Each set of criterion is assigned a point value, on a scale of zero to five, with five representing the highest score based on the fulfillment of the criteria at an above average level, and zero serving as the lowest possible score, indicative of the absence of collaboration within a given criterion. • The rubric provides a five to zero point scale for each set of criteria within each category. Twenty-one total criteria are present within the rubric, allowing for a maximum score of 105 or lower. Category 1) Community-Centered control –Joint Planning Criteria Scoring: Evidence that the criteria have been met 1.A) Community working with or conducting its own research. __ (5) Community plays an active role in the study by working with their own community and conducting research. __ (3) Community plays a role in the study beyond research participant (serving as a researcher; interviewing, consulting, etc.) __ (0-1) Community plays no role beyond research/ study participant. __ (5) Community members are able to decide or change the 1.B) Community plays study outcome, goals, and priorities before the researcher visits. __ (3) Community members know about and agree with prominent role in determining outcomes researchers’ pre-determined study goals, outcomes, and priorities. and goals of research. __ (0-1) Community members find researchers’ study goals, outcomes, and priorities to be of minimal interest/ not important/ or not necessary. __ (5) A formal agreement with informed consent exists between 1.C) Community has the community and researcher provided free, prior __ (3) Some type of agreement exists between the community and and informed consent researcher (IRB; Tribal Permission; Interview Consent, etc.) for the project __ (0-1) No agreement exists between the community and researcher Category Criteria 1) Community-Centered control –Joint Planning Scoring: Evidence that the criteria have been met __ (5) Outside researchers are aware of and discuss the political and legal context with the community’s 1.D) Political and Legal representatives, and changes plans as required. context understood. __ (3) Outside researchers demonstrate awareness of the political and legal context. __ (0-1) The political and legal context is not discussed. 1.E) Breadth of Community Participation (Participation of marginalized people i.e., gender, age, income, class, etc.) __ (5) Community participation is broad and inclusive of all members. Dialogue with all parties continues throughout the research project __ (3) Community participation is diverse in some aspects (gender, age range, income level, etc.) __ (0-1) Community participation is limited to one category (gender only; age only; etc.) Total Points: _________ 2) Joint methodology implementation Category Criteria 2.A) Shared Control of Information or data gathered 2.B) Community members as researchers: Data gathering Scoring: Evidence that the criteria have been met ___(5) Control over the data/information is balanced. The researcher and community equally manage data and develop protocol for use, storage, processing, disposal, etc. ___(3) Community has some control over study related data/ information. Researcher follows community’s standard research protocol, but community input is not unique to the study. ___(0-1) No shared control over data/ information or protocol for management. ___(5) Community members help design data gathering templates, questionnaires, and other tools. Community members also participate in data acquisition. ___(3)Community members gather data following procedures that are determined by the external research team. ___(0-1) No community members are hired to gather data. Category Criteria 2) Joint methodology implementation 2.C) Community members as researchers: analysis 2.D) Communication Scoring: Evidence that the criteria have been met ___(5) Community members participate in discovering the significant results of the research, including any findings that are surprises. ___(3) Community members help interpret results. Researcher may compute statistical tests, but the results of the tests are discussed with community members in the process of determining what the data show. ___(0-1) Researchers do all of the analysis. ___(5) Communication between researcher and community continues regularly throughout duration of project. (Quarterly reports, meetings, town hall, etc). ___(3) Communication occurs only at major project milestones,: start, end, ___(0-1) ___(5) Researcher is confident in the abilities and knowledge of the 2.E) Mutual trust community. Community is comfortable with the researcher/or institution, who between respects the needs, views, and morals of the community. researcher and ___(3) Community and researcher/institution appear to have some degree of community mutual trust (measurable through previous collaboration; community has extensive relationship w/ institution, i.e. University students work on projects every summer, etc.) ___(0-1) Researcher is legally permitted to work with community, extent of mutual trust is unclear. Total _____ Category Criteria Scoring: Evidence that the criteria have been met Community Benefit 3) Outcomes-Community Benefit 3.A) Material Benefit 3.B) Control over publication __(5) Food security enhanced; land tenure increased; new economic opportunities developed __(3)Positive material benefit, but small scale (Grants, Return of cultural materials, etc.) __(0-1) No material benefit was attained by the community __(5) Full veto power; Co-authorship __(3) Complete review of drafts __(0-1) Informed of the publication; Received a copy after published __(5) Membership on examination committee; changed procedures by government agencies which move the community to higher levels of 3.C) Arnstein’s ladder Empowerment __(3) Community experienced financial, educational, etc. empowerment __(0-1) Study did not empower community __(5) The community creates a research office or museum which stores all data collected and is capable of negotiating new projects with new 3.D) Capacity researchers. Building __(3) Capacity for community members to work/hold authoritative positions __(0-1) Community capacity building was not encouraged/ No capacity building took place Category Criteria Scoring: Evidence that the criteria have been met Community Benefit 3) Outcomes-Community Benefit 3.E) Research questions answered __(5) Were the questions that the community developed answered; and there is one or more surprising finding from the community’s viewpoint. __(3) Findings were something previously unknown or surprising. __(0-1) Findings were not new or surprising; Community questions were answered. 3.F) Sustainability __(5) Outcomes of the study demonstrate long term effectiveness of outcomes through implementation of policies, programs, etc. __(3) Outcome offers assistance in attaining sustainability, but does not implement a solution __(0-1) Outcomes of the study are not sustainable __(5) Community houses recordings, images, records in their own 3.G) Control of facility; Full control in physicality and legal rights Information __(3) Legal rights to information shared during a project/ ability to and gathered manage access data __(0-1) No control beyond signing a consent form Total: ______ 4)Outcomes- Researcher benefits Category Criteria Scoring: Evidence that the criteria have been met Researcher Benefit 4)Outcomesbenefits 4.A) Publications Researcher 4.B) Degree Attainment __(5) Journal accessible to non-scholarly readers; is it widely distributed; or does the publication fall into gray literature such as technical reports, etc? __(3) Publication is scholarly and has academic audience primarily __(0-1) Non-publishable __(5) MA, MSc, or Ph.D. awarded to the researcher if a student. __(3) BA awarded __(0-1) Degree sought but not awarded based on shortcomings of study. Category Criteria Scoring: Evidence that the criteria have been met Researcher Benefit 4)OutcomesResearcher benefits __(5) Community and researcher successfully collaborate on study, move forward in working together again and again over an extended period of time. Relationship does 4.C) Long term not end upon finalization of current project. relationships __(3) Community and researcher maintain contact after study, possibility of future work but no solidified project known. __(0-1)Community and researcher end relationship upon finishing current project. __(5) Triple loop: fundamental, unrecognized 4.D) Number of loops assumptions are recognized, causing a revision of in the learning worldview. __(3) Double loop The structure of an existing model is changed; new equations used with the same set of variables. __(0-1) Single loop: parameters in an existing model are changed (improved) based on more data. Total Score: The Rubric Applied The Collaborative Research Scoring Rubric was applied to three articles. We have scored these pieces as both text based articles, and then a score for the project as a whole was assessed. • Lloyd KWSS-PSBLCB. 2012. Reframing development through collaboration: towards a relational ontology of connection in Bawaka, North East Arnhem Land. Third World Quarterly 33(6). • Sherwood J, and Kendall S. 2013. Reframing spaces by building relationships: Community collaborative participatory action research with Aboriginal mothers in prison. Contemporary Nurse 46(1):83-94. • Stoffle, Richard and Richard Arnold. 2003. Confronting the Angry Rock: American Indians’ Situated Risks from Radioactivity. Ethnos 68.2: 230-248. Scoring Comparison 30 25 23 20 21 20 22 25 24 20 17 16 16 15 14 11 10 5 0 Lloyd et al. 2012 Community Centered Control-Joint Planning (Max=25) Joint Methodology Implementation (Max=30) Community Benefit Outcomes (Max=35) Outcome Researcher Benefits (Max=20) Sherwood & Kendall 2013 Stoffle & Arnold (2003) Scoring Comparison – Articles vs. Projects The Rubric Applied 97 100 93 89 86 90 80 72 71 70 60 50 40 30 20 10 0 Lloyd et al. Project Lloyd et al. Article Sherwood & Kendall Project Sherwood& Kendall Article Stoffle & Arnold Project Stoffle& Arnold Article References Arnstein SR. 1969. A Ladder Of Citizen Participation. Journal of the American Institute of Planners 35(4):216-224. Barsh RL, and Trosper RL. 1975. Title I of the Indian Self-Determination and Education Assistance Act of 1975. American Indian Law Review 3(2):361395. Canby WC. 2009. American Indian law in a nutshell, fifth edition. St. Paul, MN: Thomson/West. Fortmann L. 2008. Participatory research in conservation and rural livelihoods : Doing science together. Hoboken, NJ: Wiley-Blackwell. Kishchuk N, Torrie J, Horn-Miller K, Légaré-Dionne C, and Gagnon C. 2008. Stepping Into A Flowing River: Evaluation of the Social Sciences and Humanities Research Council's Aboriginal Research Pilot Program. Final Evaluation Report. Ottawa, Ontario: Social Sciences and Humanities Research Council of Canada. Lambert L. 2013. Spider Conceptual Framework. Lloyd KWSS-PSBLCB. 2012. Reframing development through collaboration: towards a relational ontology of connection in Bawaka, North East Arnhem Land. Third World Quarterly 33(6). Meeds L. 1976. The Indian Policy Review Commission. lawcontprob Law and Contemporary Problems 40(1):9-11. National Congress of American Indians. 2014. American Indian and Alaska Native Genetics Resource Center. Havasupai Tribe and the Lawsuit Settlement Aftermath. Schensul and LeCompte. 1999. The Ethnographer’s Toolkit. 7 volumes. Book (190 - 250 pages each): Introduction to Ethnographic Research Methods (1); Essential Ethnographic Methods (2); Enhanced Ethnographic Methods (3); Researching Social Networks, Spatial Data and Hidden Populations (4) ; Analysis and Interpretation of Ethnographic Data (Book 5); Researcher’s Role and Research Partnerships (Book 6); Disseminating Ethnographic Data (7). Seven Oaks Innovation. CA: Altamira Press, Schensul, J. 1982. Issues in Applied Anthropology Training Programs. Practicing Anthropology, 4. Sherwood J, and Kendall S. 2013. Reframing spaces by building relationships: Community collaborative participatory action research with Aboriginal mothers in prison. Contemporary Nurse 46(1):83-94. Shirk JL, Ballard HL, Wilderman CC, Phillips T, Wiggins A, Jordan R, McCallie E, Minarchek M, Lewenstein BV, Krasny ME et al. . 2012. Public Participation in Scientific Research: A Framework for Deliberate Design. Ecology and Society 17(2):207-227. Sobeck J, Elizabeth C, and Charles F. 2003. Conducting research with American Indians: A case study of motives, methods, and results. Journal of Ethnic and Cultural Diversity in Social Work 12(1):69-84. Stoffle, Richard and Richard Arnold. 2003. Confronting the Angry Rock: American Indians’ Situated Risks from Radioactivity. Ethnos 68.2: 230-248. Van Willigen, John. 2002. Applied Anthropology: An Introduction, 3rd Edition. Westport, CT.: Bergin and Garvey. Wulfhorst J.D, with Eisenhauer, B.W, Gripne, S.L, and Ward, J.M. 2008. Core Criteria and Assessment of Participatory Research. In Wilmsen C, Elmendorf W, Fisher L, Ross J, Sarathy B, and Wells G, editors. Partnerships for empowerment : Participatory research for community-based natural resource management. London: Earthscan.