Utilization Focused Evaluation Case Studies: UFE in Multijurisdictional/Tribal Contexts Session 757 – Room 211 C/D – 8:00 AM to 9:30 AM Presenter: Nicole Bowman (Munsee/Mohican) President, Bowman Performance Consulting UFE’s Practical Application in Indian Country….. UFE: It’s not just about the flowchart….. Or the checklists….. …..or the Pretty Faces of those of us who use it….. BTW--ELDER? That is a good thing in my community… UFE is so much more! •Process •Relationships •Analogies •Stories •Responsive •Responsible •Meaningful •Useful •Complex •Nuanced •Integrated •And so on….. •Nii allagomahteet (all my relations; all things are related) •Wunj iin Daaptoonaakanum Niiloona (stories of our local history, usefulness, and relevance to community) •Wiciwinknewuk (four sacred directions, four sacred teachings, medicine wheel) •UFE has both traditional and contemporary connections Responsibility & Relations • UN Declaration of Indigenous Rights encourages States to comply with and effectively implement all their obligations as they apply to indigenous peoples under international instruments, in particular those related to human rights, in consultation and cooperation with the peoples concerned • Emphasizing that the United Nations has an important and continuing role to play in promoting and protecting the rights of indigenous peoples • How does the UN Declaration inform my work and the choices I use for evaluation design and related activities? • Remember the 4 R’s in Indian Country: Respect, Relationships, Reciprocity, Responsibility – UFE allows evaluators to incorporate all of these constructs UN Declaration of Indigenous Rights Facts: •Adopted by UN in 9/2007 •144 nations voted in favor •11 abstentions •4 nations voted against •Guess who they were? •All 4 nations eventually adopted the Declaration •Guess who was last? •Guess what year it was finally adopted by this nation? •Why does this matter??? Grounding Indigenous Evaluation The UN Declaration on the Rights of Indigenous Peoples: • Pulls aside scales of blindness that a culture of inequity has bred • Provides that its recitation of fundamental rights “constitute the minimum standards for the survival, dignity and well-being of the Indigenous Peoples of the world.” • The Declaration’s preamble echoes the Universal Declaration of Human Rights in proclaiming it “as a standard of achievement to be pursued in a spirit of partnership and respect.” • Obliges all governments to examine how they will come to evolve systems that recognize and embrace the “minimum standards” set forth in the Declaration • 46 articles with subsections are in UN Declaration • Articles 31 & 32 are the foundation for Indigenous knowledge & intellectual property rights protections •Robinson, 2010. “Minimum Standards: The UN Declaration on the Rights of Indigenous People” •UN Nations Permanent Forum on Indigenous Issues Website: http://social.un.org/index/IndigenousPeoples/DeclarationontheRightsofIndigenousPeoples.aspx UN Indigenous Declaration: Selections from Article 31 •….Indigenous peoples have the right to maintain, control, protect and develop their cultural heritage, traditional knowledge and traditional cultural expressions, as well as the manifestations of their sciences, technologies and cultures, including human and genetic resources, seeds, medicines, knowledge of the properties of fauna and flora, oral traditions, literatures, designs, sports and traditional games and visual and performing arts. •…..They also have the right to maintain, control, protect and develop their intellectual property over such cultural heritage, traditional knowledge, and traditional cultural expressions. UN Indigenous Declaration: Selections from Article 32 •…States shall consult and cooperate in good faith with the indigenous peoples concerned through their own representative institutions in order to obtain their free and informed consent prior to the approval of any project affecting their lands or territories and other resources, particularly in connection with the development, utilization or exploitation of mineral, water or other resources. •…States shall provide effective mechanisms for just and fair redress for any such activities, and appropriate measures shall be taken to mitigate adverse environmental, economic, social, cultural or spiritual impact. •UFE has been an effective mechanism in Indian Country…. But remember meanings, perceptions and feelings that came before & extend beyond the UN policy document The Case • 5 year federally funded project at over $5,000,000 • Partnership between a state university & Tribal college • Education & STEM workforce development focus • Major grant activities/purposes: • Professional development for K-12 STEM teachers through summer academies, academic year trainings, and research internships with university faculty • K-12 curriculum development in eco-diversity, bio-energy, and sustainability aligned to cultural, contextual, and workforce needs • Curricular and professional development incorporation of private sector, traditional community, and public/tribal schools • Student activities for STEM career interest, engaging/relevant STEM curriculum and school experience, and after school clubs or internships in STEM • Synergistic and collaborative K-16+ activities designed to strengthen systems supporting STEM education UFE Steps 1-4 in Indigenous Context UFE STEP 1. Assess/build organizational readiness Applied to Indigenous Case IHE IRB; Tribal IRB; funding for evaluation available; history of IHE organizational experience with Tribes/Tribal college and vice-versa 2. Assess/enhance Strong practical knowledge, cultural/interpersonal evaluator readiness/ competence, reflective practice, and project management; competence less skilled on “rigor” (AKA generalizability/use of nationally normed instruments) as IHE wanted it but understood that this wouldn’t be appropriate or realistic given the n size and context; challenges and adaptations are continual 3. Identify, organize, & engage primary users Initial evaluation development team to co-construct evaluation comprised of evaluators, IHE, and Tribal; design and instruments are member checked; data is shared as a process for building capacity, documenting process, and impacts 4. Situational analysis with primary users Context and relations; staffing changes; sample size; relation building; participation in program; etc. all impacted evaluation; implementation was impacted too which delayed evaluation UFE Steps 5-8 in Indigenous Context UFE STEP Applied to Indigenous Case 5. Identify/prioritize primary intended users by determining primary purposes Evaluation purposes were established based on major grant activities/treatment; other purposes developed based on local needs and use; presentations & work groups on evaluation capacity/purpose completed; developmental, knowledge generation, and accountability were primary uses for data 6. Consider/build in process uses if/as appropriate Evaluation updates and work groups built into major activity agenda’s, leadership team, and community team sessions; instruments and methods continue to be discussed and modifications are made as needed 7. Focus primary evaluation questions 5 grant goals with objectives are noted however “best” data was with the cultural / academic “process” for building curriculum and with teacher PD impacts in classrooms 8. Check that fundamental evaluation inquiry areas are addressed Outcomes evaluation helps to answer to what extent goals were met; process evaluation helps to answer the challenges and supports for implementation; and new program/treatment = developmental use for evaluation UFE Steps 9-12 in Indigenous Context UFE STEP 9. Determine what intervention model/theory of change is being evaluated Applied to Indigenous Case PD process for curriculum development and training institutes utilizing a socio-cultural process with elders, academics, teachers, community members, and students = relevance in STEM curriculum and stronger impacts in classroom and STEM career pathways 10. Negotiate Methods and instrument choices challenging in year 1; appropriate methods to appropriateness for culture/context/N size challenged; generate credible findings modifications made in year 2 and 3; ongoing process 11. Make sure intended users understand controversies and implications Mix method data will be produced; frequent visits back to the evaluation plan and questions with PI is warranted; keeping costs down and methods/instruments changes less frequent is helpful; R vs. E is always an issue 12. Stimulate use of findings Abstract discussions need concrete frameworks; expectations needed to be better aligned; time up front for data analysis needed to be increased; use/value of data by Indigenous users stronger than IHE users UFE Steps 13-16 in Indigenous Context UFE STEP Applied to Indigenous Case 13. Gather data with ongoing attention and use Management of data collection and reporting needed to be clarified; quarterly PI meetings set up; creating templates and expectations for reporting at 2-3 events annually was developed; continual reminders and requests for data/sticking to the plan are needed 14. Organize/ present data for interpretation/use by primary users Re-connecting in reports/presentations (use of visual organizers, data maps, and summary eval tables used); audience is key to reports so academic vs. summary vs. community reports or data chats are utilized; requests for feedback on emerging data, initial findings, and reports is always welcomed in a non-defensive and positive manner 15. Prepare reports to facilitate use, disseminate findings, and expand influence Presentations/executive summaries for leadership team and curriculum team; talking circles for community and elders; formal evaluation report to funder and PI annually; alternative explanations needed for N & culture/context 16. Follow up with primary users to enhance use Findings and recommendations discussed; annual self-eval and monitoring of recommendation use, implementation, and completion; eval/project/budget modifications made UFE Step 17: Meta-evaluation UFE Step 17: Meta-evaluation of Useof Use •Accountability •Learning •Improvements •Accountability in many forms: • Participants • Community • PI Team • Partners • Funding Agency • Evaluation Agency (BPC) for project, time, budget management and cultural, contextual, professional standards • Next 7 generations……. UFE Step 17: Meta-evaluation of Use, con’t. • Traditional vs. contemporary uses and impacts for Indigenous participants—delicate balance •Reflective practice for what the evaluation did well and what didn’t go as planned • Analyze what evaluation has control over and negotiate the rest with others • Set evaluation priorities and recommendations for coming year • Monthly monitoring (self) of improvements and quarterly with PI team Indigenous evaluation requires walking and working between/across the worlds (multiple & dynamic contexts) Community and/or Intended Participants *Evaluation Provider And/or Project Staff Funding Source UFE is a framework, process, and tool that helps us make this a more honorable walk… Use Strengths / Asset Based Approaches Much diversity exists within Indigenous communities….. Contextual Family/ Individual Indigenous Diversity Socio-economic Cultural/ Linguistic Concluding Thoughts…. • Create a shared space that is negotiated and respectful • Linda’s story of “the scientists felt vulnerable because their knowledge wasn’t being acknowledged…” • UFE says cultural competence is interwoven with our work; everyone brings culture to the project so address it • Make evaluation work responsive to local needs and useful for the next 7 generations • UFE’s design and use of meta-evaluation in Indigenous contexts helps create evaluation work that is helpful in the present as well as builds capacity for the future • Nii allogomahteet and laapi uchkneewal • Relationships are built and sustained with trust, recognition/value of each other, and authenticity • Leave good “moccasin tracks” during and after the project ADD TEXT HERE I hope my Indigenous perspectives on UFE support your evaluation journey… Anushiik & Laapi uch Keneewal! Future Contact: Nicole Bowman (Mohican/Munsee) E-Mail: nicky@bpcwi.com Phone: 715-526-9240 URL: www.bpcwi.com www.bpcwi.com