Partners in Evaluation
Co-constructing
educational evaluation
at the local level
Ruth Pritchard and Carol McSporran
Presentation to ANZEA 2013
Nāu te rourou,
nāku te rourou,
ka ora te iwi.
With your food basket
and my food basket,
people will thrive.
‘Nga Kite Whai Matauranga’ by Mihi Williams and Roimata Emery
The Notion of
Partnership
Te Tiriti o Waitangi
• - Partnership
• - Protection
• - Participation
embedded in evaluation
principles, standards
and competencies
• Serious engagement with difference
engagement with diversity and difference in evaluation is both
substantive and a moral commitment (Hood, Hopson & Frierson,
2005, p.9)
• Values-engaged approach
evaluators must assume responsibility for explicating and
justifying the values being advanced in their work, in ways that
respect other values (Hall, Ahn & Greene, 2012, p. 206)
Tukutuku: partnership and participation
From national library and te papa archive collections
Te Whare Runanga, Waitangi House
Co-constructing the big picture
ERO’s complementary evaluation
Weaving on a 2-dimensional framework
• Ko te tamaiti te putake o te kaupapa
The child is the heart of the matter
• Evaluation – complementary review
Co-construction
Lincoln and Guba (1985); Stake (1975; 1983)
Morris (2002)
• Multiple perspectives build towards ‘a commonly accepted reality’
• “Stakeholder involvement using the principles of social constructivism should
lead to a more valid evaluation that empowers stakeholders and increases
the likelihood of the utilization of results as stakeholder investment
is greater than would otherwise be the case” (p. 51)
Participatory evaluation as partnership
(e.g. Patton, 2008, p. 224; Cousins & Earl, 1992; Cousins & Whitmore, 1998;
Greene, 1988; Ayers, 1987).
• Complex dynamic operating
at three levels: individual,
group, organisational
(Smits & Champagne, 2008)
• Comprising five interactive
processes
(Weaver & Cousins, 2004)
Cousins & Whitmore, 1998;
Cousins & Chouinard, 2012
“this shifting responsibility is a
prerequisite for the success of an
evaluation”
Themessl-Huber & Grutsch, 2003, p. 98.
Complementary evaluation
(Mutch, 2012; Nevo, 1994, 2002)
• Appreciative Inquiry approach (Cooperrider, 1982)
• Deeper participation, fewer/primary users
(Taut & Alkin, 2010)
• Transparent processes and values
• Complementary relationships
Hopkins & West (2002)
ERO’s approach
Enacting partnership at the local level
• contextually responsive design
• negotiated values
• dialogic basis for engagement
• space for shared sense-making
ERO (2013)
Opportunities for weaving threads
• Reciprocity of learning
bridge the gap between theory and practice…and create opportunities for mutual learning” (MacLellanWright. et al., 2007, p. 102).
• Enhancing validity
the experiences and expertise of stake-holders enlarge and enrich the scope of the evaluation (Abma, 2000)
acknowledged as ‘natural experts in their field’ (Themessl-Huber & Grutsch, 2003, p. 94).
• Evaluation capability building
(Compton, Baizerman & Stockdill, 2002)
• Utilization
process use (Greene, 1988; Patton, 1998)
contributes to a learning phase that reinforces understandings and the ownership of results and, eventually,
a greater sense of obligation to follow through on the results (Smits & Champagne, 2008, p. 429).
Position of the evaluator
putting ourselves into practice (Collins 1994)
it is as much about who we are, and where we position ourselves in
relation to others, as it is about what we do (ANZEA, 2011, p. 10)
• a shift from the primary investigator and participant observer to
becoming a facilitator (Garaway, 1995)
• a “high tolerance for ambiguity” (Patton, 2011, p. 26)
• cultural responsiveness with “careful attention to one’s place and
perspective” (Letiecq & Bailey, 2004, p. 32)
Repositioning evaluators to:
• use diverse methods of inquiry
• suspend judgement and sense-making
• create time and space for participants to:
- articulate
- explore
- co-construct knowledge
- develop their evaluative capacity
Challenges
• Complex contexts and competing agendas
- growth in reliance on evaluation
- diversity in the sector
- political implications
• Competence and understanding of
practice/theory
- moving from apprenticeship to a professional
model
- variability
• Evolving, dynamic process and
expectations
- high expectations for creativity, innovation and
cultural responsiveness
- we can be deliberate and responsive in the evaluation
approaches we use and they can co-exist :
participatory, empowerment, appreciative, transformative,
collaborative, deliberately democratic, developmental,
responsive, and utilization focused
Enabling partnership - what seems to
work?
• Communication and information sharing
• Inviting participation at different levels
• Shared team and context understandings
• Professional Practice Model (PPM) initiation
Enabling partnership - what seems to
work? (contd.)
• Review Officers’ understanding of expected practices and
principles for complementary practices:
- clarity of roles and responsibilities
- agreed evaluation framework
- recognition of the contribution each party
makes
- respect for different perspectives (Mutch, 2012)
• Synergy between internal and external evaluation – making
connections at the local level
Pūawaitanga o te Ringa - Fruits of our busy hands, 2001
Christchurch Central Library
References
Abma, T. A. (2000) .‘Stakeholder Conflict: A Case Study’, Evaluation and Program Planning, 23(2): 199–210.
ANZEA (2011). Evaluator Competencies 2011 Retrieved from
www.anzea.org.nz/.../05/110801_anzea_evaluator_competencies_final.pdf ·
Ayers, T. D. (1987). Stakeholders as partners in evaluation: A stakeholder-collaborative approach. Evaluation and Program
Planning, 10, 263-211.
Collins, M. (1994). Putting ourselves into practice: New prospects for program planning and evaluation. In M. Brown (Ed.), A
collection of original essays on curriculum for the workplace, 94–131. Victoria, Australia: Deakin University Press.
Compton, D. W., Baizerman, M., & Stockdill, S. H. (2002). The art, craft, and science of evaluation capacity building. New
Directions for Program Evaluation, 93.
Cooperrider, D & Srivastva, S. (1987). Appreciative Inquiry in organizational life. Research in Organizational Change and
Development 1, 129-169.
Cousins, J.B. & Whitmore, E. (1998). Framing participatory evaluation. In E. Whitmore (Ed.), Understanding and practicing
participatory evaluation. New Directions for Evaluation, 80, 2-23.
Cousins, J.B. & Chouinard, J.A. (2012). Participatory evaluation up close: An integration of research-based knowledge.
Charlotte, NC: Information Age Publishing.
Cousins, J. B., & Earl, L. M. (1992). The case for participatory evaluation. Educational Evaluation and Policy Analysis, 14,
397–418.
ERO (2013). He Pou Tātaki:How ERO reviews early childhood services. Wellington, New Zealand: Education Review Office.
ERO (2012). Statement of Intent 2012-2013. Wellington, New Zealand: Education Review Office.
Garaway, G. B. (1995). Participatory practice. Studies in Educational Evaluation, 21,. 85-102,
Greene, J.G. (1988). Stakeholder participation and utilization in program evaluation. Evaluation Review, 12, 91–116.
Hall, J. N., Ahn, J. & Greene, J. C. (2012). Values engagement in evaluation: Ideas, illustrations, and implications. American
Journal of Evaluation, 33, 195-207.
Hawkins, P. (2012). Successful evaluation management: Engaging mind and spirit. The Canadian Journal of Program
Evaluation, 25 (3) 27–36.
Hood, S., Hopson, R. & Frierson, H. (2005). Introduction : this is where we stand S. Hood, R. Hopson, and H. Frierson (Eds.). The
role of culture and cultural context: a mandate for inclusion, the discovery of truth and understanding in evaluative theory and
practice . Greenwich, CT: IAP
Hopkins, D., & West, M. (2002). Evaluation as school improvement. In D. Nevo (Ed.) School-based Evaluation: An International
Perspective. Oxford, England: Elsevier Science.
Letiecq, B. L. & Bailey, S. J. (2004). Evaluating from the outside. Evaluation Review, 28(4), 342-357
Lincoln, Y. & Guba, E. (1985). Naturalistic Inquiry. Thousand Oaks, CA.: Sage
MacLellan-Wright, M.F., Patten, S., de la Cruz, A.M. & Flaherty, A. (2007). A participatory approach to the development of an
evaluation framework: process, pitfalls, and payoffs. The Canadian Journal of Program Evaluation 22(1), 99–124.
Morris, D.B. (2002). The inclusion of stakeholders in evaluation: benefits and drawbacks. The Canadian Journal of Program
Evaluation, (17)2, 49-58.
Mutch, C. (2012). Complementary evaluation: the development of a conceptual framework to integrate external and internal
evaluation in the New Zealand school context. Policy Futures in Education, 10(5), 569-586.
Nevo, D. (1994). Combining internal and external evaluation: A case for school-based evaluation. Studies in Educational Evaluation,
20(1), 87-98.
Nevo, D. (2002). Dialogue evaluation: combining internal and external evaluation. In D. Nevo (Ed.) School-based Evaluation: An
International Perspective. Oxford, England: Elsevier Science.
Patton, M. Q. (1998) ‘Discovering Process Use’, Evaluation 4(2): 225–33.
Patton, M. Q. (2008). Utilization-Focused Evaluation (4th ed.).Thousand Oaks, CA; Sage Publications Inc.
Smits, P.A. & Champagne, F. (2008). An assessment of the theoretical underpinnings of practical participatory evaluation. American
Journal of Evaluation 29, 427-442.
Stake, R. E. (1975). Evaluating the Arts in Education. Colombus, OH: Merrill.
Stake, R. E. (1983). Program evaluation, particularly responsive evaluation. In G. F. Madaus, M. Scriven, & D. L. Stufflebeam (Eds.),
Evaluation Models. Boston: Kluwer-Nihoff.
Taut, S. & Aiken, M. C. (2010). The role of stakeholders in educational evaluation. Educational evaluation: Contexts of Educational
Evaluation, 629-635.
Themessl-Huber, M.T. & Grutsch, M.A. (2003). The shifting locus of control in participatory evaluations. Evaluation 9, 92-111.
Weaver, L., & Cousins, J. B. (2004). Unpacking the participatory process. Journal of MultiDisciplinary Evaluation 1, 19-40.