Social Justice Amidst
Standards and
Accountability
Donna M. Mertens, Keynote
Independent Consultant
Minnesota Evaluation Studies Institute, MESI Spring
Training
Minneapolis MN
March 2015
2
Why this topic? Why now?
Social Justice
Amidst Standards and
Accountability
Mertens Keynote MN MESI March 2015
3/11/2015
3
2015 International Year of Evaluation
EvalPartners, UNICEF, UN Women,
and the International Organization for
Cooperation in Evaluation
Peer 2 Peer Initiative – gender and
equity focused evaluation
Mertens Keynote MN MESI March 2015
3/11/2015
4
Need for Social Justice Link to Evaluation:
Pervasiveness of racism and other isms
Mertens Keynote MN MESI March 2015
3/11/2015
5
Need for a Social Justice Lens
Marie Battiste (2000):
“…society is sorely in need of what
Aboriginal knowledge has to offer”
Chilisa (2005)
“it is an issue of life and death”
Brookes (2006)
Failure to include racism as a potential
contributor to disparities
Mertens Keynote MN MESI March 2015
3/11/2015
6
My hypothesis:
If we begin by prioritizing social justice
and human rights
And we appropriately involve
community members in the evaluation
process,
Then we will increase the probability of
social transformation as a result of our
evaluation.
Mertens Keynote MN MESI March 2015
3/11/2015
7
What does our evidence lack in
credibility from a social justice
perspective?
Mertens Keynote MN MESI March 2015
3/11/2015
8
Social Justice
Standards
Accountability
Mertens Keynote MN MESI March 2015
3/11/2015
9
Accountability Question
Did “it” work?
Should we keep spending our money
on “it”?
Mertens Keynote MN MESI March 2015
3/11/2015
10
Social Justice Questions
 Was “it” the right thing?
 Was “it” chosen and/or developed and implemented in
culturally responsive ways?
 Were contextual issues of culture, race/ethnicity,
gender, disability, deafness, religion, language,
immigrant or refugee status, age or other dimensions of
diversity used as a basis for discrimination and
oppression addressed?
 How were issues of power addressed?
 Do we want to continue to spend money on things that
don’t work?
Mertens Keynote MN MESI March 2015
3/11/2015
11
Resources for Evaluators
AEA Guiding Principles
AEA Public Statement on Cultural Competency
Joint Committee Program Evaluation Standards
Evaluators who have dedicated their lives
towards furthering social justice
Communities who want social justice
Mertens Keynote MN MESI March 2015
3/11/2015
12
Good news; bad news for social justice
in evaluation
1.2% (n=22) in
Harner’s (2014) survey
of evaluators
identified their
theoretical frame as
Social Justice
Mertens Keynote MN MESI March 2015
 69% (n=819) of 1,187
evaluators either strongly
or somewhat agreed
with this statement:
Evaluation should
focus on bringing
about
social justice.
3/11/2015
13
AEA Guiding Principles (2004): Systematic Inquiry,
Competence, Integrity/Honesty, Respect for
People, and Responsibilities for General and
Public Welfare
To ensure recognition, accurate interpretation and respect for
diversity, evaluators should ensure that the members of the evaluation
team collectively demonstrate cultural competence. Cultural
competence would be reflected in evaluators seeking awareness of
their own culturally-based assumptions, their understanding of the
worldviews of culturally-different participants and stakeholders in the
evaluation, and the use of appropriate evaluation strategies and skills
in working with culturally different groups. Diversity may be in terms of
race, ethnicity, gender, religion, socio-economics, or other factors
pertinent to the evaluation context. (Competence B2)
Mertens Keynote MN MESI March 2015
3/11/2015
14
AEA Guiding Principle: Cultural Competence
Linkage with Transformative Axiology and Epistemology
To ensure recognition, accurate
interpretation, and respect for diversity,
evaluators should ensure that the members
of the evaluation team collectively
demonstrate cultural competence
(American Evaluation Association, Public
Statement on Cultural Competence in
Evaluation, 2011)
Mertens Keynote MN MESI March 2015
3/11/2015
15
What is cultural competence?
 Cultural competence is not a state at which one arrives; rather,
it is a process of learning, unlearning, and relearning.
 It is a sensibility cultivated throughout a lifetime.
 Cultural competence requires awareness of self, reflection on
one’s own cultural position, awareness of others’ positions, and
the ability to interact genuinely and respectfully with others.
 Culturally competent evaluators refrain from assuming they fully
understand the perspectives of stakeholders whose
backgrounds differ from their own.
 Cultural competence is context dependent. (AEA 2011)
Mertens Keynote MN MESI March 2015
3/11/2015
16
Evaluation and Validity and Cultural
Competence
 accurately and respectfully reflect the life experiences and
perspectives of program participants in their evaluations.
 establish relationships that support trustworthy
communication among all participants in the evaluation
process.
 draw upon culturally relevant, and in some cases culturally
specific, theory in the design of the evaluation and the
interpretation of findings.
 select and implement design options and measurement
strategies in ways that are compatible with the cultural
context of the study. (AEA 2004, p. 6)
Mertens Keynote MN MESI March 2015
3/11/2015
17
Recognize the dynamics of power
Cultural groupings are ascribed differential status
and power, with some holding privilege that they
may not be aware of and some being relegated
to the status of “other.” For example, language
dialect and accent can be used to determine the
status, privilege, and access to resources of
groups. (AEA, 2004, p. 7)
Mertens Keynote MN MESI March 2015
3/11/2015
18
Standards for Good Evaluation
(Yarborough et al. 2011)
 Propriety: responsive and
inclusive, formal
agreements, human rights
and respect, clarity and
fairness, transparency and
 Feasibility: management,
disclosure.
practical, contextual viability
 Accuracy: valid, reliable,
(recognize, monitor, and balance
explicit program and
the cultural and political interests
context descriptions, sound
designs and analysis
and needs of individuals and
groups)
 Evaluation accountability:
meta-evaluation
 Utility: evaluator credibility,
attention to stakeholders,
negotiated purposes, explicit
purposes
Mertens Keynote MN MESI March 2015
3/11/2015
19
Accountability and Social Justice
Code word for RCT?
Accountable to whom?
Ways to strengthen arguments about
accountability are not limited to RCTs
Theoretical streams in evaluation provide
options for demonstrating accountability
Mertens Keynote MN MESI March 2015
3/11/2015
Evaluation Theory and Social Justice –
Alkin’s Tree: Methods, Use & Values Branches
Methods
3/11/2015
Use
Values
Mertens Keynote MN MESI March 2015
20
Mertens & Wilson 2012:
Methods, Use, Values, & Social Justice Branches
Use Values
Social
Methods
Justice
3/11/2015
Mertens Keynote MN MESI March 2015
21
22
Metaphor: Tree or Water?
Mertens Keynote MN MESI March 2015
3/11/2015
23
Paradigms & Branches
Paradigm
Branch
Post-positivist
Methods
Constructivist
Values
Transformative
Social Justice
Pragmatic
Use
Mertens Keynote MN MESI March 2015
3/11/2015
24
Transformative Theories
 Feminist theories
 Critical Theories
 Transformative
Participatory Action
theories
 Human Rights Theories
 Indigenous theories
 Disability Rights Theories
 Deafness rights theories
 Critical Race Theories
 Queer theories
Mertens Keynote MN MESI March 2015
3/11/2015
Transformative Paradigm
25
Philosophical
Assumptions
Axiology
Respect for cultural norms; support for human rights and
social justice; reciprocity
Ontology
Issues of power & critical interrogation of multiple realities:
social, political, cultural, economic, race/ethnic, gender,
age, religion and disability values to unmask those that
sustain an oppressive status quo
Epistemology
Issues of power & Interactive link; knowledge is socially
and historically located; trusting relationship.
Methodology
Qualitative (dialogic)/ Quantitative / Mixed Methods;
Context
3/11/2015
Mertens Keynote MN MESI March 2015
26
Court Access Project
Start from community
Advisory board formed
Begin understandings from the vantage
point of the “least privileged”
Work with an awareness of diversity in the
community
Provide respectful support for engagement
Use a cyclical approach to data collection
and use
Mertens Keynote MN MESI March 2015
3/11/2015
27
Example: HIV/AIDS Prevention in
Botswana
Botswana youth:
addressing power
inequities in the
fight against
HIV/AIDS using a
transformative
lens
Mertens Keynote MN MESI March 2015
3/11/2015
28
Transformative Cyclical Mixed Methods
Design
Stage 1
Qual
Assemble
team; read
documents;
engage in
dialogues
Stage 2
Stage 3
Stage 4
Stage 5
Concurrent
Sequential
Sequential
Concurrent
Preliminary
studies: youth,
older men
Demographic
information;
Surveys;
Incidence
data
Mertens Keynote MN MESI March 2015
RCT
Pilot intervention:
Observations,
Interviews,
Surveys
Process eval
Pretest:
Knowledge,
Attitude,
Behavior;
Post
tests:
Quant
Qual;
Behavior
& Policy
Change;
Transfer
To other
contexts
3/11/2015
Indigenous Contributions
29
Social justice as
a frame for
evaluation
Engagement
with indigenous
communities
Mertens Keynote MN MESI March 2015
3/11/2015
30
Indigenous Paradigm
Chilisa (2012)
Relational Axiology
Wilson (2008)
Relational Ontology
Linda T. Smith (1999)
Marie Battiste (2000)
Relational
Epistemology
LaFrance & Crazy
Horse (2009)
Relational
Methodology
Mertens Keynote MN MESI March 2015
(Chilisa, 2012)
3/11/2015
31
Ethics and Indigeneity
 Connections between people, past, present, and future,
all living and nonliving things (ubuntu)
 Respect and reciprocity: listens, pays attention,
acknowledged, and crates space for the voices and
knowledge systems of Indigenous people
 Contribute to a better future
 Spirituality
Mertens Keynote MN MESI March 2015
3/11/2015
Questions
32
 What is credible evidence?
 What criteria establish credibility of evidence?
 What does our evidence lack in credibility?
 What sources give us insight into credibility?
 What is the place of social justice in credibility?
 How do voices of marginalized communities
enhance our understandings of credibility?
 Why is it important for researchers to understand
marginalized communities’ perspectives of social
justice?
Mertens Keynote MN MESI March 2015
3/11/2015
33
Methodological Questions
 What data collection methods can the evaluator use to become
better acquainted with the various stakeholder groups?
 How can the evaluator design the study so that the community
members are included in decision making?
 How could the use of mixed methods contribute to the quality of
the study?
 How can the evaluator design the methods to increase the
probability of furthering social justice?
Mertens Keynote MN MESI March 2015
3/11/2015
34
Water as Metaphor
Eventually, all things merge into one,
and a river runs through it. The river was
cut by the world's great flood and runs
over rocks from the basement of time. Norman Fitzroy Maclean
 When you put your hand in a flowing stream, you
touch the last that has gone before and the first of
what is still to come. - Leonardo da Vinci
My soul is full of longing For the secret of
the Sea, And the heart of the great
ocean Sends a thrilling pulse through
me – Henry Wadsworth Longfellow
Mertens Keynote MN MESI March 2015
http://www.finestquotes.com/select_quote-category-Water-page-1.htm#ixzz2968zeXR9
3/11/2015
35
Resources
Mertens Keynote MN MESI March 2015
 Mertens, D. M. & Wilson, A. (2012).
Program Evaluation Theory and
Practice: A Comprehensive Guide.
NY: Guilford.
 Mertens, D. M. (2015). Research
and evaluation in education and
psychology: Integrating diversity
with qual, quant and mixed
methods. 4th ed. Thousand Oaks,
CA: Sage.
 Mertens, D. M. (2009).
Transformative research &
evaluation. NY: Guilford.
 Mertens, D. M. & Ginsberg, P.
(2009).(Eds.) Handbook of Social
Research Ethics. Thousand Oaks,
CA: Sage.
 American Evaluation Association
(2011). Public Statement on
Cultural Competence in
Evaluation. AEA.
3/11/2015
36
Contact information
Donna M. Mertens, PhD
Independent Consultant
Donna.Mertens@Gallaudet.edu
Mertens Keynote MN MESI March 2015
3/11/2015