UNWOMEN/IPEN TRANSFORMATIVE
MIXED METHODS EVALUATION:
DAY 3 MIXED METHODS
Prof. Donna M. Mertens
Gallaudet University
Almaty, Kazakhstan
July 2011
Three days together
Day 1: What is the role of the evaluator?
Overview of evaluation (Terms of Reference)
Transformative paradigm
Diversity & social justice; privilege
Cultural competence
Day 2: Quantitative & qualitative approaches
Day 3: Mixed methods approaches
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Methods Options




Quantitative approaches such as experimental, quasiexperimental, causal-comparative, correlational, survey, and
single-case designs
Qualitative approaches such as group processes (e.g., focus
groups or some indigenous methods), case studies, ethnographic
research, phenomenological research, and PAR
Gender analysis is a mixed-methods approach that provides a
framework for transformative research and evaluation that has
potential for transfer to other groups that experience
discrimination.
Mixed methods are most likely to be the approach of choice
because of the need to integrate community perspectives into
the inquiry process, thus necessitating collection of qualitative
data during the research or evaluation process. (Mertens, 2009,
TRE, p. 165).
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Mixed Methods Decisions




Compatibility: Match the purpose, focus, questions,
and design.
Timing: Determine the temporal relationship between
the quantitative and qualitative data collection.
Weight: Establish priority or emphasis of the
qualitative and quantitative approaches in the study.
Mixing: Determine when quantitative and qualitative
strategies/data will be mixed in the process of
evaluation.
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Concurrent Design
Quantitative and Qualitative occur more or less simultaneously
Quantitative
Qualitative
Sequential Design: Quantitative Followed by Qualitative
OR
Sequential Design: Qualitative Followed by Quantitative
Qualitative
Quantitative
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Mertens, 2009, TRE, p. 167
Community
Participation
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Example: Making Visible
Botswana youth:
addressing power
inequities in the
fight against
HIV/AIDS using a
transformative lens
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Transformative Mixed Methods Design
Stage 1
Qual
Assemble
team; read
documents;
engage in
dialogues
Stage 2
Concurrent
Preliminary
studies: youth,
older men
Stage 3
Sequential
Stage 4
Concurrent
Pilot intervention:
Observations,
Interviews, Surveys
Process eval
Demographic
information;
Surveys;
Incidence data
Pretest:
Knowledge,
Attitude,
Behavior;
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Post
tests:
Quant
Qual;
Behavior
& Policy
Change;
Transfer
To other
contexts
Transformative Design Components:
Post-Colonial Critical Ethnography
A statement of your evaluation problem or questions (focus).
 A description of your data-collection methods, including interviewing,
journaling, and coding processes, and how these will be accomplished with the
evaluator as a co-performer in the field or participant observer.
 An explanation of your ethical methods and how the welfare of the
participants will be put first by protecting their rights, interests, privacy,
sensibility, and offering reports at key stages to them, including the final
report.
 A description of the participants in terms of population, geographic location,
norms and rules, significant historical and cultural context, and expectations for
key informants.
 A time frame for entering the field, collecting the data, departing from the
field, coding and analysis and completion of the written report, and/or public
performance.
 Use of a critical theoretical framework in the design, implementation, and
dissemination of the study.
Madison (2005)

IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
PAR and Transformative Design
1. The group decides on the focus and questions for the
evaluation.
2. Evaluators and participants observe, engage in action,
observe and record.
3. Evaluators and participants immerse themselves in action
and elaborate and deepen their understandings.
4. Group members reassemble and share their knowledge,
using this iteration as an opportunity to revise their plans
for the next cycle ofevaluation.
5. This cycle might be repeated between 6 and 10 times
depending on the complexity of the evaluation context.
(Heron & Reason, 2006).
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Transformative MM and PAR
Greenwood and Levine (2007) summarize: “Surveys,
statistical analyses, interviews, focus groups,
ethnographies, and life histories are all acceptable,
if the reason for deploying them has been agreed
upon by the AR collaborators and if they are used
in a way that does not oppress the participants” (p.
6).
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Questions for Thought: Design
What is your reaction to the idea that one group
will receive an intervention and another group will
not?
 Under what circumstances would you accept that
format as an ethical course of action?
 What other alternatives are possible?
(Mertens, 2009, TRE, p. 197)

IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Gender Analysis and MM
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Outcomes, Outputs & Impacts
Inputs —> Activities —> Outputs —> Outcomes —>
Impacts
 Outputs are the products, services, and capacities that result
from the completion of activities
 Outcomes are the intended or achieved short-term and
medium-term effects of an intervention’s outputs, usually
requiring the collective effort of partners.
 Impacts are the long term effects or change to
which the programme, through collective effort with
partners, will contribute.
GE/HR evaluation: how did the programme contribute to GE
and HR?
INDICATORS



Indicators are the means by which you determine
progress towards a result or whether an expected
result has been achieved.
Developed for ALL levels of results – outputs,
outcomes & impact
Indicators measure (quantitatively or qualitatively)
the status of an expected result
Source: UNIFEMIPEN
RBMAlmaty
training Kazakhstan July 2011Module
Mertens2-1
Mixed Methods
15
INDICATORS


Quantitative: focus on numbers & counting (percentage of
women and men in parliament, male and female wage rates,
school enrolment for girls and boys)
Qualitative: capture opinions, attitudes and feelings (nature of
dissatisfaction, extent of increased awareness)
IPEN Almaty Kazakhstan July 2011Module
Mertens2-1
Mixed Methods
16
EXAMPLES
Quantitative - Can be directly counted & expressed as a
number
 % of … # of … Frequency of … Ratio of … Amount of …
 Timeliness of …
Qualitative - Involves perception
(can be expressed quantitatively or as narrative)
 Level of Satisfaction with … Knowledge of …
 Ability to … Appropriateness of... Importance of …
 Use and usefulness of …
Source: UNIFEM RBM Training.
IPEN Almaty Kazakhstan July 2011Module
Mertens2-1
Mixed Methods
17
GENDER SENSITIVE INDICATORS


Measures gender-related changes in society over
time
Includes
 sex-disaggregated indicators that provide separate
measures for women and men
 Indicators that are gender-specific to either women
or men
Source: A. Moser. 2007
Gender and Indicators
IPEN Almaty Kazakhstan July 2011Module
Mertens2-1
Mixed Methods
18
HUMAN RIGHTS INDICATORS
Assess both state progress in guaranteeing human
rights and individual programme adherence and
promotion of rights.
HR indicators should measure:
 Improvements
in the capacities of rights holders and
duty bearers to realize rights
 Improvements
in the enjoyments of rights
IPEN Almaty Kazakhstan July 2011Module
Mertens2-1
Mixed Methods
19
GE & HR RESPONSIVE EVALUATION
Follow-Up and Use:

Dissemination strategies should make evaluation findings
accessible and barrier-free to women, including both RHs and
DBs




Targeting women’s organizations/networks and knowledge
networks
User-friendly language
Stakeholder workshops that include women and other groups
subject to discrimination (RHs &DBs)
Management Response should be issued to ensure follow-up on
key gender and human rights issues
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
EXAMPLE INDICATORS
Outcome indicator: Evidence of changes in the
capability of ministries of xxx to formulate and
implement policies responsive to indigenous women
Output indicator: Capacity assessments (focusing on
individual competencies within ministries of xxx)
indicating increase in relevant knowledge, skills and
attitudes within key public institutions
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Examples: GE/HR Indicators




proportion of target group by sex aware of the
benefits of birth registration,
employment to population ratio by age and sex,
gross primary graduation ratio by sex,
percentage of women in parliament.
“REALITY CHECK”





Expense and making use of available data; are there
too many indicators?
Unrelenting pull to quantitative measures - need a
balance of qualitative and quantitative data
Proxy indicators, where measurement takes an more
indirect path
Indicators should be disaggregated, as much as
possible, by sex, race, ethnicity, age, geographic area
Indicators do not exist in a vacuum and must be tied to a
result.
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
EXERCISE: DEVELOP INDICATORS
Develop two indicators (one qualitative and
one quantitative) for the course example.
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Sample Relevance Evaluation ?’s


How well do the programme objectives target the
identified rights and needs of male and female
beneficiaries?
What rights does the programme advance under
CEDAW, the Millennium Development Goals and other
international development commitments?
Sample Evaluation Effectiveness ?’s


To what extent have the objectives been achieved,
and do the intended and unintended benefits meet
fairly the needs of disadvantaged women?
To what extent have the capacities of duty-bearers
and rights-holders been strengthened?
Sample Evaluation Efficiency ?’s



Could the activities and outputs have been delivered
with fewer resources to the target populations without
reducing their quality and quantity?
How has the programme maximized partnerships in
the delivery of the programme?
Have UN Woman’s organisational structure,
managerial support and coordination mechanisms
effectively supported the delivery of the programme?
Sample Evaluation Impact ?’s


To what extent have efforts been successful in
stopping harmful and discriminatory practices against
women?
What is the evidence that the programme enabled the
rights-holders to claim their rights more successfully
and the duty-holders to perform their duties more
efficiently?
Sample Evaluation Sustainability ?’s


Is the programme supported by national and local
women’s organisations?
Do these organisations demonstrate leadership
commitment and technical capacity to continue to
work with the programme or advocate for change?
Additional GE/HR ?’s





Equality and non-discrimination: Did the programme benefits
affect equally men and women?
Empowerment: Did the budget designate sufficient resources and
level of effort to address the inclusion of disadvantaged or
marginalised groups?
Accountability: Were monitoring data (disaggregated according
to relevant criteria such as gender, age, ethnicity, location, and
income) collected and used to adjust implementation?
Social transformation: Does the intervention’s theory of change
include attention to GE and HR?
Participation and inclusion: Did the implementation make
systematic and appropriate efforts to include women and men,
and/or reach out to disadvantaged groups? UN Women 2011
GE/HR Evaluation Methods




Avoid bias: selection of data collection methods & in
sampling methods
Potential biases: Gender & Power (sources able to
contribute more easily because of privacy &
confidentiality issues ), Class or distance (favoring
the more accessible)
Plans for how to include marginalized groups
Mixed methods: Quan & Qual
Rigor applied to methods choices
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
In the positivist tradition, there are two important tests of
knowledge claims:
1. Is the knowledge claim true in this situation? Are the
changes observed in the dependent variable due to the effect of
the independent variable? (Internal Validity)
2. Is the knowledge true in other situations?
Generalizability? (External Validity)
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Comparative Studies: Rigor







Causal relationship assumed? Competing
explanations?
Comparable groups in causal comparative?
Third variable cause both predictor and criterion
variables?
Sub groups analysis?
Correlational: ordering of variables?
Predictive studies: Other screening variables? Level of
.8 or better?
Reliability and range of variables
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Criteria that establish rigor in
qualitative methods
Credibility: just like validity in qualitative research –
asks if there is a correspondence between the way the
respondents actually perceive social constructs and the
way the evaluators portray their viewpoints.
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Criteria that establish rigor in
qualitative methods
(a) persistent observation
(b) peer debriefing
(c) progressive subjectivity
(d) member checks
(e) triangulation
(f) transferability
(g) dependability
(h) confirmability
(i) authenticity and fairness
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
TRANSFORMATIVE EVALUATION: Rigor





Emphasizes Human Rights and Social justice
Analyses asymmetric power relations
Advocates culturally competent relations between
the evaluator and community members
Employs culturally appropriate mixed methods tied
to social action
Applies feminist theory, critical race theory,
postcolonial and indigenous theories
Mertens (2009) Transformative Research and Evaluation. The Guilford Press.
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Authenticity
(Transformative criteria)
 Fairness: The evaluator presents all value
differences, views, and conflicts.
 Ontological Authenticity: An individual’s or
groups’ conscious experience of the world
became more informed and sophisticated.
 Catalytic Authenticity: Action was
stimulated by the inquiry process.
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Power and Privilege





How do we understand the dynamics of power when
participatory methods are employed by the powerful?
Whose voices are raised and whose are heard?
How are these voices mediated as issues of
representation become more complex with the use of
participatory methods in larger-scale planning and
consultation exercises? (Mertens, 2009, P. 85)
What if I am a member of the community? How does that
prepare me to work in that community? What if I am not
a member of a community? To what extent is it necessary
to share salient characteristics of a community?
How does cultural competency come into the discussion of
interactions in evaluation contexts? (Mertens & Wilson, in
press) IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Criteria for Rigor: Utilization




Dissemination
Use
Management Response
Engagement with Stakeholders
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Dissemination Purposes





Transparency and accountability
Informing and improving an organisation’s work
Sharing good practices and ‘how to’ to advance
women’s human rights
Sharing lessons with partners on the ground and
building their capacity, and
Generating knowledge on how gender equality,
women’s empowerment and human rights can be
advanced
Dissemination




Language and presentation of the report: use graphs or
pictures, and be written in a manner that is gender and
culturally sensitive.
Translation: anticipate the need for translation
when planning the evaluation so time and budget can be
included)
Making the report public: Within the UN, there is
requirement to make all evaluations public (UNEG Norms and
Standards) and in UN Women, this is accomplished by
having the Evaluation Unit post the evaluation report in the
Evaluation Resource Center (which is publicly accessible).
Reaching target audiences in a user-friendly way: lower
rates of literacy? The manager needs to be creative in using
other dissemination channels, suited to different audiences.
Dissemination Strategies






Brochures outlining key
evaluation lessons and
recommendations
Annual reports
Articles in technical or
organisational newsletters
News releases
Press conferences
Media appearances




Public meetings, public
debates or town halls
Seminars, workshops, and
informal group
discussions
Electronic media (e-mail,
websites, blogs etc.)
Meeting with community
leaders, one on one.
Challenges to Use




Lack of consensus on recommendations & required
action
Those who don’t like the results attack the process
Dissemination is minimal: no funds, interest or time
No follow-up process
Formal Management Response



presentation of the response, action or non action to
evaluation recommendations and lessons learned
and follow-up or tracking mechanisms.
engage with stakeholders to reflect on the
evaluation process, findings, recommendations and
lessons learned.
Involve reference group in developing the
management response, with the manager.
Agenda for Action

How does the information you learned here apply
to the work that you are/will be doing in
evaluation?
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Resources




Mertens, D. M. & Wilson, A. (in press).
Program Evaluation. NY: Guilford.
Mertens, D. M. (2010). Research and
evaluation in education and psychology:
Integrating diversity with qual, quant and
mixed methods. 3rd ed. Thousand Oaks,
CA: Sage.
Mertens, D. M. (2009). Transformative
research & evaluation. NY: Guilford
Mertens, D. M. & Ginsberg, P. (2009).(Eds.)
Handbook of Social Research Ethics.
Thousand Oaks, CA: Sage.
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Contact information

Donna M. Mertens, Gallaudet University
 Donna.Mertens@Gallaudet.edu
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods