WEAVING SOCIAL JUSTICE AND
EVALUATION TOGETHER
DONNA M MERTENS, PHD
INDEPENDENT CONSULTANT
MINNEAPOLIS MN
MARCH 2015
INTRODUCTION/EXPECTATIONS• NAME, POSITION, ORGANIZATION (NO ACRONYMS PLEASE).
• WHERE DO YOU COME FROM?
• WHAT IS YOUR EXPERIENCE IN EVALUATION?
• WHAT HAS BEEN YOUR EXPERIENCE WITH SOCIAL JUSTICE?
• WHAT DO YOU WANT TO GET OUT OF THIS WORKSHOP?
Mertens Minneapolis MN March 2015
2
TRANSFORMATIVE PARADIGM
Philosophical
Assumptions
Axiology
Respect for cultural norms; support for human rights and social justice;
reciprocity
Ontology
Issues of power & critical interrogation of multiple realities: social,
political, cultural, economic, race/ethnic, gender, age, religion and
disability values to unmask those that sustain an oppressive status quo
Epistemology
Issues of power & Interactive link; knowledge is socially and
historically located; trusting relationship.
Methodology
Qualitative (dialogic)/ Quantitative / Mixed Methods; Context
Mertens Minneapolis MN March 2015
3
TRANSFORMATIVE MIXED METHODS AND ENVIRONMENTAL JUSTICE
Phase 1
Qual: Identify
stakeholders;
build
relationships;
analyze
documents
Phase 2
Qual:
Interviews,
focus
groups,
town
meetings
Quant: Identify
available
Quant: Establish
demographic
baseline for
and
Mertens Minneapolis MN March 2015
pollutants and
environmental
health status
data
Phase 3
Phase 4
Phase 5
Qual:
Develop
intervention
s based on
stakeholder
input
Quant: process
assessment of
implementation;
pretests
Quant: Posttest
measures
Quant/Qual:
Pilot test
interventions
& data
collection
Qual: observe
and interview
stakeholder
groups during
intervention
Qual:
Interviews,
observations,
policy
planning
4
CONTEXTUAL FACTORS
• HISTORY OF THE NATION/COUNTRY/LOCALE
• POLITICAL –GOVERNMENT, LEGISLATION,
CORRUPTION
• ORGANIZATIONAL – NGOS, UNIVERSITIES,
AGENCIES
• CULTURAL INCLUSION/EXCLUSION– ETHNIC
GROUPS, RELIGION, GENDER, DISABILITY,
LANGUAGES
• POWER STRUCTURES,
DISCRIMINATION/OPPRESSION
• COMMUNITY RESILIENCE
Mertens Minneapolis MN March 2015
• WHAT ABOUT OUR COMMUNITY AND THE
RELEVANT ORGANIZATIONS HINDERS OR HELPS
US ACHIEVE PROJECT GOALS?
• WHICH CONTEXTUAL FACTORS HAVE THE
GREATEST BEARING ON PROJECT SUCCESSES
OR STUMBLING BLOCKS?
• TO WHAT EXTENT WILL THE PROJECT TARGET
IMPORTANT COMMUNITY AND BENEFICIARY
NEEDS?
• WHAT ECONOMIC CONDITIONS, POPULATION
CHARACTERISTICS, RELATED PROJECTS AND
SERVICES, AND NEEDS AND PROBLEMS OF THE
TARGETED POPULATION ARE RELEVANT?
5
CONTEXT IN KYRGYZ REPUBLIC: GENDER
• WOMEN ARE MORE LIKELY TO MAKE A CHOICE IN FAVOR OF EDUCATION, WHICH IS LATER
TRANSFORMED INTO LOWER-PAYING OCCUPATIONS. THE PAY IN TRADITIONALLY “FEMALE”
OCCUPATIONS IS 2.5 TIMES LOWER THAN IN TRADITIONALLY “MALE” OCCUPATIONS, AS “FEMALE”
OCCUPATIONS ARE MOSTLY FUNDED BY THE STATE BUDGET AND HAVE BEEN TO A LESSER EXTENT
AFFECTED BY THE ONSET OF THE MARKET ECONOMY.
• MEN HAVE ALSO EXPERIENCED NEGATIVE CONSEQUENCES OF SOCIO-POLITICAL AND ECONOMIC
STEREOTYPES, WITH THE PREVALENCE OF STEREOTYPES THAT VIEW MEN ONLY “BREAD-WINNERS”.
TODAY, THE SUICIDE RATE AMONG MEN IN KYRGYZSTAN IS FOUR TIMES HIGHER THAN AMONG
WOMEN. AVERAGE LIFE EXPECTANCY OF MEN IS ONLY 63.5 YEARS, WOMEN, ON AVERAGE, LIVE
ALMOST NINE YEARS LONGER. THERE IS A DECLINE IN THE LEVEL OF EDUCATION OF MEN, WHICH
CAN LEAD TO MORE CONFLICTS IN THE FAMILY AND SOCIETY, LACK OF TOLERANCE TO DIFFERENCES,
AND GREATER WILLINGNESS TO USE BRUTE FORCE. (NATIONAL COUNCIL FOR SUSTAINABLE
Mertens Minneapolis MNOF
MarchTHE
2015 KYRGYZ REPUBLIC, 2013)
6
DEVELOPMENT
UNDERSTANDING THE CONTEXT: CAPACITY BUILDING
& ORGANIZATIONAL ASSESSMENT

WHAT IS THE CURRENT STATUS IN AN ORGANIZATION WITH REGARD TO A
SPECIFIC FOCUS (E.G., SUFFICIENT TRAINED PERSONNEL? UP TO DATE
PRACTICES?)

WHAT IS THE CURRENT LEVEL OF KNOWLEDGE ABOUT EVALUATION, ITS
CURRENT USES, AND ATTITUDES TOWARD EVALUATION?
• WHAT IS NEEDED TO SUPPORT ORGANIZATIONAL LEADERSHIP, INCREASE
FUNDING, RECRUIT STAFF WITH EVALUATION EXPERTISE, CONDUCT
ORGANIZATION-WIDE TRAINING, TECHNICAL ASSISTANCE FOR THOSE
UNDERTAKING EVALUATIONS, AND SHARING OF INFORMATION IN TERMS OF
STRATEGIES FOR EVALUATIONS AND USE OF THEIR FINDINGS?
Mertens Minneapolis MN March 2015
7
EVALUATION IN THIS CONTEXT
• WHAT DOES EVALUATION MEAN HERE?
• WHAT DOES IT LOOK LIKE WHEN IT IS DONE?
• HOW DO PEOPLE TALK ABOUT IT?
• WHO DOES IT?
• HOW IS IT DONE?
• HOW IS IT USED?
Mertens Minneapolis MN March 2015
8
EVALUATION: DEFINITION FROM UN
• AN ASSESSMENT OF PROGRAMME ACTIVITY
• SYSTEMATIC AND IMPARTIAL
• FOCUSES ON ACCOMPLISHMENTS (EXPECTED AND ACHIEVED)
• CONSIDERS THE RESULTS CHAIN, PROCESSES, CONTEXT, AND CAUSALITY
• IN ORDER TO DETERMINE THE “RELEVANCE, IMPACT, EFFECTIVENESS,
EFFICIENCY AND SUSTAINABILITY OF THE INTERVENTION”
SOURCE: UNEG, 2005
Mertens Minneapolis MN March 2015
9
GENDER EQUALITY
• REFERS TO THE EQUAL RIGHTS, RESPONSIBILITIES, AND
OPPORTUNITIES OF WOMEN AND MEN.
• IT IMPLIES THAT THE INTERESTS, NEEDS AND PRIORITIES OF
BOTH WOMEN AND MEN ARE TAKEN INTO
CONSIDERATION, RECOGNIZING THE DIVERSITY OF
DIFFERENT GROUPS OF WOMEN AND MEN.
• IT IS BOTH A HUMAN RIGHTS ISSUE AND A PRECONDITION
FOR, AS WELL AS AN INDICATOR OF, DEVELOPMENT.
Mertens Minneapolis MN March 2015
SOURCE: UNEG GUIDANCE
10
GENDER EQUALITY AND HUMAN RIGHTS
RESPONSIVE EVALUATION
WHAT DO WE MEAN BY GENDER EQUALITY AND HUMAN RIGHTS RESPONSIVE
EVALUATION?
IT IS AN EVALUATION THAT INTEGRATES A GENDER EQUALITY AND A HUMAN RIGHTS
BASED APPROACH BY:
1) SPECIFICALLY ASSESSING THE EXTENT TO WHICH THE PROGRAMME EVALUATED
IS GUIDED BY ORGANIZATIONAL AND SYSTEM-WIDE OBJECTIVES ON GENDER
AND HUMAN RIGHTS AND
2) INCORPORATING THESE APPROACHES IN THE ACTUAL EVALUATION PROCESSES
Mertens Minneapolis MN March 2015
11
Module 1-2
11
TRANSFORMATIVE EVALUATION
• RECOGNIZE POWER RELATIONS;
• IDENTIFY THE STRUCTURAL CAUSES OF INEQUALITY AND DISCRIMINATION;
• DETERMINE THE IMPACTS OF PROGRAMMES ON IMPOVERISHED PEOPLE AND GROUPS FACING
DISCRIMINATION;
• INVOLVE PARTICIPATORY AND REFLECTIVE PROCESSES; AND
• ACKNOWLEDGE HUMAN RIGHTS AND EQUALITY AND DETERMINE THE CLAIMS OF RIGHTS- HOLDERS AND
OBLIGATIONS OF DUTY- BEARERS
• AIM FOR THE PROGRESSIVE REALIZATION OF HUMAN RIGHTS AND GENDER EQUALITY
SOURCE: UNEG/UNSSC, MODULE, 2008; MERTENS & WILSON, 2012
Mertens Minneapolis MN March 2015
12
Module 1-2
12
BELIEFS ABOUT EVALUATION
BELIEFS
BY STAKEHOLDER GROUPS
• WHAT MAKES A GOOD EVALUATION?
• GOVERNMENT
• PURPOSE OF EVALUATION
• EVALUATORS
• USEFULNESS OF EVALUATION
• NGOS
• APPROACHES TO EVALUATION
• DONOR AGENCIES
• MARGINALIZED COMMUNITIES
• ETC.
Mertens Minneapolis MN March 2015
13
ADVOCACY FOR GOOD EVALUATION:
STANDARDS FOR JUDGING EVALUATION
• UTILITY
• FEASIBILITY
• PROPRIETY
• ACCURACY
• META-EVALUATION
Mertens Minneapolis MN March 2015
14
THINK ABOUT POSSIBLE EVALUATION PURPOSES
AND IMPLIED USES…..
• CYNICAL: BECAUSE IT IS REQUIRED BY THE FUNDING
AGENCY. POSSIBLE USE? FUNDING AGENCY MAY MAKE
DECISIONS TO CONTINUE OR CUT FUNDING, BUT PROBABLY
THE PROGRAM WILL NOT USE IT TO IMPROVE…
• PUBLIC RELATIONS: ONLY POSITIVE EVALUATION DATA WILL
BE REPORTED. POSSIBLE USE: MAKE US LOOK GOOD….
Mertens Minneapolis MN March 2015
15
MULTIPLE PURPOSES ARE POSSIBLE FOR ONE
EVALUATION
• PURPOSE OF EVALUATION MAY BE LIMITED TO A SHORT TIME
FRAME AND MAY BE SPECIFICALLY FOCUSED ON ONE ASPECT OF A
PROGRAM
• PURPOSE OF EVALUATION MAY CHANGE AS THE EVALUAND GOES
THROUGH DIFFERENT PHASES OVER A PERIOD OF TIME (YEARS…)
• YOU MAY HAVE SEVERAL PURPOSES IN YOUR EVALUATION TO MEET
DIFFERENT STAKEHOLDERS NEEDS.
Mertens Minneapolis MN March 2015
16
ADDRESSING EVALUATION PURPOSES: INFLUENCING
THE CONVERSATION
• GAIN INSIGHTS OR DETERMINE
NECESSARY INPUTS (CONTEXT
AND NEEDS ASSESSMENT)
• ASSESS PROGRAM
EFFECTIVENESS
(SUMMATIVE)
• FIND AREAS IN NEED OF
IMPROVEMENT OR TO CHANGE
PRACTICES (FORMATIVE)
• ADDRESS ISSUES OF
HUMAN RIGHTS AND
SOCIAL JUSTICE
(TRANSFORMATIVE)
Mertens Minneapolis MN March 2015
17
PURPOSE: TO GAIN INSIGHT/DETERMINE
NECESSARY INPUTS
EVALUATION TYPES:
• CONTEXT
• CAPACITY BUILDING/ORGANIZATIONAL ASSESSMENT
• NEEDS AND ASSETS ASSESSMENT
• RELEVANCE OF CURRENT SERVICES/STRUCTURE
Mertens Minneapolis MN March 2015
18
EVALUATION PURPOSE: IMPROVEMENT OR
CHANGE PRACTICES
• IMPLEMENTATION
• RESPONSIVE
• PARTICIPATORY
• PROCESS,
• MONITORING,
• FORMATIVE
• DEVELOPMENTAL
Mertens Minneapolis MN March 2015
19
PURPOSE: TO ASSESS PROGRAM EFFECTIVENESS
• OUTCOMES,
• IMPACT,
• SUMMATIVE,
• COST ANALYSIS,
• POLICY,
• REPLICABILITY/
• EXPORTABILITY
• SUSTAINABILITY
Mertens Minneapolis MN March 2015
20
EVALUATION PURPOSE – HUMAN RIGHTS & SOCIAL JUSTICE
• COUNTRY LED
• CRITICAL RACE THEORY
• INDIGENOUS
• CULTURALLY RESPONSIVE
• DISABILITY AND DEAFNESS RIGHTS
• FEMINIST
• GENDER ANALYSIS
• TRANSFORMATIVE PARTICIPATORY
Mertens Minneapolis MN March 2015
21
HOW DO YOU GET PEOPLE TO AGREE ON THE
PURPOSE OF THE EVALUATION?
• ASSEMBLE APPROPRIATE PEOPLE (STAKEHOLDERS)
• ASK APPROPRIATE QUESTIONS:
• WHAT ARE WE DOING RIGHT?
• IS THERE A PROBLEM?
• WHAT HAS CHANGED IN THE ENVIRONMENT/ORGANIZATION
THAT SUGGESTS A NEED FOR CHANGE?
• WHAT NEW DIRECTIONS ARE POSSIBLE?
Mertens Minneapolis MN March 2015
22
TEAM DEVELOPMENT: WHO ARE YOUR
STAKEHOLDERS?
• EVERYONE IMPACTED BY RESULTS – EVEN IF THEY ARE IMPACTED BY BEING EXCLUDED FROM
PARTICIPATION…
• PROJECT ADMINISTRATORS/FUNDERS/POLICY MAKERS;
• PROJECT IMPLEMENTORS (STAFF)
• PARTICIPANTS AND NONPARTICIPANTS
BUT HOW CAN YOU INCLUDE ALL THOSE PEOPLE?
HINT: SOMETIMES YOU HAVE TO SAMPLE W/IN GROUPS TO HAVE A MANAGEABLE NUMBER OF
STAKEHOLDERS. SMILE
Mertens Minneapolis MN March 2015
23
TEAM DEVELOPMENT
• IDENTIFY EXISTING COMMUNITY ACTION GROUPS AND UNDERSTAND THE HISTORY
OF THEIR EFFORTS;
• IDENTIFY EXISTING FORMAL, INFORMAL, AND POTENTIAL LEADERS;
• IDENTIFY COMMUNITY NEEDS AND GAPS IN SERVICES;
• IDENTIFY COMMUNITY STRENGTHS AND OPPORTUNITIES;
• UNDERSTAND YOUR TARGET POPULATION (BOTH NEEDS AND ASSETS) IN ORDER TO
IMPROVE, BUILD, AND SECURE PROJECT CREDIBILITY WITHIN THE COMMUNITY; AND
• CREATE A MOMENTUM FOR PROJECT ACTIVITIES BY GETTING COMMUNITY INPUT.
Mertens Minneapolis MN March 2015
24
• WHO IS ON THE PROGRAM TEAM?
• HOW REFLECTIVE ARE TEAM MEMBERS OF THE TARGETED
COMMUNITY?
• TO WHAT EXTENT DO UNDERREPRESENTED GROUPS
(DISAGGREGATED) HAVE INPUT INTO DECISIONS ABOUT WHAT
AND HOW ISSUES WILL BE ADDRESSED AND HOW THE IMPACT
OF THE INTERVENTIONS WILL BE MEASURED?
• HOW IS RESOURCE DISTRIBUTION AFFECTING THE ABILITY OF
STAKEHOLDERS TO BENEFIT FROM THE INNOVATIONS?
• WHO CANNOT PARTICIPATE AND WHY?
• HOW CAN POWER DIFFERENCES BE SAFELY ACKNOWLEDGED
AND ACCOMMODATED?
MERTENS, 2009, P. 206, TRE
Mertens Minneapolis MN March 2015
25
HOW TO INVOLVE STAKEHOLDERS?
• ESTABLISH A CONSULTATIVE & ADVISORY GROUP
• WHO? KEY REPRESENTATIVES FROM NATIONAL GOVERNMENT COUNTERPARTS, PARTNERS,
CSOS, WOMEN’S ORGANIZATIONS, BENEFICIARIES, PROGRAM MANAGERS, OTHER AGENCIES
AND/ OR DONORS
• ROLE: DEVELOP EVALUATION TERMS OF REFERENCE, DECISION MAKING, PROVIDING
CONTEXTUAL OR TECHNICAL EXPERTISE AND/OR AIDING IN EVALUATION DATA COLLECTION,
REPORT PREPARATION, DISSEMINATION AND USE
Mertens Minneapolis MN March 2015
26
CONSTRAINTS & OPPORTUNITIES
• SOCIOPOLITICAL FACTORS – THE BRIDGE BETWEEN THE
STAKEHOLDERS AND THE CONSTRAINTS
• IDENTIFY AND EFFECTIVELY ADDRESS THE
AFFECTED/CONCERNED GROUPS
• INTERNAL INFO THAT IS GATHERED – WHO NEEDS ACCESS TO
THAT? HOW WILL IT BE PRESENTED?
• REALISTIC EXPECTATIONS – SET LIMITS – HOW CAN YOU
COMMUNICATE WITH THE STAKEHOLDERS, AND WHAT CAN
THE EVALUATION REALLY DO?
• HOW WILL YOU BUILD CONFIDENCE IN YOU? RESPECT AND
HONESTY WITH THEM WILL IMPROVE THEIR CONFIDENCE, SO
BE SURE TO SHOW THEM THOSE QUALITIES
Mertens Minneapolis MN March 2015
27
SPECIFIC CONSTRAINTS
• BE SURE TO INCLUDE TIME, MONEY, AND POLITICS
• IS THE PROGRAM OPEN TO BEING EVALUATED? ACCEPTING OF THE
EVALUATION?
• HOW SUPPORTIVE ARE THE LEADERS OF THE EVALUATION?
• WHAT IS THE HISTORY IN THE ORGANIZATION RELATED TO EVALUATION?
• CONSIDER THE SETTING – IS YOUR EVALUATION TO OCCUR IN ONE PLACE
OR SPREAD OUT (E.G., THE PT3 JOIN TOGETHER EVALUATION OR THE NAT’L
INFO CENTER – THESE ARE "VIRTUAL COMMUNITIES", WHERE ARE THEIR
“BOUNDARIES”?)
• ECONOMICS CAN PLAY A ROLE IN HOW SUPPORTIVE THE ORGANIZATION
IS OF THE EVALUATION
• SOCIAL ATMOSPHERE OF THE PROGRAM? OK TO CRITICIZE OR NOT?
Mertens Minneapolis MN March 2015
28
SELECTING A PROGRAM (PROJECT):
THINGS TO THINK ABOUT …
 REPRESENTATIVES OF MARGINALIZED GROUPS ARE INCLUDED IN THE
DECISIONS ABOUT WHICH PROGRAM TO EVALUATE AND HOW TO
EVALUATE IT.
 EVALUATORS AND INTENDED USERS AGREE ON THE GOALS,
OBJECTIVES, SIDE EFFECTS, AND PERFORMANCE CRITERIA
 PROGRAM GOALS AND OBJECTIVES ARE FOUND TO BE REALISTIC
GIVEN THE RESOURCES AVAILABLE
 RELEVANT INFORMATION ON PROGRAM PERFORMANCE IS
AVAILABLE
 DECISION-MAKERS ON THE POLICY OR OPERATING LEVEL ARE
WILLING TO CHANGE THE PROGRAM ON THE BASIS OF
EVALUATION INFORMATION.
29
Mertens Minneapolis MN March 2015
WHAT IS A LOGIC MODEL?
A LOGIC MODEL IS A DIAGRAM AND TEXT THAT
DESCRIBES/ ILLUSTRATES THE LOGICAL (CAUSAL)
RELATIONSHIPS AMONG PROGRAM ELEMENTS AND THE
PROBLEM TO BE SOLVED, THUS DEFINING MEASUREMENTS
OF SUCCESS.
Mertens Minneapolis MN March 2015
30
Mertens Minneapolis MN March 2015
31
EVERYDAY EXAMPLE
H
U
N
G
R
Y
Mertens Minneapolis MN March 2015
Get food
Eat food
Feel better
32
LEVEL I LOGIC MODEL
RESOURCES/INPUTS
What you need
to implement
your program!
Mertens Minneapolis MN March 2015
YOUR
PROGRAM
RESULTS/
IMPACT
What you do
to achieve
your longterm aims!
Why you
are in
business!
33
Every day logic model –
Family Vacation
Family Members
Drive to state park
Budget
Set up camp
Car
Camping
Equipment
INPUTS
Mertens Minneapolis MN March 2015
Cook, play, talk,
laugh, hike
OUTPUTS
Family members
learn about each
other; family
bonds; family has
a good time
OUTCOMES
34
THE EVALUAND AND THE LOGIC MODEL
1. RESOURCES AND/OR BARRIERS, WHICH POTENTIALLY ENABLE OR LIMIT PROGRAM
EFFECTIVENESS.
ENABLING PROTECTIVE FACTORS OR RESOURCES MAY INCLUDE FUNDING, EXISTING
ORGANIZATIONS, POTENTIAL COLLABORATING PARTNERS, EXISTING
ORGANIZATIONAL OR INTERPERSONAL NETWORKS, STAFF AND VOLUNTEERS,
TIME, FACILITIES, EQUIPMENT, AND SUPPLIES. LIMITING RISK FACTORS OR
BARRIERS MIGHT INCLUDE SUCH THINGS AS ATTITUDES, LACK OF RESOURCES,
POLICIES, LAWS, REGULATIONS, AND GEOGRAPHY.
2. ACTIVITIES ARE THE PROCESSES, TECHNIQUES, TOOLS, EVENTS, TECHNOLOGY, AND
ACTIONS OF THE PLANNED PROGRAM.
THESE MAY INCLUDE PRODUCTS – PROMOTIONAL MATERIALS AND EDUCATIONAL
CURRICULA; SERVICES – EDUCATION AND TRAINING, COUNSELING, OR HEALTH
SCREENING; AND INFRASTRUCTURE – STRUCTURE, RELATIONSHIPS, AND
CAPACITY USED TO BRING ABOUT THE DESIRED RESULTS.
Mertens Minneapolis MN March 2015
35
EVALUAND/LOGIC MODEL CONTINUED
3. OUTPUTS ARE THE DIRECT RESULTS OF PROGRAM ACTIVITIES. THEY ARE USUALLY DESCRIBED IN TERMS
OF THE SIZE AND/OR SCOPE OF THE SERVICES AND PRODUCTS DELIVERED OR PRODUCED BY THE PROGRAM.
THEY INDICATE IF A PROGRAM WAS DELIVERED TO THE INTENDED AUDIENCES AT THE INTENDED
“DOSE.” A PROGRAM OUTPUT, FOR EXAMPLE, MIGHT BE THE NUMBER OF CLASSES TAUGHT, MEETINGS
HELD, OR MATERIALS PRODUCED AND DISTRIBUTED; PROGRAM PARTICIPATION RATES AND DEMOGRAPHY;
OR HOURS OF EACH TYPE OF SERVICE PROVIDED.
4. OUTCOMES ARE SPECIFIC CHANGES IN ATTITUDES, BEHAVIORS, KNOWLEDGE, SKILLS, STATUS, OR LEVEL OF
FUNCTIONING EXPECTED TO RESULT FROM PROGRAM ACTIVITIES AND WHICH ARE MOST OFTEN
EXPRESSED AT AN INDIVIDUAL LEVEL.
5. IMPACTS ARE ORGANIZATIONAL, COMMUNITY, AND/OR SYSTEM LEVEL CHANGES EXPECTED TO RESULT
FROM PROGRAM ACTIVITIES, WHICH MIGHT INCLUDE IMPROVED CONDITIONS, INCREASED CAPACITY,
AND/OR CHANGES IN THE POLICY ARENA.
Mertens Minneapolis MN March 2015
36
Mertens Minneapolis MN March 2015
37
Mertens Minneapolis MN March 2015
38
VERIFY LOGIC MODEL WITH STAKEHOLDERS
• IS THE LEVEL OF DETAIL SUFFICIENT TO CREATE UNDERSTANDINGS OF THE ELEMENTS AND
THEIR INTERRELATIONSHIPS?
• IS THE PROGRAM LOGIC COMPLETE? THAT IS, ARE ALL THE KEY ELEMENTS ACCOUNTED FOR?
• IS THE PROGRAM LOGIC THEORETICALLY SOUND? DO ALL THE ELEMENTS FIT TOGETHER
LOGICALLY? ARE THERE OTHER PLAUSIBLE PATHWAYS TO ACHIEVING THE PROGRAM
OUTCOMES?
• HAVE ALL THE RELEVANT EXTERNAL CONTEXTUAL FACTORS BEEN IDENTIFIED AND THEIR
POTENTIAL INFLUENCES DESCRIBED?
Mertens Minneapolis MN March 2015
39
CHALLENGING THEORIES OF CHANGE
• THEORIES OF CHANGE NEED TO REFLECT:
• DIVERSITY
• POWER DIFFERENTIALS
• THE REAL ON-THE-GROUND EXPERIENCE OF INTENDED STAFF AND
PARTICIPANTS
• SOCIAL OR PSYCHOLOGICAL THEORIES ABOUT BEHAVIOR CHANGE
Mertens Minneapolis MN March 2015
40
EXAMPLE: HIV/AIDS PREVENTION IN
BOTSWANA
41
BOTSWANA YOUTH:
ADDRESSING POWER
INEQUITIES IN THE
FIGHT AGAINST
HIV/AIDS USING A
TRANSFORMATIVE
LENS
Mertens Minneapolis MN March 2015
EVALUATION QUESTIONS FOR GAINING
INSIGHT/DETERMINING NEEDS & ASSETS
• NEEDS ASSESSMENT TO DETERMINE THE NEEDS FOR THE
TARGETED STAKEHOLDERS –
• WHAT IS THE CURRENT STATUS; WHAT NEEDS ARE UNMET OR
INADEQUATELY MET?
• WHAT WOULD BE HELPFUL?
• WHAT RESOURCES ARE AVAILABLE?
• WHAT ACTIVITIES SEEM MOST LIKELY TO HELP REACH THE EVALUAND’S
GOALS?
Mertens Minneapolis MN March 2015
42
EVALUATION PURPOSE: IMPROVEMENT OR
CHANGE PRACTICES – IMPLEMENTATION
QUESTIONS
• WHAT ARE THE CRITICAL COMPONENTS/ACTIVITIES OF THIS PROJECT
(BOTH EXPLICIT AND IMPLICIT)?
• HOW DO THESE COMPONENTS CONNECT TO THE GOALS AND INTENDED
OUTCOMES FOR THIS PROJECT?
• WHAT ASPECTS OF THE IMPLEMENTATION PROCESS ARE FACILITATING
SUCCESS OR ACTING AS STUMBLING BLOCKS FOR THE PROJECT?
• HOW IS THE PROGRAM BEING IMPLEMENTED AND HOW DOES THAT
COMPARE TO THE INITIAL PLAN FOR IMPLEMENTATION?
Mertens Minneapolis MN March 2015
43
EVALUATION PURPOSE – ASSESS PROGRAM EFFECTIVENESS –
IMPACT EVALUATION QUESTIONS
• TO WHAT EXTENT WERE THE OBJECTIVES ACHIEVED OR LIKELY TO BE
ACHIEVED?
• WHAT WERE THE MAJOR FACTORS INFLUENCING THE ACHIEVEMENT OR
NON-ACHIEVEMENT OF THE OBJECTIVES?
• WHAT HAS HAPPENED AS A RESULT OF THE PROJECT OR PROGRAM?
• WHAT REAL DIFFERENCE HAS THE ACTIVITY MADE TO THE BENEFICIARIES?
• HOW MANY PEOPLE WERE AFFECTED?
(IMPACT EVALUATION IS USUALLY INTERPRETED AS LONGER
TERM EFFECTS OF THE INTERVENTION.)
Mertens Minneapolis MN March 2015
44
EVALUATION PURPOSE – HUMAN RIGHTS & SOCIAL JUSTICE
–GENDER ANALYSIS QUESTIONS
• WHO WILL BENEFIT AND WHO WILL LOSE FROM THIS PROJECT IN TERMS
OF GENDER? SHORT-TERM AND LONG-TERM?
• TO WHAT EXTENT WERE WOMEN INVOLVED IN THE DEVELOPMENT OF
THE PROGRAM?
• WHICH ORGANIZATIONS FROM GOVERNMENT AND CIVIL SOCIETY WERE
INCLUDED?
• HOW DOES THIS PROJECT CHALLENGE EXISTING GENDER DIVISIONS IN
TERMS OF LABOR, TASKS, RESPONSIBILITIES, OPPORTUNITIES, ACCESS TO
AND CONTROL OVER RESOURCES? WHAT ARE THE RISKS OF BACKLASH IF
CHANGES ARE MADE?
Mertens Minneapolis MN March 2015
45
TRANSFORMATIVE EVALUATION: RIGOR
• EMPHASIZES HUMAN RIGHTS AND SOCIAL JUSTICE
• ANALYSES ASYMMETRIC POWER RELATIONS
• ADVOCATES CULTURALLY COMPETENT RELATIONS BETWEEN THE
EVALUATOR AND COMMUNITY MEMBERS
• EMPLOYS CULTURALLY APPROPRIATE MIXED METHODS TIED TO SOCIAL
ACTION
• APPLIES FEMINIST THEORY, CRITICAL RACE THEORY, POSTCOLONIAL AND
INDIGENOUS THEORIES
Mertens Minneapolis MN March 2015
Mertens (2009) Transformative Research and Evaluation. The Guilford Press.
46
Concurrent Design
Quantitative and Qualitative occur more or less simultaneously
Quantitative
Qualitative
Sequential Design: Quantitative Followed by Qualitative
OR
Sequential Design: Qualitative Followed by Quantitative
Qualitative
Quantitative
47
Mertens Minneapolis MN March 2015
Mertens, 2009, TRE, p. 167
TRANSFORMATIVE MIXED METHODS DESIGN
Stage 1
Qual
Assemble
team; read
documents;
engage in
dialogues
Mertens Minneapolis MN March 2015
Stage 2
Concurrent
Preliminary
studies: youth,
older men
Stage 3
Sequential
Stage 4
Concurrent
Pilot intervention:
Observations,
Interviews, Surveys
Process eval
Demographic
information;
Surveys;
Incidence data
Pretest:
Knowledge,
Attitude,
Behavior;
Post
tests:
Quant
Qual;
Behavior
& Policy
Change;
Transfer
To other
contexts
48
TRANSFORMATIVE EVALUATION: RIGOR
• EMPHASIZES HUMAN RIGHTS AND SOCIAL JUSTICE
• ANALYSES ASYMMETRIC POWER RELATIONS
• ADVOCATES CULTURALLY COMPETENT RELATIONS BETWEEN THE
EVALUATOR AND COMMUNITY MEMBERS
• EMPLOYS CULTURALLY APPROPRIATE MIXED METHODS TIED TO SOCIAL
ACTION
• APPLIES FEMINIST THEORY, CRITICAL RACE THEORY, POSTCOLONIAL AND
INDIGENOUS THEORIES
Mertens Minneapolis MN March 2015
Mertens (2009) Transformative Research and Evaluation. The Guilford Press.
49
UTILIZATION
• THE VALUE OF SHARING INFORMATION THROUGHOUT THE COURSE OF THE
RESEARCH OR EVALUATION STUDY IS EMPHASIZED TO FACILITATE MIDCOURSE
CORRECTIONS IF AN INTERVENTION IS NOT MOVING TOWARD THE DESIRED
GOAL.
• PLANNING FOR UTILIZATION IS ESSENTIAL DURING THE INITIAL DESIGN OF THE
STUDY; THE TOPIC OF THE STUDY MUST BE PRESENTED TO PARTICIPANTS IN ORDER
TO ENSURE THAT THE DATA ARE GATHERED AND DISSEMINATED IN A WAY THAT
THEY CAN BE USED TO ACHIEVE THE GOALS OF SOCIAL CHANGE AND SOCIAL
JUSTICE.
• POLICY ANALYSIS AND ADVOCACY ARE AVENUES TO SOCIAL CHANGE FROM THE
PERSPECTIVE OF GRASS-ROOTS ORGANIZATIONS.
Mertens Minneapolis MN March 2015
50
QUESTIONS: DATA REPORTING & USE
• WHO WOULD YOU INCLUDE IN THE PREPARATION AND DISSEMINATION OF
FINDINGS?
• HOW WOULD YOU INCLUDE MEMBERS OF THE COMMUNITY IN THIS PROCESS?
• WHAT CHALLENGES MIGHT YOU ANTICIPATE ENCOUNTERING AT THIS STAGE OF
YOUR STUDY?
• HOW WOULD YOU DEAL WITH POWER DIFFERENCES AMONG THOSE INVOLVED IN
THESE ACTIVITIES?
• WHAT IS YOUR THINKING ABOUT OWNERSHIP OF THE DATA AND HOW WOULD YOU
HANDLE THIS ISSUE?
• WHAT ASPECTS OF THE TRANSFORMATIVE PARADIGM ARE MOST SALIENT AT THIS
POINT IN THE STUDY?
Mertens Minneapolis MN March 2015
51
REPORTING FORMATS
• WRITTEN FORMATS: VOICE AND PRIVILEGE
• FOCUS GROUPS & INTERVIEWS
• VISUAL PRESENTATIONS
• DRAWING PICTURES
• PHOTOS, VIDEOS, SLIDE SHOWS
• ETHNODRAMA
• WEB-BASED
• COMMUNITY-BASED
Mertens Minneapolis MN March 2015
52
CONCLUSIONS
• COURAGE
• MIXED METHODS: CAPTURE COMPLEXITY, ADDRESS
SOCIAL JUSTICE, AND ETHICS
• TEAMS
• CAPACITY DEVELOPMENT
Mertens Minneapolis MN March 2015
53
AGENDA FOR ACTION
HOW WILL YOU MODIFY YOUR OWN
EVALUATION APPROACH TO
INCORPORATE IDEAS FROM THIS
PRESENTATION?
Mertens Minneapolis MN March 2015
54
RESOURCES
• MERTENS, D. M. & WILSON, A. (2012).
PROGRAM EVALUATION THEORY AND
PRACTICE: A COMPREHENSIVE GUIDE. NY:
GUILFORD.
• MERTENS, D. M. (2015). RESEARCH AND
EVALUATION IN EDUCATION AND
PSYCHOLOGY: INTEGRATING DIVERSITY
WITH QUAL, QUANT AND MIXED
METHODS. 4TH ED. THOUSAND OAKS, CA:
SAGE.
• MERTENS, D. M. (2009). TRANSFORMATIVE
RESEARCH & EVALUATION. NY: GUILFORD.
• MERTENS, D. M. & GINSBERG, P.
(2009).(EDS.) HANDBOOK OF SOCIAL
RESEARCH ETHICS. THOUSAND OAKS, CA:
SAGE.
• AMERICAN EVALUATION ASSOCIATION
(2011). PUBLIC STATEMENT ON CULTURAL
COMPETENCE IN EVALUATION. AEA.
Mertens Minneapolis MN March 2015
55
CONTACT INFORMATION
DONNA M. MERTENS, PHD
INDEPENDENT CONSULTANT
DONNA.MERTENS@GALLAUDET.EDU
Mertens Minneapolis MN March 2015
56