Outcomes Working Group
Webinar 3: Indicators to measure
outcomes
Presenters: Bobbi Gray, Research and Evaluation
Specialist, Freedom from Hunger
Anne Hastings, Executive Director,
Microfinance CEO Working Group
Working Group Facilitator: Frances Sinha, Director EDA Rural
Systems (India) and board member of SPTF
4 February 2015
Agenda
• Introduction (5mins)
• Bobbi (FFH): criteria to develop indicators related to
health outcomes; application by 4 MFIs, lessons
(15-20mins)
• Anne (MCWG): developing a system for standardized
measurement, review of outcomes studies (69), selecting
themes and indicators for microfinance (15-20mins)
DISCUSSION
Recap
• Previous webinar : Theory of Change framework:
▫ Defining short-term/intermediary and long term changes
▫ Assumptions – links from different services/uses to changes
▫ Attrition factor - not all participants will stay the course
• Next logical step to identify relevant practical
indicators to measure changes - and the steps that
lead to changes:
•
•
•
•
Criteria for identifying practical, relevant indicators
Selection relevant to microfinance
What is the experience?
Can we build consensus on a ‘standard menu’ ?
Indicators
The basics
•
•
•
•
•
Specific, relevant - define objectives
Practical to measure - credible
Unambiguous - clear
Can compare, benchmark
Limited number – necessarily selective
Trade-offs in e.g. what is practical to measure
(household income or expenditure) and/or unambiguous
(women’s role in decision making)
Health Outcome Performance
Indicators Project
Theories of Change:
Improved Health
Access to and use of
financial services:
loans; insurance;
savings; payments;
health loans and savings
Access to and use of
health services:
education, provision of
curative and preventive
health services
Integrated health and
financial services:
direct provision and
linkages between
sectors
Increased income
Consumption
smoothing
Coping with shocks
Seek prompt
medical treatment
Seek preventive
health care
Improved health knowledge and seeking
prompt medical treatment and preventive
health care
Cross-sectoral
efficiency gains in
provision of financial
and health services to
poor populations
Improved
health
outcomes
Seek prompt
medical treatment
Seek preventive
health care
6
Choosing Health Indicators
Usability/
Reliability
Measurabl
e by an
FSP
Can be
reported in
client
survey
Can change
in shortterm
Addresses
relevant
measures
for FSPs
Cannot rely
on specific
interventio
ns to
change
outcomes
Be
applicable
for both
genders
Can be
benchmarke
d to other
data (MDGs,
regional
data, etc.)
Reliability
Likelihood of
inclusion
Usability
Criteria
Feasibility
PPI/PAT
Yes
Yes
Maybe
Yes
Yes
Yes
Yes
Yes
High
Food
security
index
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Maybe
High
Use of
preventive
health
services
Yes
Yes
Yes
Maybe
Maybe
Yes
Maybe
Maybe
High/
Moderate
Access to
safe
drinking
water
(MDG 7)
Yes
Yes
Maybe
Yes
Maybe
Yes
Yes
Maybe
High/
Moderate
7
Current Pilot Partners
Financial Service
Provider
Country
No of Clients being
served by FSP
No. of clients
participating in
health indicators
survey
ADRA
Peru
17,039
95
CARD
Philippines
1,828,052
472
ESAF
India
322,590
1,000
Equitas
India
1,344,361
250
8
Survey Adaptations
 Poverty measurement – Use of country-specific PPIs
 Food security and nutrition –added a focus on food items in
India to reflect stronger focus on nutrition
 Preventive health care – focuses on institutional births in India;
annual exams and Pap tests in Peru; various annual exams in the
Philippines and use of health insurance (PhilHealth)
 Curative Health Care: Questions same in all three countries –
forgoing medical treatment and purchase of medicines due to cost
 Water and Sanitation– Focuses on defecating in the open and
treating water to make it safer to drink in India; Open defecation,
water sources, and water treatment in Peru; Water sources and
treatment of water in the Philippines
 Attitudes: Only measured in Peru and Philippines, accesses levels of
confidence related to ability to cover future medical costs and seek
adequate medical care
9
Lessons Learned
Standardization of indicators may be difficult.
Proceed with caution in the interpretation of results.
Baseline values will be important to establish.
Baseline values with high levels of performance may not be useful to
track.
 The value of statistical analysis – ex. correlations between indicators
of interest with poverty – can help refine our “theory of change” as
well as determine which indicators may be the most useful to help us
understand changes in client outcomes.
 Who to track and for how long? This is a very important question to
answer as it influences which indicators will be the most useful.
 This process requires patience.




10
Accion
BRAC & BRAC International
CARE Access Africa
FINCA International
Freedom from Hunger
Grameen Foundation
Opportunity International
Pro Mujer
VisionFund International
Women’s World Banking
Collaborating on Outcomes
The Goal of Our Work on Outcomes
To develop, test and disseminate a common core
measurement and monitoring system designed to:






Be cost-effectively embedded within an MFI’s operations
Use industry-standard indicators and metrics (PPI)
Provide affiliated MFIs with actionable client outcome
data that can be tracked over time
Allow MFIs the flexibility to measure those changes in
client outcomes they wish to affect using the same
indicators for the same outcomes
Enhance the Working Group’s ability to demonstrate and
improve the benefits of Responsible Microfinance for the
clients being served
Contribute to the sector’s understanding of client
outcomes
The Proposed Method
1. Select the indicators to test
2. Target countries and the MFIs
3. Pilot the indicators




Social performance (SP) leads will work with their
partners to collect, analyze and report on the data
SP leads and MFI partners would convene to discuss
experiences, identify lessons learned and develop
recommendations for future use
SP leads will develop and publish a series of briefs by
outcome area with theory of change and recommended indicators
Package and disseminate the final indicators along
with supporting documentation
Lessons from the Working Group’s
Review of Outcomes
• Phase 1: Catalogued 69 different outcomes
research from all 8 members and their affiliates
▫ Wide variation in product studied, methodology,
indicators, quality of research, etc.
▫ Difficult to draw definitive conclusions when
comparing outcomes across programmes
Different methods
RESEARCH COMPONENTS
QUANT. SURVEY
44
QUAL. SURVEY
15
CONTROL GROUP
30
LIT REVIEW
27
KII
16
BANK RECORDS
16
FGD
15
OTHER DATA
1
ENDLINE
25
MIDLINE
4
BASELINE
24
0
10
20
30
40
50
Different themes
INDICATORS
ASSETS
EDUCATION
INCOME
WELL-BEING
HEALTH
BUSINESS (NON-FARMING)
EXPERIENCE WITH MF PROGRAMS
FIN LITERACY
POVERTY OUTREACH
POVERTY STATUS
EMPOWERMENT
SAVINGS
FOOD SECURITY
FARMING (BUSINESS)
PERSONAL VIEW OF EC'C STANDING
SHOCKS / COPING MECHS
NUTRITION
SOCIAL CAPITAL
CHILD GROWTH
6
31
28
11
30
29
16
7
4
14
17
23
25
15
10
14
10
10
3
0
5
10
15
20
25
30
35
A Closer Look at Education Studies
1. There was a big range of indicators being used
across and within the networks.
2. There was little consistency across institutions.
3. Most studies only looked at one dimension of
education.
Phase 2: Identifying Potential Indicators
for 7 Types of Client Outcomes
1.
2.
3.
4.
5.
6.
7.
Food security
Coping strategies/shocks
Economic poverty (income or financial status)
Health
Assets, housing and business
Social capital and empowerment
Child and youth education
Phase 2: Summary of Indicators
Food security
% of households that are food insecure (levels measured
vary by tool)
Coping strategies/shocks
% of households that had to [make a specific adjustment
or level of change] to cope with the effects of [household
or community shock]
Economic poverty (income or financial status)
% of households living above/below a given poverty line
Health
% of households with [level of access] to health care
[services/supplies]
Phase 2: Summary of Indicators
Assets, housing and business
% of households that were able to [purchase
assets/improve their homes] in the past year
Social capital and empowerment
% of [women] expressing confidence in making decisions
regarding [specific category]
% of clients who perceive [specify benefit or value] in
group participation
Child and youth education
% of children in the household who are regularly
attending school
% of households with increased ability to pay for school
fees
Discussion
1. Any questions/clarifications?
2. What is your experience in indicator selection?
3. Can we aim at standardisation - synchronisation?
4. Our working group: contributing to
guidelines for selection (and a menu) of
indicators to measure client outcomes?
Thank you
• For follow up, please contact: info@sptf.info,
francessinha@edarural.com
• Please note: presentations and recordings from
all Outcomes Working Group Meetings are being
posted to the SPTF website, working groups
page: http://sptf.info/sp-task-force/workinggroups