Pre-meeting: Outcomes Working Group
Session leader: Frances Sinha, Director EDA Rural Systems (India)
Introduction

Objectives of the day:
o Explain the purpose and the process of the OWG
o Help develop some key themes
o Contribute to the plan for the OWG: development of guidelines, identify good/bad
examples, resources, future webinars
o Overall drawing on experience of people in the room, engagement for “the last mile in
SPM.”
 Frances noted that investors interested in the OWG. They’re having a parallel meeting right
now, but Frances will go in the afternoon to discuss the OWG with the investors.
 Purpose of OWG: To develop practical guidelines for credible measurement and reporting of
outcomes, drawing on experience with different approaches and tools.
o The group was very purposeful about this language:
 Measuring AND reporting is important.
 Have to be practical, specific
 Want guidelines to be used by practitioners, investors, TA providers, networks.
 Not a randomized control trial group, but the group recognizes that RCTs can be useful in
certain situations with certain resources
 Audience members have several comments:
o Calum Scott (Opportunity International) noted that it is also important that MFIs use the
data to change when something is not working
o Anton Simanowitz, noted that management decisions have to be a main point of this.
What are the very direct, simple ways we could have MFIs integrate outcomes data into
their management system?
o Debbie Watkins (Fern Software) – Important to match quantitative data with
qualitative. Qualitative can help explain patterns and exceptions.
 OWG so far:
o At 2014 SPTF Annual Meeting in Dakar, had plenary session where we brought
together experiences regarding outcomes.
o Then, started the OWG in Oct 2014
o Have held three webinars, which are repeated to make sure that it works for
stakeholders in various time zones. Working briefs:
 Lessons (from Dakar meeting)
 Mapping out the issues. Looking at the starting point.
 Theory of change. Link between what we want to do and the outcomes we want
to achieve. What do we do and what are the intended steps we take to get to those
outcomes?
 Themes and indicators. Instead of a huge laundry list, maybe some relate to
household and well-being, some relate to women’s empowerment, employment.
What are some of the indicators people are already using that we could learn from
and apply?
 Engaging with key players:
Microfinance Genome Project
Microfinance CEO Working Group (MCWG). Calum noted that MCWG is focused on
collecting data that will be useful to local partners, which is dependent on being able to
get data from the MFI. Then, the indicators really should be indicators for further
investigation. The data won’t tell you why there’s a problem, but it could tell you
there’s a problem. There also could be intermediate indicators that could be used to
identify where outcomes are likely to be better for certain areas.
o Freedom from Hunger. We identify indicators about health outcomes, resilience
o Action Group of e-MFP
OWG – our approach
o All methods have their uses. Which ones work and how do they work?
o We want to learn from the good examples. What are the indicators that have been
used? Framing of questions, methods, quality control, analysis (getting beyond the
average, disaggregating) and reporting.
o Use of findings. Monitoring results (investors/funder) and improving services
 Bobbi Gray noted that it’s not just the use but the credibility of the data, the source
of the data. The group can provide guidelines on looking at that critically. It’s not
just about using data for management decisions but being critical about the data
we are collecting so it’s an accurate depiction.
 Calum added that it is important to generate buy-in from institution
 A participant noted that framing of questions is important. Unless indicators are
clear and simple, training staff to collect this data will be a big challenge. Frances
noted that there are several methods for collecting data, such as using field staff,
internal research department and external consultant.
 Debbie noted that choice of words is important. Some people don’t consider
savings as savings. They consider it money for children’s education, etc.
Agenda for the day:
o Until 10:40 a.m. – Outcome themes, indicators and theory of change (group work)
o 11 a.m. to noon – developing the guidelines
 Link with e-MFP
 Our list of issues (credibility checklist)
 Suggestions, ideas for developing the guidelines
o Noon to 12:30 p.m. – Planning next webinars
 We’ve planned on every other month or every six weeks. What would be good
topics to discuss?
o
o


Outcome Themes, Indicators and Having a Theory of Change
Speaker, Margaret Richards, Research Director of the Microfinance Impact
Genome Project.

The Genome Project is asking, “What are the specific factors in what we’re doing that drive
outcomes?”
o Using evaluation as a way of improving vs proving. MFIs have to prove and report, but
we also want them to be trying to improve.
o Taking forward webinars 3&4
 Identifying broad outcome areas (themes) and relevant, practical indicators
 Challenge with terminology. What’s an outcome? What’s an indicator? What’s a
theme? What’s a question? In practice, the lines get blurry.
 How do institutions move from doing the research to getting the outcomes?
Thinking about a Theory of Change
 Effects of different inputs – services
 Time for change – near term, longer term
Microfinance Impact Genome Project
o What are the primary outcomes we want to achieve and what does the research tell us
about it? Taking world of research on outcomes, aggregating it all into when dataset
and then trying to find patterns. We can learn more collectively.
Process:
o Codify evidence base, creating bibliography
 Digitize all RCTs, quasi-experimental studies, unpublished research, etc.
 Code evidence by level of rigor, associated outcomes and enter into relational
database.
 Look at each individual study and pull out what happened. What was the
institution? What information is provided about the institution, in terms of its
size, its clients, its products and services?
 It’s not just about a certain loan leading to a certain outcome, but what were
the things about it that drove that outcome? If we aggregate examples, what
can we learn about management of institutions? What are the programmatic
factors that lead to a certain outcome?
 Developing practitioner survey, which can help confirm or contradict what is
found in the literature.
o Standardize data – derive efficacy factors.
 Conduct meta-analysis
 Standardize outcomes
 Derive, score and weight efficacy factors
 Calculate effect sizes
o Analyze programs – derive benchmark data
 Develop algorithm for calculating efficacy
 Integrate practitioner feedback
 Create program scorecards
 Determine benchmarks
 Pilot test with practitioners
o End goal: tool that MFIs can use to improve what they’re doing to achieve certain
outcomes.
MGP is one element of a larger Genome Project that analyzes different development sectors.
Microfinance is the first one.
MGP is working a few particular outcomes that are widely applicable. We developed the list
in a few different ways:
o In 2013, worked with MCWG to do mini-version of the research. Focused mainly on
credit and looked at about 70 studies
 Focus on credit
 Multiple methodologies
 Included few meta-analyses
o Then, expanded it to MGP study, with review of 650+ studies.
 Includes credit, savings, remittances, insurance
 Includes many meta-analyses
 A lot of looking at income or income proxies (consumption, expenditure
indicators), savings, health, women’s empowerment
o




Keywords per outcome
 Income/poverty: income, consumption, expenditure, poverty, asset
 Food security: food
 Resilience: resilience, shock, smooth, saving
 Women’s empowerment: empowerment, decision, women, girls
 Education: education, school
 Business development: business, enterprise
 Agriculture development: farm, agriculture, agricultural
o Comparison of MCWG review to broader GP review
 Results are pretty similar. Income poverty and resilience at top of both, health also
high for both. It helps look at what people are interested in, and what kinds of
things we want to focus on.
Outcomes Framework
o How can we prioritize? Good to be intentional. This framework to understand where
MFIs’ products and services can have the most impact on clients’ lives
 Access to financial tools and training (Increase access to financial products and
services, how many people have taken loans, engaged in savings groups) 
 Skill building and investment (develop financial, business and/or life skills,
increase social capital, increase business investment) 
 Financial growth, stability & security (business creation/growth, increase
household economic well-being, increase household resilience) 
 Improved household well-being (food secure, healthier, more access to education)
 Target outcomes
 Economic well-being of poor households
o Definition: Substantial improvements in well-being that can be
understood through direct measures of income or proxy measures of
expenditure, consumption, poverty indices or food security
o Trying to decide if we want to look at economic well-being for poor
households or all households;
 Build Resilience
o Definition: Substantial decrease in use of severe or moderately severe
coping strategies or successful use of savings, insurance and other
strategies to handle unforeseen events, financial and otherwise.
o Savings, insurance – but that’s not an outcome. That’s a tool.
 Business and enterprise growth
o Definition: Substantial increase in business turnover, profits and/or
sales, or time dedicated to owned enterprises
o Microbusiness, not SME.
 Employment creation.
o Definition: Creation of employment for adults (family and non-family
members) or self-employment
o Has community benefit as well
 Women’s Economic Empowerment. Financial decision making,
contribution to households
 Tier 2 indicators (important, but less relevant to larger group of
institutions):
o Increased financial capability
o Economic empowerment for other disadvantaged populations (elderly,
disabled)
o

o
o
o
Increased food security,
Increase health (outcomes, access or skills?)
Increase children’s/youth access to education
Small Group Work:


Four groups: resilience, women’s empowerment, household well-being, enterprise growth.
Each received a list of outcome themes. Some work well, some don’t.
o STEP 1: Groups to consider – select, add – the most relevant and practical indicators.
 What is good indicator? Group brainstorm:
 Measurable
 Specific
 Clear
 Relevant
o Type of change
o Context
o Use
 Objective/subjective, opinions
 Salience
 Adaptability
o STEP 2: Place them on flip charts within a Theory of Change framework
 Short term, long term – how long?
 Add the inputs, services that lead to these changes
Small Group – Report Back
o Group 1: Resilience
 Challenge: defining resilience. It’s difficult to define and no agreed upon decision.
Hard to speak same language.
 Concept incorporates many aspect of life so hard to narrow down indicators.
Not one or two that will really capture what we mean by resilience
 Theory of change: talking about the different products involved led to a discussion
of why those products don’t work sometimes (access, speed of access, stigma
about certain products)
 Two main challenges
 Not sure how to bundle products together to effectively make client more
resilient
 Quality of product. Not enough in TOC to say “Have an insurance product.”
(vary in quality, type)
 Inputs
 Insurance
o Death/funeral, credit/health, agricultural
o Amounts, payout
 Savings
o Overdraft procedure
o How easy to get out
o Pre-approve
o Commitment to save
 Credit
o Leniency/restructuring loans
o Emergency loans (fast access, quick approval, grace interest)


o
Cash transfers and “loans,” other
Trainings
o Importance of anticipating shocks
o Create demand
 Remittances
 Short-term outcomes
 % who rely on diversified income streams, # of people involved in IGAs
(Average # of reported IGAs)
 % who have had to sell (large assets, livestock) to cover emergency expenses
 Average savings, % that report having savings (informal and formal,
nonformal)
 Social capital
o Number of groups one belongs to
o % that borrow from family and friends
o Send children to eat at family’s household
o Feel supported
 % that report pulling children out of school due to shock
 % that forgo medical treatment due to cost
 % that report taking on labor jobs (hard labor)
 % that report migrating
 % that feel hopeful for the future (hopeful for what?)
o Depends..measurement for security for the future
 Long-term outcomes
 % who report feeling “resilient” to shocks
 % food secure (short and long)
 % consuming a diverse diet (based on a dietary diversity score)
 % that are “poor” based on a standardized measure of household of poverty
(PPI, PAT, etc.)
Group 2: Women’s empowerment
 Challenge: in defining, context is important. Since empowerment can be broken up
into categories of other outcomes (health, education, etc), the contextual issues
matter much more. Maybe we need multiple TOCs for what women’s
empowerment means in different communities.
 TOC for women’s empowerment: Giving women financial services  using
loan to increase income  more decision  consumption to education
 Which are the outcomes we can focus on as an industry? What is the component of
having a loan, being part of a savings group that contributes to that outcome?
 Inputs:
 Credit (C)
 Savings (S)
 Insurance (I)
 Training (T) (Men, women)
 Short-term Outcomes
 Access to education
 Ability to spend as want/need
 Use loan for own economic activity
 Savings goals
 Participate in financial education programs
o
o
 Change in economic well-being (income/assets)
 Long-term outcomes
 Children’s education (C, S)
 Girls enrolled in school
 Likelihood of further education
 Decrease in domestic violence (C, S)
 Health/nutrition (sustained) (S, I)
 Planned fertility
 Decision-making (C, S, I, T)
 Roles in the community/positions in society (C, S, T)
 Frequency of meetings
 Intra-household dynamics (C, S, I, T)
 Decisions on financial transactions
 Last household decision on purchase
Group 3: Household well-being
 Challenges:
 It’s important to understand the perspective of the program. Rural, urban,
both? Affects selection and definition of indicators.
 How you measure -- Two-year vs one-year loan cycle – affects types of
measurement
 Inputs
 Smaller assets
 Use key assets (toilet, water access)
 Access to health services
 Outcomes
 Children attending school (girls/boys)
 Key assets (ownership/value)
 Food security Current food situation (% eat particular food items, had 2/3
meals per day in previous week)
 Housing quality
 Have land ownership
 “Household pool” income (Women’s income on back)
 Source of income
 Per capita household income (Per capita household
consumption/expenditure)
 Food security (Maintenance quality and quantity, are you worried?)
 Self-sufficiency (Grow/buy own food, don’t borrow)
 PPI – Poverty likelihood (% below and above specified line)
Group 4: Enterprise growth and employment
 A bit more concrete, self-explanatory than some of the other themes.
 Need for segmentation (shopkeepers vs rural)
 Inputs
 Appropriate loan size (context and business specific) and terms
 Line of credit/cash advance option
 Biz skills training (focused on behaviors and practices, not just info)
 Access to market support services (agriculture: info on value chain
integration)
 Outcomes (moving along spectrum from short-term to long-term)











New own enterprise
How often do you run out of money before your next income
arrives?/Financial stability  line of credit/cash advance options
“Business savvy” eg: separation of accounts and hard to measure  biz skills
training
Investment in equipment (agriculture especially)
Formality/degrees of stabilization (eg: paying taxes, registering businesses)
Increase in sales/inventory turnover
Whether a biz owner pays regular wage to herself and others (leads to profit
measurement)
Increase in number of family members working (men/women)
Market differentiation/specialization/brand
Hired labor
Increase in profits
Developing Practical Guidelines
Speaker Daniel Rozas, e-MFP

e-MFP is interested and has some funding. e-MFP – network of microfinance organizations
based in Europe (Investors, rating agencies, DFIs)
o Objective – Create guidelines document for FSPs on the end-to-end process of
operationalizing outcomes measurement, analysis and reporting
o Motivation. Fills a gap, coincides with increased push to collect outcomes data
o Structure. e-MFP Action Group in collaboration with SPTF, funds to pay a (Europebased) consultant. Consultant would take in feedback from OWG and work with the
Action Group to create a guidelines document aimed for publication in middle of last
year. Draw upon what OWG is doing. Not exclusive, if others have ideas.
o Timeline. June 2015 create Europe-based Action Group; meetings in Nov. 2015 and
Spring 2016; one-on-one interviews periodically; expert review spring/summer 2016.
Finalize and publish by Nov. 2016
o How would the document be used? Daniel said this is up to investors themselves. eMFP has relatively little say of what that document looks like. The shape of the
document is going to be determined by Action Group. SPTF will be a member of the
group. Probably not likely to be used for investment decisions.
 Calum: Opportunity Intl gives members best policies they would hope to
implement. Most partners are looking for investment. Key that OWG work builds
momentum with other investors. MFIs should be rewarded for good work they’re
doing on outcomes measurement.
 Amelia Greenberg: Demand for this work came from the field. Guidelines meant
for someone who isn’t doing this yet but is interested – how do you generate buyin, pick indicators. SPTF does not want to start this from scratch. Look at what’s
out there and pull it together.
o Christophe Bochatay said that Triple Jump is interested in outcomes and getting MFIs
to measure them. Triple Jump started pilot two years ago on three MFIs in Latin
America, where a consultant analyzed how the MFIs gather data. Pilot found that MFIs
were gathering data but had difficulty using the data.
o Anton expressed concern about writing guidelines without experience. Writing
guidelines takes experience, but there are only a small number of highly competent,
o
o
research-based MFIs. Are we to write guidelines for the 99% from the perspective of
the 1%? The document needs to be written for the mainstream.
 Daniel noted that many members of the Action Group have been involved with
implementing SPM. Idea is to aggregate those experiences.
 Anton – Small subset of MFIs that do SPM have done outcomes. Are we able to
take that small subset of a subset and make something to give to the mainstream?
 Margaret: but this is all we have. We need to use something.
 Anton: We’re in a good position to pull together how to understand outcomes in
microfinance. But, if we’re providing guidelines to MFis to be collecting, using and
reporting outcomes, that’s different.
Calum noted that what investors look for and what would be useful for MFIs is
different. The document needs to speak to both groups.
Jaclyn Berfond said the group needs to consider which data are MFIs expected to
collect. It’s unlikely that MFIs will be able to collect everything on a regular basis, so
there will need to be external collection. The document should take into account these
costs.
Speaker, Frances Sinha
 Credibility issues. When we get information, what do we make of it? How do we know the
information is credible? Whatever the methodology, what makes the data credible? How to
assess different types of reports (including RCTs)
o What we measure – What are reasonable expectations? What has been shown to be
practical to measure?
1. Are the indicators appropriate, clear and practical?
2. Is there a realistic conceptual framework – Theory of Change?
 Calum noted that you could start before that with a question on why we measure.
There is a difference between collecting data to report because the investor asks
for it and collecting data because you think it will help your organization.
 Frances noted that we can come up with a list of indicators, but we don’t yet have
a model for the Theory of Change.
 A participant said that it also helpful to note for whom you are measuring
outcomes. Are we looking at specific segment?
o How we measure – What are the options and standards of quality in data collection?
Applies to all options, quantitative and qualitative, in-house or commissioned.
3. In the overall design – what has been the technical expertise/guidance?
What are the limitations?
4. If there is a sample, are the sample details clearly referenced? Is the sample
adequate and representative?
5. Is the methodology clearly set out?
6. Have principles of lean research been applied?
7. Are there adequate quality checks in place as part of data collection?
8. If paper based survey/information, are there adequate quality checks in
place as part of data entry?
 Bobbi noted that who asks the questions is important.
 Be aware of tools’ limitations
o How we analyze – What types of analysis are most relevant and useful?
9. Analysis does not rely only on averages
10. Statistical tests
 How does this fit into performance measurement? How would you integrate th
How we report and use – How do we make reporting focused and relevant for use by
different stakeholders?
11. Concise, clear reporting and conclusions
12. Use of data
Drawing on credibility issues to develop specific, operational guidelines – how to measure
and report– with clear examples and process
Margaret: we should look at tools used in other industries. What’s the minimum you have to
do to get credible outcomes? We don’t want to give people a big book of complicated
guidelines. We need to make sure it’s useful at the implementation level.
Your suggestions
o Add to the content – any particular issues
o Good examples
o The opposite – what not to do
o Resources – organizations, people, names
o



Speaker Amelia Greenberg, SPTF Deputy Director
 For the FSPs interacting with this document, we need to propose something that is easy to
understand, affordable and relevant to business.
 Guidelines often have two broad statements: ensure guidelines and have quality control.
But how do I do that?
 Working group will also be doing one-on-one interviews with people who have been doing
this. How long did this takeH? how much does it cost? How did you integrate? How many
people are involved? How did you get board involved? What did you do when your staff
pushed back?
 That was original idea. Then brainstormed about what specifically we would talk about.
o What is the indicator selection process like?
o Adding questions to loan forms? Client satisfaction survey? Is it your loan officers that
are doing this?
 Plan for content development
o Topic to Address 1: Core decisions
 Measuring outcomes as part of SPM – setting out the basics
 Drivers – external reporting to funders/investors (regulators?); internal
reporting to Board and management: different purposes and implications
 Approach – practical, demonstrating what is doable; not prescriptive but
avoid pitfalls;
 Whose responsibility? - role of the Board, role of top management
 Include in strategic plan
 Consider the options – approach (routine monitoring and/or sample
research); in-house, commissioned; from baseline to endline
 Consider the costs - allocate funds in budget
 Whatever the choice - allocate responsibilities (existing staff/new staff
needed?)
 Set timeline for implementation - frequency
 Select indicators
 Mission alignment – target clients, Theory of Change
 Consider the ‘menu’
 Choose the right indicators (what criteria?)
o Topic to Address 2: Putting systems in place
 Map out process for gathering data (steps, responsibilities, timeline)
Identify system to store data
 Modify MIS?
 Invest in a new one?
 Design training program
 Who gets trained? In what order?
 Who leads the training?
 How long does it take?
 Curriculum
 Set up monitoring and quality control process
o Topic to Address 3: Analysis and reporting
 Analyze data for different purpose
 Who does analysis
 Types of analysis (examples)
 What data/analysis to share with funders, board, senior management, others
 Ideas for how data can/should inform decision making
o Throughout the guidelines: discuss common challenges and ways to address them
 Ex: choice of method; pressure to show positive results, staff resistance if adding
to their work; bringing in appropriate technical expertise
o Systems to follow if commissioning external research
o Pilot test
 Design, implementation
Suggestions/are we missing main areas for guidance?
o Decision tree of what methodology you may want to employ. Tool to help them decide
what kind of methodology
o Amelia: Maybe have a step zero – why am I doing this? What am I trying to get out of
this?
o Debbie: Once you’ve analyzed the data, what should you do with the findings to drive
future findings? If there are outliers, examining why that is?
o Bobbi thinks an infographic would be helpful
o Needs to be short but still needs the right information. Maybe not targeting 99% but
should definitely be initially targeting 20%.
Ideas for next webinars
o Does it work to commission external research?
o Conducting research internally
o The end-to-end process for internal monitoring
o The limits – and uses – of case studies


