Developing a System for Evaluating
Small Microcredit Programs
Interim Research Report
January 10, 2012
York University Community Finance Project (YUCFP)
in partnership with
ACCESS Community Loan Fund
Brenda Spotton Visano, PhD and Neil Britto
Overview
Topic
Slides
1.
Project Objective…………………………………………………
2.
Process and Outcomes to Date……………………………
#3
#4
3.
Goals of Evaluation and Program Evaluation
Commitments of ACCESS.......................................
#5-7
4.
Comparison of Industry Options ………………….…….
#8-11
5.
The ACCESS Alternative: Proposed Program
Evaluation Indicators…….........................................
#12-17
6.
The Flow of Social Performance Information……….
#18
10 January 2012
2
Project Objective
• Develop a system of self-evaluation for small
volunteer microcredit programs in Canada that
– Has comprehensive coverage options spanning an
array of performance objectives
– Offers an evaluation of the program’s effectiveness,
efficiency, impact, and social performance
– Is informed by program evaluation industry standards
– Offers an independent and credible set of metrics for
volunteer organizations to use when reporting their
performance
– Is low cost, flexible, accessible, and easy to use
10 January 2012
3
Process and Outcomes to Date
When
What
Who
Review existing microcredit program evaluation instruments and
services
YUCFP
Fall 2010 - Winter 2011
Review existing literature on Social Performance Indicators
YUCFP
Summer 2011
Assess advantages and disadvantages of existing program
evaluation instruments and services in relation to Social
Performance Literature
YUCFP
Summer 2011
Proposal to Pilot the Development of a System to Evaluate Small
Microcredit Programs in Partnership with ACCESS Community
Capital Fund
YUCFP, ACCESS
Fall 2011
Review evaluation commitments and representations of ACCESS
Identify/clarify program evaluation objectives
ACCESS , YUCFP
Fall 2011
Identify Indicators to be included
ACCESS, YUCFP
Summer 2010
Winter 2012
Interim Research Report
Draft Program Evaluation Data Report
YUCFP, ACCESS
Winter 2012
Complete Survey Instruments for Data Collection and
Pilot System for Data Management
YUCFP, ACCESS
10 January 2012
4
Goals of Program Evaluation
•
Effectiveness
– for investigating whether a program has demonstrable effects on specific targeted outputs and
activities.
– Illustrative Question: “Is the program achieving the goals and objectives it was intended to
accomplish?”
•
Efficiency
– for assessing the cost-effectiveness of the program.
– Illustrative Questions: “Are the program’s activities being produced with appropriate use of
resources such as budget and staff time?”
•
Policies and Practices
– For assessing the integrity and effectiveness of program policies and practices.
– Illustrative Question: “How well does the program manage client information?”
•
Impact
– for investigating any impacts beyond the immediate outputs and activities.
– Illustrative Question “Do clients experience changes in personal income through microloans?”
10 January 2012
5
ACCESS’ Evaluation Commitments and Representations
Trillium Grant
Indicator
Source
Provided
to Board
Provided
to Trillium
Loan Portfolio Information
Volume of loans granted by each Chapter
Quality of loans at each chapter
Portfolio report
Portfolio report
Monthly
Monthly
Quarterly
Quarterly
Other Financial Data
Additions to Guarantee Fund
Financial statements
Monthly
Quarterly
Donations received
Financial statements
Monthly
Quarterly
Other Data
Chapters formed
Chapter Agreements executed
Loan Officer report
Loan Officer report
Monthly
Monthly
Quarterly
Quarterly
Chapters in dialogue with Access
Volunteers trained in lending, fundraising, other as required
Trainers identified at each chapter
Fund raising (investors and donors) formulated
Loan Officer report
Loan Officer report
Loan Officer report
Loan Officer report
Monthly
Monthly
Monthly
Monthly
Quarterly
Quarterly
Quarterly
Quarterly
Calls made on potential investors and donors
Fundraising events held
Articles published, other media exposure
Loan Officer report
Loan Officer report
Loan Officer report
Monthly
Monthly
Monthly
Quarterly
Quarterly
Quarterly
10 January 2012
6
ACCESS’ Evaluation Commitments and Representations
Presented by Alex Kjorven to Future Fund (Trillium) Evaluation Workshop July 19, 2010
as posted http://futurefundevaluation.wordpress.com/projects-evaluations/
Indicator
Current
Goal
Number of loans disbursed
15
30
Number of open houses
N/A
3
Number of articles published
N/A
2
Number of chapters formed and
operating
1
3
Number of active volunteers
35
50
$220,000
$300,000
$3,000
$10,000
Amount of guarantee fund
Amount of donations
Evaluation metrics currently under consideration to evaluate the effectiveness of loans disbursed and the social impact
of our lending program include:
1.Number of new businesses created or new lines of business initiated
2.Number of referrals for mentorship services (internally/externally)
3.Number of new employees hired by the business since receipt of loan
4.Financial metrics such as profits and total spending
5.Improvement to lenders credit rating as a reflection of their improved ability to access capital in the future
10 January 2012
7
Comparison of Industry Options – Operation
CERISE’s
SPI Tool
Cost to Acquire
as @
August 2011
$1,0000 - $4,000
w/ the use of a
supporting analyst.
Planet
Rating’s
Social
Rating
$4,000 - $9,000
Microfinanza
Microrate
Impact
Reporting &
Investment
Standards
(IRIS)
$17,000 to $20,000
$5,000 - $10,000
Freely available on
open access website
None
Number of
Consultants
Required/Provided
1 Provided
1 Required
2 to 3 Required
Unknown (Consultants
who provide financial
rating, also provide
social rating)
Timeline: On site
consultant visit
1 to 2 days
4 to 5 days
8 to 10 days
4 to 5 days
Not applicable
Timeline: Draft
report
N/A
1 to 2 months after
on-site visit
1 to 2 months after on-site
visit
2 to 4 weeks after onsite visit
Not applicable
Timeline: Final
report
1 to 5 months
after on-site visit
1 to 3.5 months after
on-site visit
2 to 3 months after on-site
visit
2 to 5 months after onsite visit
10 January 2012
Automatically
generated
8
Profile: Planet Rating:
Social Rating Product
Planet Rating, headquartered in Paris, France, is a specialized microfinance rating agency
offering evaluation and rating services to microfinance institutions (MFIs), using
the GIRAFE and the Social Performance methodologies. To date, Planet Rating has conducted
over 600 missions in more than 75 countries and continues to expand its international
presence. Planet Rating has been recognized by the Rating Fund I, the Rating Initiative, the
Moroccan APP and the Latin American Rating Fund II.
-From their website: http://www.planetrating.com/EN/
Relevant for small, volunteer microloan programs in Canada?
NO
… Costs are prohibitive
….designed for large formal microfinance organizations
…benchmark measures are unavailable for Canada
10 January 2012
9
Profile: Microfinanza
Social Rating Product
MicroFinanza Rating is a leading microfinance rating agency. Since 2000 we strive to produce independent and
qualitative information that promotes responsible flow of investment towards the microfinance industry. This is
achieved through provision of a comprehensive package of institutional and social performance assessment
services to microfinance institutions (MFIs), investors, regulators, donors and networks. MicroFinanza Rating
has performed about 600 assessments so far in over 60 countries worldwide.
- From their website: http://www.microfinanzarating.com/
Relevant for small, volunteer microloan programs in Canada?
NO
… Costs are prohibitive
….designed for large formal microfinance organizations
10 January 2012
10
Profile: Impact Reporting and Investment Standards
(IRIS)
Financial Service Sector Supplement
Impact Reporting & Investment Standards, or IRIS, is a common language for describing the
social and environmental performance of an organization. IRIS provides an independent and
credible set of metrics for organizations to use when reporting their impact. IRIS indicators
span an array of performance objectives and include specialized metrics for a range of sectors
including financial services, agriculture, and energy. Like financial accounting standards, IRIS
provides a basis for performance reporting and organizations need only report on applicable
metrics from the IRIS library
-From their website: http://iris.thegiin.org/
Relevant for small, volunteer microloan programs in Canada?
NO
… no scope to assess volunteer and capacity building contributions
….designed for large formal microfinance organizations
…bases of comparison irrelevant for Canadian context
10 January 2012
11
The ACCESS Alternative: Performance Indicators,
Reporting Format and Information Management
► The proposed social performance assessment includes assessments of both
efficiency and effectiveness.
► The proposed measures express greater consideration for the small size of
Canadian microcredit organizations.
► The social performance assessment requires significantly less resources to
complete.
► The proposed indicators avoid using any summary or composite score in the final
report.
► The proposed assessment rebalances and refocuses the set of indicators to better
reflect an urban setting.
► The proposed measures are intended for use in the Canadian context.
► The proposed social performance assessment is designed to produce information
valuable to internal stakeholders as well as external audiences.
► The provision of supports to help manage social performance information is
unique to the proposed assessment.
10 January 2012
12
Proposed Indicators - Effectiveness
Category
Indicator
Measure
Client Relations - Client
Targeting
Whether the loan portfolio indicates
success of the organization's mission to
target specific groups
Whether the groups mentioned in mission statement correspond
to groups in the loan portfolio: OS11 corresponds to OS13
Client Relations - Client
Targeting
Whether the organization successful in
targeting the groups that its mission
statement suggests.
Whether the groups targeted in the organization's outreach
efforts are the groups represented in the loan portfolio over the
last 12 months. (a) Yes(b) No
Operational
Performance - Loan
Delivery
Application success rate
The total number of loans granted divided by the total number of
microloan applications this calendar year.
Operational Performance
- Loan Delivery
Effective delivery of loan product
The number of microloans provided this calendar year.
Social and Client Impact
Gender diversity of loan recipients
The number of female clients as % of total during the reporting
period.
Social and Client Impact
Client Diversity
The number of clients that identify as: (a) Caucasian, (b) visible
minority, (c ) aboriginal, (d) differently abled, (e ) other
Personal changes identified by the Client
The personal changes that clients identify: (a) increase in business
skills, (b) increase in personal income, (c ) neither, (d) increase
both personal income and business skills, (e) other
Social and Client Impact
10 January 2012
13
Proposed Indicators – Effectiveness (Cont’d)
Category
Indicator
Measure
Social and Client Impact
Change in business performance since
the microloan
The approximate change in business (a) revenues since the client
received the microloan, (b ) profit since receiving the microloan; > $10K;
< $10K
Contribution to financial inclusion
The number of loan recipients who progress to (a) new bank account, (b)
a registered account, (c ) a credit card, (d) mutual fund products, (e) tax
free savings account, (f ) additional business loans,
(g ) line of credit (h) Other (please list below)
Social and Client Impact
Contribution to financial inclusion
The number of loan recipients in the current loan portfolio that identify
as (a) female, (b) aboriginal, (c) Caucasian, (d) visible minority), (e)
differently abled, (f) low income, (g) poor credit history
Social and Client Impact
Contribution to personal skills
development
The number of mentorship hours provided to clients
Social and Client Impact
Community connections
The number of partnerships with other organizations and the number
purchases from local businesses.
Social and Client Impact
Gender diversity of the organization
The number of females as % of total who are board volunteers, non
board volunteers or paid staff.
Social and Client Impact
Ethnic diversity of the organization
The number of paid staff, board volunteers and non-board volunteers
that identify as: (a) a visible minority, (b) caucasian, (c ) aboriginal
Skills improvement for organizational
members
The number of individuals who identify skill development in: (a) Time
Management, (b) Money Management, (c ) Human Resource
Management, (d ) Information Organization, Interpretation and
Maintenance, (e ) Team Skills, (f ) Customer Service, (g ) Leadership, (h )
Cultural Sensitivities, (i) Other
Social and Client Impact
Social and Client Impact
10 January 2012
14
Proposed Indicators - Efficiency
Category
Indicator
Measure
Operational Performance
- Loan Servicing
The total cost of providing skills
development and mentorship to
microloan recipients
Allocated salary of paid staff to loan delivery + (number of board
volunteer hours x minimum wage)
Operational Performance
- Loan Delivery
Efficiency of pre-loan activities
The number of non-board volunteer hours needed to deliver one
microloan: (total average monthly hours for pre loan activities + total
average monthly hours for post-loan activities) / average number of new
loans granted per month.
Operational Performance
- Loan Delivery
Cost of pre-loan delivery
Allocated salary of paid staff to loan delivery + (number of non-board
volunteer hours on loan delivery activities x minimum wage)
Operational Performance
- Loan Servicing
Efficiency of post-loan activities
The number of paid staff hours needed to deliver one microloan: (total
average monthly hours for pre loan activities + total average monthly
hours for post-loan activities) / average number of new loans granted per
month.
Operational Performance
- Loan Servicing
Cost of post- loan servicing
Allocated salary of paid staff to loan servicing + (number of non-board
volunteer hours on loan delivery x minimum wage)
Operational Performance
- Loan Servicing
Resource requirements to support the
activities of one microloan
The total time of 'other activities': (a) operations/management, (b)
outreach committee meetings, ( c) steering committee meetings, (d)
promotional efforts and (e ) other activities, divided by the average
number of new loans granted per month.
10 January 2012
15
Proposed Indicators – Policies and Practices
Category
Indicator
Measure
Client Relations - Client
Communication
Consideration for client
communication
Are clients understanding their rights, obligations and product terms
when (a) submitting the loan application, (b) signing the loan
agreement?
Client Relations - Client
Participation
Opportunities for client participation
Whether policies and/or current practices exist that enable client
participation in the organization, in addition to being a loan recipient (a)
Yes, (b) No
Client Relations - Client
Targeting
Consideration for targeting specific
client groups
Whether the organizations outreach efforts target (a) women, (b)
aboriginal, (c ) caucasian, (d) visible minorities, (e ) differently abled, (f )
low income, (g ) poor credit history (h ) other priority groups
Client Relations Communications
Consideration for client feedback
Whether the loan program have a process in place to allow clients to
provide feedback. (a) Yes, (b) No
Employee Relations Employment Terms
Provision of non-wage benefits
Whether the organization provides its employees and volunteers with
non-wage benefits. If so, please indicate (a) long term employment, (b)
health care benefits and (c ) training opportunities
Employee Relations Performance Appraisal
Consideration for social objectives as
part of staff and volunteer appraisal
Whether the MFI incorporate the fulfillment of social objectives into
performance appraisal of staff and volunteers
10 January 2012
16
Proposed Indicators – Policies and Practices (Cont’d)
Category
Indicator
Measure
Employee Relations Training
Consideration for social
performance training
Whether the organization provide social performance training If so, does it
provide training on (a) Payment Collection Practices, (b) Responding to Client
Needs, (c ) Social Performance Reporting, (d ) Client Communication, (e ) Gender
Sensitivity, (f ) Safeguarding Client Data, (g ) Over Indebtedness Prevention. Also,
who is the training provided to: (a) Non-board member volunteers, (b) Paid staff,
(c) Board Member Volunteers
Employee Relations Workplace Diversity
Consideration for workplace
diversity
Whether the organization has specific policies to ensure diversity in the
workplace: (a) Yes, (b) No
Governance Environment
Demonstrated concern of
environmental impact and
influence of the organization
A demonstration of environmental concern. (a) in the form of policy statements
addressing environmental impact, (b) tracking of environmental impacts, (c )
other
Governance - Ethical
Conduct
Existence of formal policies on
ethical conduct
Whether the organization has a formal code of ethical conduct for (a) volunteers,
(b) employees, (c ) clients
Formal policies addressing
governance activities.
Whether the MFI has formal policies for its governance activities. This could
include (a) regular Board of Directors meetings with at least three Board
members, (b) at least one Female member of the board, (c ) statements to
address potential conflicts of interest between members of the organization
Operational
Performance - Loan
Delivery
Flexibility of microloan products
The range of products provided by the organization: (a) Business start-up loans,
(b) Business Loans > $5,000, (c ) Training Loans, (d) Line of Credit, (e) Other
(please list below)
Employee Relations Workplace Diversity
Alignment between operational
staff and the communities the
organization serves.
Whether board members, volunteers and staff have residence or work in the
neighborhoods where the organization provides services
Governance - Social
Performance
10 January 2012
17
The Flow of Social Performance Information
Tool Used
Sequence of Tasks
Social Performance Indicators
(Effectiveness, Efficiency and Policy / Practice Indicators)
Organizational Survey
(Completed by one
director)
Non-Board Volunteer
and Staff Survey
(Completed by all
volunteers and staff)
Client Survey
(Completed by
all clients)
Spreadsheet Template
(Aggregates all survey information. Allows for periodic reporting
and storage of historical social performance information )
Social Performance Report
(Completed report includes an indicator scorecard, discussion on
historical social performance of the organization and provides
recommendations for potential improvements)
10 January 2012
Step One: Selection of Indicators
Selected social performance indicators are represented
in the three surveys in the form of a question.
Step Two: Survey Completion
The respective groups submit completed surveys to the
individual managing the social performance report
Step Three: Populating the Spreadsheet
Survey data are input into the spreadsheet template.
The spreadsheet populates the key performance
indicators from the survey data
Step Four: Report Completion
The social performance indicators from the spreadsheet
template provide the foundation of the social
performance report
18