Namibia looking forward:
Effectiveness of capacity development
support to local partners
Stakeholder data validation workshop  July 2, 2012  Windhoek
Why we are here today
• To share Pact Namibia’s internal
evaluation findings with stakeholders
• To get stakeholder validation of the
evaluation recommendations
Agenda
Time
Activity
9:30–10:45
Presentations
10:45–11:00
Tea break
11:00–11:30
Q&A
11:30–noon
Small group work: Discuss findings & review recommendations
Group 1. Capacity development impact
Group 2. Graduation to direct USAID funding
Group 3. Lessons learned for capacity development
organizations
Noon–12:45
Report back
12:45–1:00
Conclusions & next steps
1:00–2:00
Lunch
Background
• USAID Namibia began scaling up HIV programs
in 2003.
• In 2007 USAID awarded Pact a 5-year project
which included grants, organizational and
technical assistance to CSOs, predominantly in
areas of:
• HIV/AIDS prevention
• Care and support to orphans, HIV-vulnerable children
and home-based clients
• Strengthening capacity of the Ministry of Gender Equality
and Child Welfare
Background
• In 2010 – award changed to focus on two
objectives:
1.
Strengthening capacity of the Ministry of Gender
Equality and Child Welfare to effectively coordinate
activities relating to gender equality and orphans and
vulnerable children
2. Strengthening six civil-society organizations to be
eligible for U.S. direct funding while maintaining high
standards of existing services
Tools to strengthen civil society
• Organizational Development (OD)
Roadmaps measures CSO partner organizational
systems and structures objectively
• Basis for identifying organizational efficiency gaps and
prioritizing interventions
• Progression on a scale from 1 (nascent) to 5 (mature)
• Comprehensive Institutional Strengthening
Plans to identify, schedule and monitor all capacitybuilding activities
Tools to strengthen civil society
• Comprehensive Institutional Strengthening
Plans to identify, schedule and monitor all capacitybuilding activities
• Organizational Development (OD)
Roadmaps
• 10 capacity areas:
Monitoring &
evaluation
Organizational
sustainability
Grants &
compliance
Purpose &
HR management
planning
Governanc
Networking
e
Financial
management
Operations
Projects &
management
services
Organizational Performance Index
(OPI)
Scored on a scale of 1 to 4
on each of the following dimensions:
• Effectiveness.
Achieving outcomes, meeting standards
• Efficiency.
Delivering services, increasing reach
• Relevance.
Engaging target populations, promoting learning
• Sustainability.
Mobilizing resources, increasing legitimacy
Why do this evaluation?
1. Assess the sustainability of graduated partners’ capacity
development
2. Review the quality and quantity of services offered to
beneficiaries of former and current CSO partners
(partially done)
3. Assess the effectiveness of the Pact Namibia capacity
building assistance to the partners that have graduated
or plan to graduate
4. Draw lessons for other Pact programs that are facing
similar challenges of graduating partners
Evaluation questions
• To what extent was Pact Namibia’s
capacity development support effective?
• To what extent were partners ready for
and able to effectively manage direct
funding?
• To what extent did service quality and
coverage change over the last two years?
Process
3 phases:
1. Designed terms of reference and data
collection tools (virtual group)
2. Collected data (Pact staff from 5 country
offices, for neutrality)
3. Analyzed data, drafted and disseminated
report (current group of Pact staff)
Data collection tools
1. Partner survey (Mobenzi)
• 17 current & former partner organizations interviewed
• Quantitative & qualitative
2. Organizational Performance Index (OPI)
• 12 of 17 partners
3. Pact staff survey (Mobenzi)
• 4 core Pact Staff from each technical area
4. USAID tool
5. Historical OD Roadmap scores
Data analysis
• Quantitative data:
• Merged datasets and did selected frequencies with statistics and
cross-tabulations using SPSS and Excel.
• Qualitative data: thematic analysis.
Partner breakdown
OD Roadmap
support,
graduated to
direct funding
OD
Roadmap
support,
not
graduated
No OD
Roadmap
support,
graduated
No OD
Total
Roadmap
support,
not
graduated
Number of
orgs
3
(18%)
3
(18%)
2
(12%)
9
(53%)
17
Number of
partner
respondents
11
(23%)
12
(25%)
3
(6%)
22
(46%)
48
Study limitations
Sample size
• Only 17 organizations
• Limited number of respondents in some
organizations; some interviewed were previous staff
Time
• Could not conduct survey of beneficiaries
Presentation of the findings
• Capacity development impact
• Graduation to direct USAID Funding
• Lessons learned for capacity
development organizations
• Preliminary recommendations
Capacity development impact
Key areas of analysis
1. How did length of support influence partners’ OPI
scores and perceptions of usefulness of support?
2. How do Pact and partner perceptions of support
compare?
3. How do OD Roadmap supported partners differ from
other partners?
4. Which areas of Pact support did partners rate as most
useful and of highest quality?
5. What were the most significant changes resulting from
Pact’s support?
1. Partners scored longer support less
useful
Usefulness rating
5.0
4.5
4.0
3.5
Monitoring & evaluat
Strategic
Financial
Program thematic
3.0
1-3 years of
support
3+ years of
support
2. But Pact scored longer support more
useful
4.5
4.0
3.5
3.0
1-3 years
of support
5.0
M&E
Strategic
Financial
Program
thematic
3+ years
of support
Partners score longer
support less useful
Pact rating
Partner rating
5.0
4.5
4.0
3.5
Strategic
M&E
Program
thematic
Financial
3.0
1-3 years
of support
3+ years
of support
Pact scores longer support
more useful (except financial)
% of partners scoring
over 3
3. Longer support – higher OPI scores
80%
60%
40%
Program thematic
Monitoring & evaluat
Financial
Strategic
20%
0%
1-3 years of
support
3+ years of
support
OPI score above 3
4. OD Roadmap partners scored higher on
OPI
100%
67%
50%
17%
0%
OD Roadmap
Non-OD Roadmap
1a. But non-graduates more keen on financial
management
% of all partner staff
60%
50%
40%
30%
Non-graduated partners
20%
10%
0%
Technical
Support
Financial
management
support
M and E
reporting
support
Advocacy
Support
Strategic
planning
support
HR
Management
Other
5. Non-OD Roadmap partners rated Pact support more
useful
Usefulness rating
4.50
4.25
Non-OD Roadmap
4.00
3.75
OD Roadmap
3.50
Strategic
Support
Financial
Support
Program
Thematic
M&E
6. Partners rated M&E support highest
4.50
Usefulness rating
4.25
4.00
Quality rating
3.75
3.50
Program
Technical
Financial
Planning
Strategic
Planning
M&E
7. Most significant changes
M&E and Financial management the
“cornerstones of programs” (partner, Pact and
USAID responses)
• M&E
• Timely reporting
• Capturing the right and required data
• Use data for decision making
• Finance
• Proper management of organization finance & USAID funds
• Complete and accurate finance reports
• Proper budget management
7. Most significant changes
• Moved from no system to very good
system in all areas (partner responses)
• Buy-in from organizations (Pact
responses)
• How do we sustain the change?
• Experience handover and staff retention
• Resources
Summing up
• More years of support = lower usefulness rating, but
higher OPI scores
• Pact and partners see usefulness differently over the
duration of support
• OD Roadmap support = lower usefulness rating, but
higher OPI scores
• M&E support rated most useful and of highest quality
• Most significant change in M&E and financial
management
Graduation to direct USAID
funding
Key areas of analysis
1. Capacity-building support that partners
would like more of before graduation
2. Graduate vs. non-graduate performance
on OD Roadmap and OPI
3. Prerequisites for graduation
4. Key capacity-building support for
graduated partners
1. Partners want more technical support before
graduation
% of all partner staff
60%
50%
40%
30%
20%
10%
0%
Technical
Support
Financial
management
support
M and E
reporting
support
Advocacy
Support
Strategic
planning
support
HR
Management
Other
1b. And graduates far more interested in technical
support
% of all partner staff
60%
50%
Graduated partners
40%
30%
Non-graduated partners
20%
10%
0%
Technical
Support
Financial
management
support
M and E
reporting
support
Advocacy
Support
Strategic
planning
support
HR
Management
Other
2a. Graduates scored higher on OD
Roadmap
4
Graduates
3.5
3
Nongraduates
2.5
2
1.5
1
0
1
2
3
4
5
6
7
8
9
10
11
2b. But OPI not different
4
3.5
3
2.5
2
1.5
1
Graduates
Non-graduates
0
1
2
3
4
5
3. Graduation prerequisites from Pact and partners
1.
2.
3.
4.
Strategic plan
Governance
Strong financial systems and internal controls
Administrative systems, including human
resource management
5. Skilled staff and technical capacity
6. Strong monitoring and evaluation
4. Key post-graduation capacity
development
Priority
• Resource mobilization support
• Technical/Program support
As needed
• Monitoring and evaluation
• Database development
• Evaluations
• Routine data quality assessments
• Financial management
Summing up
1. Graduated partners would have liked more technical
support before graduation
2. Graduated partners averaged 3.4 on Roadmap, up from
2.5 for non-graduated partners (no difference on OPI)
3. Six graduation prerequisites: strategic plan, governance,
finance, admin, technical capacity, M&E
4. Key post-graduation needs: technical and resource
mobilization support
Lessons learned for capacity
development organizations
Key areas of analysis
1. Retention of systems after capacity building,
and impact of who decided to engage in it
2. OPI scores and trends across partner subgroups
3. Different capacity needs identified by partners
and donors
4. Challenges and advantages of graduating to
direct funding
1a. OD Roadmap-supported partners more likely
to retain systems
% of orgs still using
system
OD Roadmap Support
100%
80%
60%
Non-OD Roadmap Support
40%
20%
Strategic
Planning
Financial
Management
Program
Thematic
Area
Monitoring &
Evaluation
1b. Retention likelier if system is partnerdriven
Who decides to
acquire system
% retaining
system
N
Partner alone
95%
60
Both partner
and Pact
83%
87
Pact alone
75%
24
1c. Retention likelier if system is partnerdriven
Strategic
Monitoring &
evaluation
% retaining system
100%
80%
Financial
60%
Program
thematic
40%
20%
Partner's
Decision
Both Decision
Pact's
Decision
2. OPI scores
• Average OPI scores show lower resource
mobilization but high social capital
• Graduated partners generally score higher
• Current partners score low on resource
mobilization and service delivery
2a. Average OPI shows lower resource mobilization, high
social capital
4.0
3.5
3.0
2.5
2.0
1.5
1.0
0.5
0.0
Average OPI
Effectiveness
Results
Efficiency
Standards Delivery
Reach
Relevance
Target
Sustainability
Learning Resources
Social
Capital
2b. Graduates need to focus on results
and external evaluation of
4.0 standards
OD Roadmap graduates
3.5
3.0
2.5
2.0
1.5
1.0
0.5
0.0
Average OPI
Effectiveness
Results
Efficiency
Standards Delivery
Reach
Relevance
Target
Sustainability
Learning Resources
Social
Capital
2b. Current partners scored lower on
resource mobilization and service delivery
4.0
3.5
3.0
2.5
2.0
1.5
1.0
0.5
0.0
Non-graduated OD Roadmap partners
Average OPI
Effectiveness
Results
Efficiency
Standards Delivery
Reach
Relevance
Target
Sustainability
Learning Resources
Social
Capital
3a. Different stakeholders, different top-ranked capacity
needs
OD Roadmap partners
USAID
1. Monitoring & evaluation
1. Resource mobilization
2. Technical support
2. Financial management
3. Financial management
3. Administration
4. Program development
4. Monitoring & evaluation
Non-OD Roadmap
partners
Pact
1. Technical support
2. Financial management
2. Financial management
3. Institutional support
3. Advocacy
4. Technical support
4. Monitoring & evaluation
1. Resource mobilization
3b. Graduated partners & USAID: contrasting
expectations
Partners expect:
USAID expects:
1. Same relationship/treatment
with USAID as partners had
with Pact
1. Strong organizational
capacity and governance
systems, ability to work
independently and maintain
capacity even after staff
turnover
2. Continued technical
assistance, but often lack
adequate budget
3. Increased USAID capacitybuilding support after
graduation
2. USAID support to partners,
but also partner ability to
address own needs
3. Performance like long-term
international USAID partners,
responsive and accountable
4. Graduates found both challenges &
benefits
Challenges
Benefits
1. Increased accountability in
line with U.S. Government rules
& regulations
1. Shorter reporting channels
give more time to prepare
reports
2. Less frequent capacitybuilding support
2. Improved interaction with
USAID & understanding of
USAID programs
3. Would like clearer guidance
from USAID on budgets,
funding & programs
4. Would like more regular
feedback on financial reports
3. Less cumbersome financial
reporting requirements
4. International exposure &
recognition
5. Good working relations
Summing up
1. Current partners most likely to retain M&E, financial
and technical systems
2. Sustainability of capacity building depends on whose
decision it was to engage in capacity development
3. On average partners tend to score lower in resource
mobilization on OPI scoring
4. Partners still need to develop capacity, but are
struggling to understand how to access support under
direct funding.
Preliminary recommendations
Preliminary recommendations
• Capacity development impact
• Graduation to direct USAID funding
• Lessons learned for capacity
development (CD) organizations
• General
Capacity development impact
• Capacity development support should be provided for
three or more years (more research is needed).
• Capacity development organizations need to keep their
support “fresh” for organizations served for longer
periods.
• OD roadmap support is an important approach for
preparing effective partners and should continue to be
used.
• Capacity development support should increase focus on
knowledge transfer and resource mobilization for
increased sustainability.
Graduation to direct USAID funding
• The OD Roadmap should be considered as the tool to measure
preparedness for an organization’s readiness for graduation.
• The prerequisites for transition to direct funding should
include:
•
•
•
•
•
•
•
•
Strong strategic planning capacity
Governance structures
Strong financial systems and internal controls
Strong administration (including HR) systems
Skilled staff and technical capacity
Strong M&E
An ability to mobilize resources
An ability to advocate with donors to meet (internal and external)
constituent needs.
Lesson learned for CD organizations
• CD organizations should ensure ownership of
interventions for maximum impact in the area of system
retention.
• CD organizations need to work with partners to diversify
their funding base.
• CD service providers need to help partners to remain
relevant and vibrant by planning to incorporate internal
capacity development support on an on-going basis.
• Increased communication on expectations will improve
the transition to direct funding for both partners and
USAID.
General
• Pact should consider expanding this dataset to
include more Namibian respondents and more
international graduates (33 worldwide) and
partners in order to make the findings more
robust.
• Pact should assess the impact of the capacity
development approach (and the transition to
direct funding) at the beneficiary level.
Thank you!