Presentation of Final Report
The Results of the Thematic Evaluation on
Maternal Health (2000-2011)
United Nations Population Fund
Evaluation Branch
Division for Oversight Services
New York, 07.11.2012
Overview of proceedings
10:30 Introduction:
• Objectives, scope of
evaluation
• Methodology used
Louis Charpentier, Evaluation
Manager, Evaluation Branch /
DOS (UNFPA)
10:40 Presentation of key findings and
conclusions of thematic
evaluation
Martin Steinmeyer, Team Leader
(AGEG)
Isabelle Cazottes, Co-Team
leader (AGEG)
11:20 Open discussion
11:40 Presentation of the
recommendations of the
evaluation
Martin Steinmeyer
12:00 Use of evaluation results
Laura Laski, Chief – Sexual and
Reproductive Health Branch,
Technical Division (UNFPA)
12:30 Open discussion
12:50 Next steps
2
Evaluation Branch, Division for Oversight Services
Louis Charpentier
Introduction: Objectives, Scope and Methodology
of the Thematic Evaluation
Evaluation Manager
Chief, Evaluation Branch,
Division for Oversight Services
3
Evaluation Branch, Division for Oversight Services
Objective, users, rationale of maternal
health thematic evaluation
Objective
• Assess relevance, effectiveness, efficiency and
sustainable of UNFPA in contributing to improvement of
maternal health
• UNFPA country offices, Programme and Technical Divisions,
including MHTF staff
Evaluation • Partner Governments, member states, donors, civil society
Users
•
•
Rationale •
& Purpose •
4
Inform decision-making on policy / project management
Establish accountability and oversight of fund management
Promote lesson-learning culture
Provide guidance on strengthening partnerships
Evaluation Branch, Division for Oversight Services
Thematic & temporal scope
MYFF 2000 - 2003 MYFF 2004 - 2007 Strategic Plan 2008 - 11
Relevance of UNFPA support
Capacity Development / Human Resources for Health
SRH Services (Family Planning / EmONC)
Integrating MH into national policies & frameworks
MH support in humanitarian contexts
Results & evidence orientation
Coherence of SRH, Population & Development, Gender
Harmonization & coordination of support
Coherence between country, regional & global programmes
Visibility of UNFPA in maternal health
2010/11
2000
5
Evaluation Branch, Division for Oversight Services
Geographic scope of evaluation
1st stage sampling (“Universe”)
55 programme countries with MMR
greater than 300 death / 100,000 life
births
2nd sample
22 countries (range of MH
performance, GNP, quality of public
admin, HIV prevalence)
3rd sample (country case
studies)
Burkina Faso, Cambodia, DRC,
Ethiopia, Ghana, Kenya, Lao
PDR, Madagascar, Sudan and
Zambia
6
Evaluation Branch, Division for Oversight Services
Methodology – Definition of Scope
UNFPA results
frameworks
(MYFFs, Strategic
Plan, etc.)
MHTF Business
Plan, etc.
Analysis and reconstruction
of UNFPA’s own
intervention logic (Core
fund (& MHTF))
Analysis of
strategies of other
UN agencies &
donors
Academic and
“gray” literature on
reproductive &
maternal health
Analysis of the context of
UNFPA support to
maternal health
Identification of critical aspects
(Core Fund & MHTF)
2 sets of Evaluation Questions:
1. Maternal Health Thematic
Evaluation (12 EQs)
2. MHTF Mid-Term Evaluation (8 EQs)
7
Evaluation Branch, Division for Oversight Services
ToR / Input from
Reference
Group
Main Topics / Issues /
Purpose identified in ToR;
specified by Reference
Group
DAC Criteria
(Relevance,
Effectiveness,
Efficiency,
Sustainability)
Methodology – Data Collection
Data were collected to answer 12 Evaluation Question of
Thematic Evaluation
Data collected by following means:
8
Desk review of existing evaluations, reviews and other
documents;
Ten country case studies (Burkina Faso, Cambodia, Ethiopia,
Ghana, Lao PDR, Madagascar, Sudan (North) and Zambia); focus
groups, interviews with UNFPA staff, Government, development
partners, civil society, beneficiaries
Online survey of 55 country offices on technical support to
country offices & on organizational capacity
Face-to-face and telephone interviews with UNFPA staff
members (headquarters, regional and sub-regional offices), other
external partners.
Evaluation Branch, Division for Oversight Services
Methodology – Number of People Interviewed
Number of People
Interviewed
Institutions
9
UNFPA
23 (global), 6 (regional),
65 (country)
Central Government
90
Local Authorities
50
Development Partners
8 (global) 65 (country)
Civil Society Organisations
55
Training Institutions
30
Health Service Providers
40
Final Beneficiaries (focus group
discussions)
18 discussions (10-15
people each)
Evaluation Branch, Division for Oversight Services
Key Findings and Conclusions: The UNFPA
contributions to changes in maternal health
outcomes
Martin Steinmeyer, team leader (AGEG)
Isabelle Cazottes, co-team leader (AGEG)
10
Evaluation Branch, Division for Oversight Services
Answers to the Evaluation Questions
SELECTED KEY FINDINGS
11
Evaluation Branch, Division for Oversight Services
Evaluation Question 1: Relevance
Resources not allocated proportional to severity
of maternal health needs
No clear, operational definition of maternal
health-related vulnerability
12
Resource allocation system has not fully considered
relative need levels in each country
Resources distributed based on criteria such as “degree
of political support for ICPD”, “absorptive capacity”
Common practice: targeting MH support geographically
(regions / districts with high maternal mortality
prevalence)
Less common: targeting systemic socio-economic
barriers to MH services (cost, distance, transportation)
Evaluation Branch, Division for Oversight Services
Evaluation Questions 4: Capacity
Development – Human Resources for Health
Training support thematic priority of UNFPA
Challenge: Inappropriate deployment & placement
of trained staff / high mobility
13
2000 – 2005: in-service trainings; Challenges: Low alignment
with national HRH systems; Little discernible sustained effect
on quality of service
2005 – 2010: pre-service trainings, support of nat. training
oversight agencies (curricula review, regulatory frameworks);
Advantages: alignment w. nat. structures, sustained effects on
SRH training systems
Linked to weak nat. HRH systems (planning, implementation,
monitoring) – beyond SRH / MH
UNFPA offices struggled to approach cross-cutting challenges
Some partnerships w. HRH offices in health ministries (e.g.,
Cambodia)
Evaluation Branch, Division for Oversight Services
Evaluation Question 7 – Emergency
Obstetric and Newborn Care (EmONC)
EmONC support prior to 2008, but MHTF has
intensified UNFPA EmONC support
But: Availability of; access to EmONC affected by
admin. gaps, cost, weak referral systems
UNFPA offices struggled to approach cross-cutting challenges
(barriers) (UNFPA mandate? Capacity?)
Positive exceptions: 1) Analysis of barriers; 2)
Long-term Partnerships in- & outside SRH; 3)
Advocacy
14
Technical assistances, finances sped up EmONC needs
assessments; basis for scale-up plans (Lao PDR)
Burkina Faso: EmONC subsidy, emerged out of long-term
UNFPA advocacy, partnership w. Direction de La Famille
(MoH), Parliamentarians; increase in facility-based deliveries
(28% overall (2003 - 2010))
Evaluation Branch, Division for Oversight Services
Overview of key findings for other
evaluation questions (selection)
Questions
EQ2: Harmonization
/ Coordination
Key Findings
• Long-term partnerships prerequisites for UNFPA
effect on evidence-based harmonized MH support
EQ3: Communities & • MH awareness raising to be coupled with
MH demand
addressing financial barriers to be successful
EQ6: Family
planning
• Good results / partnerships in commodity security
(GPRHCS); less focus on FP demand / delivery
EQ8: Results /
Evidence
• Weak monitoring and evaluation of MH
interventions; low results-focus
EQ9: MH policy
frameworks
• Success depended on combining data & research,
advocacy, technical assistance, partnerships
EQ10: SRH,
Gender, P&D
• Country offices commonly lacked management
mechanisms for integrated programming (strategy!)
EQ12: Visibility
• Visible advocate for MH; extent depended on CO
capacity to bring technical knowledge to bear
15
Evaluation Branch, Division for Oversight Services
Performance patterns of country-level
support – Example HRH
Health policy arena (&
other policy arenas)
SBA plan;
midwifery
training policy
SBA
assessm.
Policy
advocacy
TA SBA
Long-term strategy
16
Evaluation Branch, Division for Oversight Services
Gender
SRH
P&D
CO
Management
Ohter
donors
RH policy arena
E.g,
WHO
MH Centre
/ Nursing
Councils
HRH
Dep. /
MoH
HRH Policies /
Programmes
Performance patterns of country-level
Health policy arena (&
support - positive scenario
other policy arenas)
MH policies &
programmes
Policy
advocacy
Technical
Assistance
Data /
research
Long-term strategy
Country Office
17
Evaluation Branch, Division for Oversight Services
Gender
SRH
P&D
Management
Dev.
partner
RH policy arena
Dev.
partner
Gov.
partner
Gov.
partner
Health Policies
/ Programmes
Performance patterns of country-level support
– opportunities for improvement Health policy arena (&
other policy arenas)
MH policies &
programmes
Policy
advocacy
Technical
Assistance
Data /
research
Short-term planning (project by project)
Country Office
18
Evaluation Branch, Division for Oversight Services
Gender
SRH
P&D
CO
Management
Dev.
partner
RH policy arena
Dev.
partner
Gov.
partner
Gov.
partner
Health Policies
/ Programmes
Conclusions
Conclusions - Appropriateness of
UNFPA maternal health strategy
C1:
UNFPA maternal health support in programme countries
has not been sufficiently based on country-specific
medium or long-term strategies
20
Maternal health support has been more effective when country
offices based interventions on multi-annual strategic vision
Current templates not conducive for strategic planning; No
requirement to develop multi-annual operational plan.
Country offices without multi-annual perspective more likely to
manage interventions separately from each other;
Fewer incentives for staff working on different interventions to
coordinate work; pool financial and organizational resources
from different funding sources
Evaluation Branch, Division for Oversight Services
Conclusions - Appropriateness of
UNFPA maternal health strategy (2)
C2:
In its approach to support maternal health, UNFPA has
not sufficiently responded to its mandate to focus on
addressing the root causes of poor maternal health of the
most vulnerable
21
HQ has not sufficiently defined operational implications of focusing
on needs of “most vulnerable”
Country offices did not sufficiently analyse risks and barriers
keeping women and girls from accessing MH services (instead
targeted geographical pockets of high maternal mortality)
COs did not address weakness of health systems that made
specific groups more vulnerable to poor maternal health (e.g.,
deployment challenges, in particular for rural areas; deficiencies in
supervision and service quality control)
Evaluation Branch, Division for Oversight Services
Conclusions - Appropriateness of
UNFPA maternal health strategy (3)
C3:
UNFPA support of maternal health services at subnational level has not consistently reflected comparative
strengths of UNFPA as primarily knowledge- and
evidence-based organization
22
Significant portion of small SRH budget allocated to subnational level (sub-national offices, staff, interventions); often
one of several organisations active there
Country offices have not consistently used presence at subnational level to generate data, lessons and knowledge to
further national MH policy agenda
Evaluation Branch, Division for Oversight Services
Conclusions – role & capacity of UNFPA
country offices
C4:
Insufficient staff capacity and skill gaps have negatively
affected ability of country offices to act as brokers of
maternal health expertise and as facilitator of national
and international maternal health commitments and
partnerships
23
Small numbers of RH staff made it difficult to be present in
national technical working groups or policy forums
Annual planning based on inadequate templates (AWPs, CPAPs)
and limited experience in strategic planning made it difficult to
develop multi-annual strategies that combined resources and
skills from different areas to facilitate sustained changes in health
systems
Evaluation Branch, Division for Oversight Services
Conclusions – role & capacity of UNFPA
country offices (2)
C5:
Country offices have not received sufficient technical
support from regional offices and headquarters to fulfill
central role delivering maternal health support
24
Availability of technical support has been limited in human
resources for health, EmONC and midwifery (pre-MHTF),
strategic planning, results-based management and monitoring
and evaluation
Maternal Health Thematic Fund (MHTF) has been valuable
instrument to make available additional resources for technical
support
Evaluation Branch, Division for Oversight Services
Conclusions – partnerships with donors,
governments and other stakeholders
C6:
Varying capacity of country offices to establish
partnerships with government, donors and civil society in
and outside of reproductive health has reduced ability to
address service access barriers and to strengthen MH
systems
25
Long-term, proven working partnerships allow UNFPA to extend
reach beyond sexual and reproductive health
Partnerships allow UNFPA to tie maternal health HRH
support to larger governmental efforts to strengthen national
health systems
Pre-conditions: UNFPA to repeatedly prove worth as reliable
partner, gain trust and collaboration, consistent leadership
through country representatives
Evaluation Branch, Division for Oversight Services
Conclusions – Use of Evidence; Monitoring
and Evaluation
C7:
Lack of appropriate monitoring and evaluation
mechanisms has affected capacity of UNFPA to assess
results of maternal health support and to optimize
corporate and country-level MH strategies over time.
26
Monitoring has focused on activities or higher level societal
changes relating to maternal health,
Monitoring system thus not providing information on UNFPA
contribution to these changes (which mirrors deficiencies in
UNFPA planning)
UNFPA implementing partners w/o required technical capacity to
fulfil M&E responsibilities; UNFPA country offices w/o sufficient
skilled M&E staff to support implementing partners
Evaluation Branch, Division for Oversight Services
Conclusions – Added value of the Maternal
Health Thematic Fund (MHTF)
C8:
Maternal Health Thematic Fund (MHTF) has helped to
provide much needed financial and staff resources to
UNFPA country offices and headquarters in the short and
medium-term.
27
MHTF-financed staff positions bolstered staff capacity of
country offices, allowing them to intensify engagement in
in EmONC, midwifery, obstetric fistula
Partnerships made available additional technical support
(e.g., Averting Maternal Death and Disability (AMDD),
International Confederation of Midwives (ICM))
Helped to raise UNFPA profile and visibility globally & in
countries
Evaluation Branch, Division for Oversight Services
Conclusions – Added value of the Maternal
Health Thematic Fund (MHTF)
C9:
The MHTF has not been sufficiently integrated into
organizational structure of UNFPA and the overall
planning process at country level to ensure sustainability
of its interventions
28
MHTF has proved instrumental in facilitating EmONC
assessments, midwifery needs assessments, which constituted
basis for governments to launch MH-relevant reforms
MHTF not sufficiently integrated into UNFPA planning to ensure
timely resource mobilization from within UNFPA and partners for
continued support of reforms
Evaluation Branch, Division for Oversight Services
QUESTIONS?
COMMENTS?
29
Evaluation Branch, Division for Oversight Services
Recommendations
Recommendations: Reviewing UNFPA
maternal health strategy and support
R1:
UNFPA should revise internal procedures, tools and
templates for strategic planning. New approach should
require country offices to develop maternal health support
strategies for medium to long-term, and to detail how
resources from different sub-programmes will be used to
implement these strategies. (based on conclusions C1, C7)
31
Present detailed analyses of specific political, administrative,
socio-economic challenges in revised CPDs and CPAPs and
corresponding strategy to address them
Put in place multi-annual plans that detail theory of change,
risks & assumptions and monitoring indicators for strategy
Evaluation Branch, Division for Oversight Services
Recommendations: Reviewing UNFPA
maternal health strategy and support (2)
R2:
UNFPA needs to better define operational implications of
targeting the needs of the “most vulnerable”. Concept is
relevant part of maternal health strategy, yet it is too vague
in its current form to guide maternal health programming at
country level. (based on conclusion C2)
32
Develop typology of MH-relevant barriers for different types of
services and contexts
Prepare operational and programming guide to a) offer clear
definition of MH vulnerability and the contributing social,
economic, political variables; b) identify policy sectors that
influence MH vulnerability; c) lay out options to influence
determinants of vulnerability; d) define UNFPA role in
contributing to donor response; e) discuss use of partnerships
Evaluation Branch, Division for Oversight Services
Recommendations: Reviewing UNFPA
maternal health strategy and support
R3:
UNFPA needs to increase focus on knowledge generation
and learning, to make use of lessons from sub-national
service delivery support (and other interventions) to inform
evidence-based policy advocacy at core of UNFPA
mandate. (based on conclusion C3, C7)
33
Service-delivery support needs to contribute to body of
knowledge on maternal health support in programme country
and beyond
Strengthen provisions for monitoring, evaluation, learning, in
particular for interventions at sub-national level
Evaluation Branch, Division for Oversight Services
Recommendation: Improving capacity of
country offices
R4:
UNFPA needs to better align capacity and skill mix of staff
and managers to ensure that country offices can fulfill their
role as knowledge brokers and facilitators of evidencebased approaches to improve maternal health. (based on
conclusions C1, C4, C7)
34
Country offices to develop resource plan (part of CPAP) to
allocate staff time to strategy components
Strengthen skills related to health policy and management,
public health ; also project management, M&E
Strengthen accountability of country representative for strategic
positioning of country offices for partnerships, advocacy to
complement UNFPA interventions
Evaluation Branch, Division for Oversight Services
Recommendation: Improving capacity of
country offices
R5:
Planning process for regional-level technical support to be
better aligned with long-term strategic and operational
planning for maternal health support at country level.
(based on conclusions C4, C5)
Long-term strategic planning at country level needs to
be mirrored by long-term planning of technical support
at regional level
Regional offices’ planning processes need to address
current country-level programming needs and
anticipate future support requirements
RPAPs / Country Programmes to be developed jointly
with country offices, HQ, MHTF, GPRHCS, including
resource plan (similar to country level)
35
Evaluation Branch, Division for Oversight Services
Recommendations: Improving guidance on
UNFPA partnerships
R6:
Anchor concept of partnerships more firmly in strategic
documents, operational guidelines and job descriptions of
managers. Strategic documents need to explain
importance of different types partnerships to ensure
sustainable results. UNFPA managers need to be
responsible for setting up required partnerships at
country, regional and global levels. (based on conclusions
C1, C6)
36
Develop typology of stakeholders and their significance for
addressing root causes of poor maternal health
Strengthen capacity of regional offices to support country offices
in establishing partnerships
Job descriptions of managers need to emphasize their
responsibility for strategic positioning of UNFPA through
partnerships (performance criterion)
Evaluation Branch, Division for Oversight Services
Recommendations: Improving UNFPA
provisions for monitoring
R7:
Strengthen result-oriented monitoring for country offices
to measure results and not only activities and inputs.
Assign greater responsibilities and offer more guidance to
country offices for supporting the set-up of appropriate
monitoring mechanisms with implementing partners.
(based on conclusions C1, C4, C5, C7)
37
Develop guidelines for results-oriented monitoring & related training
resources
Strengthen responsibilities and capacity of country offices to support
set-up of monitoring mechanisms with implementing partners
(consider partnerships)
Evaluation Branch, Division for Oversight Services
Recommendations: Revise role & integration
of MHTF
R8:
Strengthen MHTF as catalytic tool to facilitate
implementation of evidence-based maternal health
interventions. Clarify that MHTF is not only responsible for
launching initiatives (EmONC, midwifery), but also for
assisting country offices to support these initiatives until
completion. (based on conclusions C8, C9 + MHTF-midterm evaluation)
38
Clarify: MHTF helps launch initiatives; and supports their
implementation (business plan)
Update guidance to country office on purpose and role of MHTF
Support to expect from MHTF
Responsibilities of country offices when working with MHTF
funds
Harmonize MHTF planning with (proposed) multi-annual planning
approach for core funds
Evaluation Branch, Division for Oversight Services
Recommendations: Revise role & integration
of MHTF
R9:
UNFPA should use MHTF funds to carry out pilot
interventions on selected core maternal health issues,
such as the development of appropriate support strategies
to better target populations with high vulnerability to poor
maternal health. (based on conclusions C3, C8, C9)
39
Inventory of MH issues and topics that require pilot-testing
Invite country offices to submit proposals for pilots for
maternal health interventions
Ensure access of country offices to required resources for
pilots through MHTF
Ensure analysis and dissemination of results with MHTF
resources
Evaluation Branch, Division for Oversight Services
QUESTIONS?
COMMENTS?
40
Evaluation Branch, Division for Oversight Services
NEXT STEPS
- Presentation of the evaluation results
and the management response to the
Executive Board – January 2013
- Webinar presentation of the
management response – February 2013
41
Evaluation Branch, Division for Oversight Services
Maternal Health Thematic Evaluation Report and
related products are available at the following link:
http://www.unfpa.org/public/home/
about/Evaluation/EBIER/TE/pid/10094
42
Evaluation Branch, Division for Oversight Services
THANK YOU FOR YOUR
ATTENTION
Any other questions?
Contact us: evb@unfpa.org
43
Evaluation Branch, Division for Oversight Services
