Monitoring and Evaluation:
FAMILY PLANNING PROGRAMS
Session Objectives
• Be able to apply basic M&E concepts
(frameworks, indicators, etc.) to family-planning
programs
• Be able to summarize the main issues in M&E
of family-planning programs from a post-Cairo
perspective.
• Be able to summarize the emerging issues for
M&E of family-planning programs in high HIV
prevalence countries.
Session Overview
•
•
•
•
•
•
Family-planning frameworks
M&E implications of the Cairo agenda
Contraceptive prevalence and unmet need
Monitoring quality of care
Evaluating the impact of quality
Family planning and HIV
Family Planning Frameworks
Conceptual Framework for FP Demand
and Program Impact on Fertility
Other
intermediate
variables
Societal &
individual
factors
Value &
demand for
children
Fertility
FP demand
• Wanted
• Spacing
• Limiting
Contraceptive
practice
• Unwanted
Service outputs:
Development
programs
• Access
FP supply
factors
Service Utilization
• Quality
• Acceptability
Source: Bertrand, Magnani, and Rutenberg, 1996.
Other health &
social
improvements
Conceptual framework of family
planning supply factors
External
Development
Assistance
FP Organizational
Structure
• Service
infrastructure
Political and
Administrative
System
• Sectoral integration
• Political
support
• Public-private
partnerships
• Delivery strategies
• Management &
supervision
Service Outputs
• Training
• Access
• Commodity
acquisition &
distribution
• Quality
• Acceptability
• IEC
• Research &
evaluation
• Resource
allocations
• Legal code /
regulations
Operations
Larger
societal &
political
factors
Source: Bertrand, Magnani, and
Rutenberg 1996
Applying the frameworks for FP
M&E
• Inputs, e.g.
– Types and levels of resources
– Qualified personnel
– Unit and total costs of program resources
• Outputs – functional areas, e.g.
– People trained
– Performance of people trained
– Cost per person trained
Applying the frameworks for FP
M&E
• Outputs – Service outputs, e.g.,
– Service delivery points providing FP services
– Quality of FP services
– Cost of increasing access/quality of FP services
• Outputs – Service utilization
– New FP acceptors, Couple Years of Protection
(CYP)
– Returning clients
– Cost of increasing CYP, etc.
Applying the frameworks for FP
M&E
• Outcome – intermediate outcomes
– Contraceptive prevalence rate (CPR)
– Unmet need
– Costs associated with increased CPR
• Outcome – long term outcome
– Fertility rates
– Unintended pregnancy
– Costs of changes in fertility, unintended
pregnancy
Indicators for FP programs
• See Bertrand and Escudero, 2002,
Compendium of Indicators for Evaluating
Reproductive Health Programs, 2 volumes
– Indicators that crosscut program areas
– Indicators for specific program areas
What is different about M&E of FP
programs?
• Basic principles are the same as in other
health programs
• Outcomes relatively well-defined, focused,
and measurable
• Long history of data collection on FP
outcomes through WFS, DHS – document
global trends
• Attempts to link outcomes to program
outputs - evidence of program effects
Programme of Action adopted at
ICPD, Cairo 1994
Traditional (pre-Cairo) focus of FP
program M&E
•
•
•
•
•
•
Demographic impact
Focus on married women
Availability of services
Contraceptive adoption (new users)
Characteristics of women
Cross-sectional measurement
Cairo: Objectives of FP Programs
• To help couples and individuals meet their
reproductive goals
• To prevent unwanted and high-risk pregnancies
• To make quality FP services affordable,
acceptable, and accessible
• To improve the quality of family planning IEC,
counseling, and services
• To increase the participation and sharing of
responsibility of men in FP
• To promote breastfeeding to enhance birth spacing
Exercise 1
• Discuss the implications of the Cairo programme
of action for M&E of FP programs. Identify 3 or
more ways in which the traditional focus of FP
programs listed on the earlier slide should change
to respond to the Cairo agenda. What are the
implications of these changes for M&E?
Contraceptive Prevalence Rate
(CPR)
• Percentage of (married) women of reproductive
age (15-49) who are currently using a
contraceptive method.
Unmet Need for Family Planning
• Percentage of fecund women exposed to the risk
of pregnancy who say they want to wait at least
two years for another birth (spacing) or do not want
any more children (limiting), but are not currently
using a method of contraception.
Related Indicators
• Demand for FP = % (married) women using FP +
% (married) women with unmet need for FP
• Percentage of demand satisfied = % (married)
women using FP / % (married) women with
demand for FP
Unmet Need Exercise
CPR vs Unmet Need
CPR
• Relatively simple to
define
• Uni-dimensional
• Consistency over time
• Does not capture
concept of meeting
needs
Unmet Need
• Relatively complex to
define
• Multi-dimensional –
demand & use
• Definition has evolved
• Captures concept of
meeting need
Monitoring Quality of Care
What is Quality of Care in FP?
• General, loosely-defined concept
• Different people define quality in different
ways
• Multi-dimensional
• Appropriate standards against which to
measure quality vary
Bruce-Jain Framework
• Choice of contraceptive methods
• Information given to users
• Provider competence
• client/provider relations
• re-contact and follow-up mechanisms
• appropriate constellation of services
Indicators for QOC
• No single indicator can capture the
different components of QOC
• Indicators need to be adapted to specific
program context and priorities
• Shortlist of 24 QOC indicators (see
Bertrand and Sullivan, Evaluation Bulletin
No. 1, Table 1 page 2).
Facility Surveys for QOC
Indicators
• Situation Analysis
• MEASURE Evaluation Quick Investigation
of Quality (QIQ)
• MEASURE DHS+ Service Provision
Assessments (SPA)
• DHS service availability modules and
community surveys (SAM)
Some Data Collection Issues
• Small sample sizes for FP clients,
especially in low prevalence countries
• Observation in clinics that use a client
flow approach
• Sampling
• Courtesy bias and hawthorn effects
• Unit of analysis (client, provider, facility)
Case Study: QOC in Turkey
Turkey’s Strategic Framework
Strategic Objective
Increased utilization of FP/RH services
Intermediate Result 1
Strengthened sustainability
of FP/RH program
Intermediate Result 2
Expansion of high quality FP/RH services
in the public and private sectors
The Quality Index
•
•
•
•
•
•
Method availability
Availability of trained personnel
Perceived quality of FP counseling
Adequate infection-prevention measures
Availability of IEC materials
Physical access to FP services
Data Source
• Istanbul Quality Surveys
– Facility inventory
– Client exit interviews
• Based on MEASURE Evaluation QIQ
The Quality Index
• Sum of scores from
the 6 components
(range 0-6)
Private
hospitals
SSK
hospitals
Health
centers
MCH/FP
centers
MOH
hospitals
0
2
4
6
Method Availability
• Proportion of facilities
that distribute or
prescribe 3 or more
modern FP methods
Private
hospitals
SSK
hospitals
Health
centers
MCH/FP
centers
MOH
hospitals
0
0.2
0.4
0.6
0.8
1
Perceived Quality of FP
Counseling
• Proportion of clients who
report
– they were seated
– had sufficient time with
the provider
– clearly understood the
information provided
Private
hospitals
SSK
hospitals
Health
centers
MCH/FP
centers
MOH
hospitals
0
0.2
0.4
0.6
0.8
1
Adequate Infection Prevention
Measures
• Proportion of facilities
that meet the following
standards :
– Plastic bucket for CL
solution
– Unused IUD kits kept
sterile
– Medical waste kept in
leak-proof containers
with lids
– Appropriate containers
for sharp objects
Private
hospitals
SSK
hospitals
Health
centers
MCH/FP
centers
MOH
hospitals
0
0.2
0.4
0.6
0.8
1
Evaluating the impact of quality
of care
Framework for links between quality of
family planning services and fertility
Quality of
services
•Choice
Information to
users
Provider
competence
Client-provider
relations
Follow-up
Appropriate
constellation of
services
Other factors
Acceptance
Contraceptive
prevalence
Continuation
Fertility
Other
proximate
determinants
Known effects
Hypothesized effects
Source: Jain, 1989
Outcomes of interest
• Intention to use
• Contraceptive adoption
• Contraceptive discontinuation
– Failure
– Switching
– Stopping
• Current contraceptive use
– Contraceptive choice
• Unwanted pregnancy
Examples of impact studies
• Peru (Mensch, et al., 1996)
• Morocco (Steele, et al., 1999)
• Bangladesh (Koenig et al., 1997)
Morocco Study Design (1)
• To explore whether the service environment in
which a woman resides affects adoption and
continuation of the pill
• Linkage of 1995 Demographic and Health Survey
calendar with 1992 DHS Service Availability
Module
• Multi-level hazards models with contraceptive
adoption and discontinuation as outcomes
• 862 births and 775 episodes of pill use in 107
clusters
Morocco Study Design (2)
• Explanatory factors - Individual and
Community
– age, education, residence, community drinking
water & toilet facilities, principle economic
activity
– Contraceptive intention (discontinuation)
– Breastfeeding status, last child wanted
(adoption)
• Explanatory factors – Program
– Public health center <10km, pharmacy <5km,
outreach services, 3+ methods available at
clinic
– Source of pills (discontinuation)
Predicted percentage of women
adopting a modern contraceptive
method within 12 months of giving
birth by service factors
70
60
50
40
%
30
20
10
0
Yes
No
Health center <10km
<3
3+
No. methods offered at
closest set of facilities
Predicted 12-month pill discontinuation
rate by reason and service factors,
Morocco
35%
30%
25%
20%
15%
10%
5%
0%
Government
Pharmacy or
other
Source
Failure
Desired pregnancy
Yes
No
Health center within 10KM
Side effects/ health concerns
Yes
No
Pharmacy within 5KM
Other method-related
Main Findings: Morocco
• Relatively strong service effects on postpartum adoption
• Service availability associated with both
adoption and discontinuation
• Number of methods available only
associated with adoption
• Users of government sources have lower
discontinuation
Limitations of Impact Studies
• Measures of quality inadequate (often
limited to access and method choice)
• Cross-sectional designs (endogenous
inputs)
• Linking individual and program data
(geographic boundaries, service
environment vs. individual service
experience)
Emerging areas: FP/HIV linkages
and integration
Context
• Considerable progress in preventing unwanted
pregnancy but unmet need remains substantial
• Rapid increases in HIV in many countries
• Changing funding focus to HIV from FP
• Integrated vs. vertical programs
Synergies between FP and HIV
programs
• Both are central to reproductive health
• “ABC” messages in HIV programs also
relevant to FP programs
• Youth programs that encourage responsible
sexual behavior prevent both HIV and teen
pregnancy
• Strong RH policies support both HIV and FP
programs
Dual Protection
• Abstinence
• Monogamous couples using effective
contraception
• Correct and consistent condom use
FP in high HIV-prevalence
countries
• Relationship between HIV and fertility desires
• FP/RH needs differ for:
–
–
–
–
HIV- concordant monogamous couple
HIV- concordant non-monogamous couples
HIV discordant couples
HIV+ concordant couples
• HIV counseling in FP services
FP and VCT
• FP counseling opportunity for VCT or general HIV
counseling and VCT referral
• VCT services could include FP services or FP
counseling and referral
• Concern over unintended consequences of
integration
– Provider burn-out
– Discourage FP clients
– Quality of integrated vs. vertical FP & VCT services
FP and PMTCT
• Averts child infections by preventing
unintended pregnancies among HIV+
women
• PMTCT programs provide opportunity for
prenatal FP counseling and post-partum
contraceptive use
• Reduced breastfeeding by HIV+ mothers will
lead to shorter birth intervals in the absence
of FP
FP Counseling of PMTCT clients,
Zambia
60
50
40
% 30
20
10
0
ANC visit
3 months PP
pregnant women
HIV+
6 months PP
HIV-
Source: Rutenberg & Baek, 2004
PMTCT-Client FP Use 6 Months Postpartum, Zambia
45
40
35
30
25
%
20
15
10
5
0
Modern
Condom
HIV+
Sex active, no
method
HIV-
Source: Rutenberg & Baek, 2004
% FP sessions
HIV Counseling in FP Sessions,
Uganda
40
35
30
25
20
15
10
5
0
HIV needs
assessed
HIV risk discussed Dual protection
proposed
Baseline
Follow-up
Source: Rutenberg & Baek, 2004
Exercise 3
• Select an area of FP/HIV integration (e.g.
PMTCT, VCT, HIV counseling in FP etc.).
– Develop a basic input-output-outcome-impact
framework for a simple program in this area.
– Suggest 3-6 indicators to monitor your program.
– What data sources would you propose to collect
these indicators?