Measuring the Impact of the
Second Urban Primary Health Care
Project in Bangladesh
Manuel Leonard F. Albis, University of the Philippines
Subrata K. Bhadra, National Institute of Population Research and Training
Brian Chin, Asian Development Bank
1
Outline of Presentation
1. Introduction
–
–
–
Background on UPHCP-II
Public-private partnership
Monitoring and evaluation component
2. Methods used to determine project impact
– Recalibration of baseline sampling weights
– Propensity Score Matching Difference-in-Differences
3. Results and discussions
4. Conclusion and recommendations
2
UPHCP-II in Bangladesh
• Second Urban Primary Health Care Project (UPHCP-II)
(2005-2012), the follow-up phase to UPHCP, developed by
the Government of Bangladesh with support from
ADB, SIDA, DFID, UNFPA, and ORBIS International
• Four main outputs:
1.
2.
3.
4.
Provide PHC services through partnership agreements and
BCCM
Strengthen urban PHC infrastructure and environmental health
Build capacity and policy support for urban PHC
Support research into project implementation and operations
research
Introduction
3
UPHCP-II in Bangladesh
• Government facilitated 24 partnership agreements, for
areas covering 200,000 to 300,000 people, with NGOs
• Comprehensive reproductive health care center
(CRHCC) was established in each partnership
agreement area, providing full emergency obstetric
care, newborn care, and other specialized services
• At least one primary health care center (PHCC) catering
to 30,000 to 50,000 people, which provides basic
emergency obstetric care
Introduction
4
UPHCP-II Health Services
• Essential Services Package Plus (ESP+)
1.
2.
3.
4.
5.
immunization and growth monitoring of children
micronutrient support and malnutrition
family planning
prenatal, obstetric and postnatal care with special attention to
prevent eclampsia, STI and HIV/AIDS
other reproductive health, and child health
• Systematic case management of
pneumonia, diarrhea, tuberculosis, leprosy, malaria, etc.
• Free health services and medicines to the poor; 32% of each
major type of service was provided to poor patients
Introduction
5
UPHCP-II BCCM Component
• Behavior Change Communication and Marketing (BCCM)
component aimed to educate the urban population in order
to increase their knowledge, improve their attitude, behavior
and practices related to health
• Included comprehensive dissemination of information
through posters, stickers, billboards, radio programs, and TV
serials
• Capacity-building seminars and trainings were conducted to
sustain effective and efficient delivery of health care with a
focus on pro-poor and gender-sensitive targeting and
monitoring
Introduction
6
Contracting Health Services to NGO
• The public-private partnership (PPP) design to deliver
health services has been found efficient and effective
• Advantages of this approach (Loevinsohn 2008) include:
i.
ii.
iii.
reduced “red tape” and unhelpful politics
constructive competition among NGOs
less administrative burden to the government
• Effectiveness of such design was documented for several
projects in
Bolivia, Guatemala, Haiti, India, Madagascar, Senegal and
Pakistan (Loevinsohn and Harding 2005)
Introduction
7
PPP in UPHCP: Major Findings
• Health services were contracted out to NGOs in
Dhaka, Chittagong, Khulna, Barisal, Rajshahi and Sylhet city
corporations, and other district municipalities
• Assessment of the UPHCP design (Heard, Nath & Loevinsohn
2013) showed that NGO contracted areas were better than
local government (Chittagong) contracted areas in terms of:
i.
ii.
iii.
iv.
v.
quality of care and higher health care services per capita
wider coverage of the poorest 50% of the population
higher likelihood of health seeking
absorptive capacity to utilize budget
more motivation
Introduction
8
PPP in UPHCP: Major Findings
• The drivers of efficiency in the design of
contracting-out health services are attributed
to:
i.
ii.
iii.
iv.
v.
vi.
Proximate competition
Direct procurement from approved suppliers
Simpler decision making
Flexibility in human resources
Financial management
Close ties with community
Introduction
9
UPHCP-II Monitoring and Evaluation
• UPHCP-II included an M&E component
– Baseline and Endline surveys were conducted for each
of the three phases of the project for planned impact
evaluation
• A sample of households from project areas (PA)
and non-project areas (NPA) were included in the
survey following a DHS-type questionnaire
• For UPHCP-II, the baseline and endline surveys
were conducted in 2005 and 2012, respectively
Introduction
10
UPHCP-II Baseline Survey
• The 2005 UPHCP-II baseline survey was not able to
obtain a control sample due to operational problems
• To conduct impact evaluation, the 2006 Urban Health
Survey (UHS) was used as the baseline for UPHCP-II; it
has a DHS-type questionnaire, making the estimation
and comparison of health outcome indicators possible
• The domains between the two surveys are
different, thus adjustments were made to establish
comparability
Introduction
11
Objectives and Significance of the Study
• This study evaluates the impact of UPHCP-II on selected
health outcomes using the 2006 UHS as baseline and 2012
UPHCP-II Endline survey
• The effect of the project was estimated through differencein-differences (DID) with propensity score matching (PSM)
between designated project and non-project areas
• The innovation introduced by this paper is the recalibration
of sampling weights that allows the use of two unrelated
surveys in impact evaluation
Introduction
12
METHODOLOGY IN ASSESSING
IMPACT OF UPHCP-II
13
Reweighting the 2006 UHS
• The 2006 UHS and 2012 UHPCP-II endline surveys do not
have the same domain structure
• To ensure comparability, 2006 UHS sampling design was
transformed to mirror the domain structure of the endline
survey (recalculating the 2006 UHS survey weights by
assuming the sampling procedure was that of the endline
survey)
• The rationale is that the sampling weights reflects the
sampling design of the survey, thus recalculating the
sampling weights restructures the sampling design in order
for the two surveys to be comparable
Methodology
14
Reweighting the 2006 UHS
• For illustration, suppose that Domain 1 of UHS contains 8
wards, which belong to the first five Domains of UPHCP-II
Endline
UPHCP-II Design
UHS Design
Domain 1
Ward 1
Ward 2
Ward 3
Wards 1 & 2
Domain 2
Ward 4
Domain 3
Wards 5 & 6
Domain 4
Ward 7
Domain 5
Ward 4
Ward 5
Ward 6
Ward 7
Ward 8
Ward 8
Domains in UPHCP-II
consist of wards
UHS Design is different
from UPHCP-II
Regroup
wards in UHS
to match the
domain
structure of
UPHCP-II
Sampling
weights of
UHS are
recomputed
following
UPHCP-II
design
Revised UHS Design
Ward 1
Ward 2
DROPPED
D1
D2
D3
D4
D5
Ward 4
Ward 5
Ward 6
Ward 7
Ward 8
Methodology
15
The Baseline and Endline Surveys
• The 2006 UHS surveyed 14,191 women with eight main
domains representing slum and nonslum areas across
Dhaka City Corporation, Other City Corporations, and
municipalities
• The 2012 UPHCP-II Endline survey has 21,269 women
respondents, having a total of 32 PA domains with
corresponding 32 NPA domains
• In matching, the sampling design between the two
surveys, individuals located inside the specified PA and NPA
areas in UPHCP are included in the analysis; 2,405
individuals or 16% of the sample were outside the
designated PA and NPAs
Methodology
16
Number of Women in PA and NPA
Area
Dhaka
Khulna
Rajshahi
Barisal
Sylhet
Chittagong
Comilla
Narayanganj
Rangpur
Municipalities
Total
2006
PA
2,046
798
551
398
299
3,434
61
62
124
7,773
2012
NPA
1,769
794
349
215
234
334
28
86
204
4,013
PA
3,314
711
746
347
363
1,092
375
358
390
2,872
10,568
NPA
3,320
707
711
341
387
1,143
373
362
383
2,974
10,701
Methodology
17
DCC PA and NPA Map
• There are 20 survey
domains in DCC, 24 in
OCC and 16 in
municipality areas
• All PAs in DCC have
their corresponding
NPAs
Methodology
18
Propensity Score Matching
• PSM was applied to health outcome indicators by
matching using household and individual
characteristics; correcting selection bias is the main
objective of this approach
• The approach is to perform PSM by survey period
between PA and NPA areas, using individual and
household characteristics as matching variables
• Treatment effects were then computed between the
baseline and endline surveys using difference-indifferences
Methodology
19
Estimating Propensity Scores
Methodology
20
Nearest Neighbor Matching
Individual in the treatment group is matched to an
individual in the control if they have almost the same
propensity score
Nearest Neighbor to
Propensity Score
0
1
Shortest Distance
Methodology
21
Impact Evaluation
• PSM variables that were used for matching women are (1)
wealth index, (2) age, (3) parity, (4) highest educational
attainment, (5) religion, and (6) major geographic grouping
(DCC, OCC and Municipalities)
• For matching children, variables used were (1) age of child, (2)
gender of child, in addition to the women variables given
above
• The treatment group is a sample of individuals from
designated PA and not restricted to individuals who have
accessed UPHCP-II health facilities
Methodology
22
Matched Difference-in-differences (DID)
Methodology
23
Matched DID
•
•
•
DID captures the effect of unobservable characteristics that may affect the
outcome of the project
DID assumes that change in health outcomes of counterfactual is the same
as that of the control group
Estimated project impact is the change in health outcomes in the
treatment group minus the change in the health outcomes in the control
24
RESULTS OF IMPACT EVALUATION
25
Child Nutrition
• Of the three indicators of child malnutrition, only stunting was
significantly reduced
• Even though wasting and underweight indicators were not
reduced significantly, all have negative estimated ATT
Health Indicator
Stunting (height for age)
cumulative effect of chronic malnutrition
Wasting (weight for height)
acute or recent nutritional deficit
Underweight (weight for age)
overall indicator of nutritional health
N
ATT
Boot SE
3,512
-5.5***
2.46
q
3,608
-0.5***
1.84

3,736
-2.8***
2.28

** Significant at 5%
UPHCP-II Impact
26
Diarrhea and ARI Prevalence
• No significant impact on diarrhea prevalence and access to
health facility, although the signs of the estimated effects are
in line with expectations
• ARI and Fever prevalence decreased significantly
Health Indicator
N
ATT
Boot SE
Diarrhea prevalence
5,050
-1.4***
0.89

ARI prevalence
5,050
-5.4***
1.26
q
Fever prevalence
5,050
-4.2***
1.94
q
ARI and Fever prevalence
5,050
-5.6***
1.15
q
* Significant at 10%; ** Significant at 5%; *** Significant at 1%
27
ANC, PNC and Attended Births
•
•
•
Coverage of antenatal care of at least three visits significantly increased
Increase in postnatal care coverage in NPA areas from baseline to endline
is higher than the change in PA areas perhaps due to other programs
present in NPA
Attended births increased at a slightly higher rate in PA and NPA areas
although not statistically significant
Health Indicator
N
ATT
Boot SE
ANC (at least 1 visit)
ANC (at least 2 visits)
ANC (at least 3 visits)
3,788
3,788
3,788
5.2***
5.3***
7.4***
2.03
2.28
2.45
p
p
p
PNC (mothers who gave birth within 5
years preceding the survey)
3,788
-6.8***
2.36
q
Attended births at health facility or at
home
3,788
3.3***
2.38

* Significant at 10%; ** Significant at 5%; *** Significant at 1%
28
Breastfeeding
• Proportion of ever breastfed children decreased from 2006 to
2012, also reflecting a significant negative ATT
• Significantly increased complimentary breastfeeding yields an
estimated ATT of 3.8 percentage points
Health Indicator
N
ATT
Boot SE
Ever Breastfed
2,057
-2.6**
1.16
q
Breastfed within 1 day of birth
2,057
0.9**
2.66

Complementary breastfeeding 6 months
onwards
3,205
3.8**
2.23
p
* Significant at 10%; ** Significant at 5%; *** Significant at 1%
29
CPR, RTI and AIDS
• Modern contraceptive use among women also benefitted
from information drive in reducing RTI
• RTI prevalence dropped
• Highly effective in behavioral change and communication on
sexually transmitted infection such as AIDS
Health Indicator
N
ATT
Boot SE
CPR (all women)
CPR (married women)
17,752
15,560
3.7***
3.9***
1.15 p
1.21 p
RTI Prevalence on Women
18,341
-1.4***
0.82 q
AIDS Awareness
18,341
2.3***
0.60 p
AIDS Avoidance
18,341
4.4***
0.94 p
* Significant at 10%; ** Significant at 5%; *** Significant at 1%
30
DISCUSSION OF UPHCP-II IMPACT
31
Effectiveness
•
•
•
•
Stunting in children improved
ARI and fever prevalence in children improved
Antenatal care improved
Knowledge dissemination changed the
behavior of women as seen in the increase of
modern contraceptive use and awareness of
AIDS/HIV and its avoidance
Discussion
32
Efficiency of Contracting Health Services
• UPHCP-II delivered health care services to 10.2 million
clients; higher than the project target of 9.4 million
• UPHCP-II efficiently delivered health care services to
the poor at reduced or even no cost at all despite
utilizing only 80% of allocated funds
• Modality flexible in addressing the evolving needs and
concerns of the health workers in terms of their
tasks, incentives and supervisory mechanisms, enabled
rapid and wide dissemination of health services
Discussion
33
Quality of Care
• The project areas in Dhaka and other city corporations have
high percentage of trained providers, while around 60% of
the staff in municipality project areas received training
• Quality of health services is not highly correlated with
lowest cost of health services, high maternal and child
health service coverage, quantity of services, or duration of
contract
• Quality of care is not uniform across all PA areas; Dhaka and
Khulna delivered high quality of care, while municipalities
generally had room for improvement
Discussion
34
Satisfaction
• Exit client surveys report that around 3 out of 4
patients were satisfied with the health services of
UPHCP-II (Midterm Exit Client Report 2010)
• Proximity and good quality of service were main
factors
• Municipalities fared lower due to high cost per
service, and unavailability of some health services
Discussion
35
CONCLUSION AND
RECOMMENDATIONS
36
Conclusion
• The project benefitted from the efficiency of contracting-out health
services to NGOs—cost effective for:
– Urban local bodies (ULBs) as technical and financial support were
provided by government and development partners vs. ULB budgets
– PA NGOs’ technical capacity for PHC service delivery strengthened
– Urban poor who received quality PHC for no or reduced cost
• Successful in terms of its effectiveness in delivering health services
and thus positive impacts on various health indicators
• BCCM was crucial for comprehensive dissemination of health
knowledge
37
Conclusion
• Monitoring of project performance through
unified record keeping system (HMIS), integrated
supervisory instrument surveys (QA), and field
visits helped provide timely updates to NGOs
• Experience from UPHCP-II improved
design, implementation, management, and
delivery of health services for the follow-up
phase of the project, Urban Primary Health Care
Services Delivery Project (UPHCSDP)
38
Recommendations
1. Health-related
– More efforts on malnutrition, breastfeeding, and postnatal care
– Continuous dissemination of maternal health
knowledge through rigorous behavioral change
campaigns to improve attended births
– PA areas in municipalities should be strengthened to
catch-up with the performance of the PA areas in city
corporations
39
Recommendations
2. Monitoring and Evaluation
– Enhanced and efficient M&E dashboard mechanism
with outcome, process and input-driven indicators for
timely feedback to project implementers
– Survey design should be given greater thought at the
onset to avoid changing of design and questionnaire
during project implementation
– Standardized and up-to-date data repository of health
statistics (HMIS) would help to efficiently measure
urban health status
40
Recommendations
3. Overall Project Implementation
– Poverty Targeting (transparent selection process for
health entitlement cards; scheduled verification;
update poverty definition regularly; provide unique
beneficiary IDs to monitor health status across PA
areas)
– Coordination across various programs minimizes
redundancy of benefits and fully optimizes the
delivery of health services
– The Urban Health Care Services Coordination
Committee could include regular follow-ups and
updates on implementation of various programs
41
Thank you.
42
Regions of Common Support
Children
Women
UHS
Baseline
UPHCP
Endline
43
Health Status in Baseline and Endline
Health Indicator
Neonatal Mortality
Infant Mortality (1q0)
Child Mortality
Child Mortality (4q1)
U5MR (5q0)
Stunting Prevalence
Child Nutrition Wasting Prevalence
Underweight Prevalence
Diarrhea Prevalence
Diarrhea & ARI
ARI and Fever Prevalence
ANC Coverage
ANC/PNC
PNC Coverage
Modern Contraceptive Use
Sexually
Transmitted
AIDS Awareness
Infection
RTI Prevalence on women
PA
2006 2012
36.7 27.2
51.0 39.0
15.7 10.4
65.9 49.0
42.3 40.8
16.7 19.6
36.5 32.8
6.3 2.2
11.6 3.6
51.8 59.7
33.6 50.8
43.9 60.8
91.3 91.9
26.1 3.2
NPA
2006 2012
33.2 28.3
47.6 39.7
9.3 10.4
56.4 49.6
39.3 38.1
13.5 20.4
30.9 30.7
5.4 3.0
10.0 4.4
52.7 56.2
30.8 48.4
44.1 58.7
92.8 92.0
24.4 3.5
44
Reducing Child Mortality
• The project achieved its target of reducing U5MR by 15%
• All child mortality indicators were reduced in project areas by
more than 15% between 2006 and 2012
65.9
51.0
49.0
39.0
36.7
27.2
-33.8%
15.7
-25.6%
U5MR
(0-5y)
-25.9%
Neonatal Mortality
(0-27d)
Baseline
-23.5%
Infant Mortality
(0-1y)
10.4
Child Mortality
(0-4y)
Endline
Source: UPHCP-II Project Completion Report
45
Reducing Child Malnutrition
• The target of reducing child malnutrition by 10% between
baseline and endline was achieved only for underweight
children
• Incidence of wasting unexpectedly increased by 17.4% in
project areas
42.3
40.8
36.5
17.4%
-3.5%
16.7
Stunting - cumulative effect of
chronic malnutrition
(h/a)
19.6
Wasting - acute or recent
nutritional deficit
(w/h)
Baseline
32.8
-10.1%
Underweight - overall indicator
of nutritional health
(w/a)
Endline
Source: UPHCP-II Project Completion Report
46
Reducing Diarrhea and ARI Prevalence
• ARI and fever incidence experienced a large reduction from
14.2% in the baseline to 3.7% in the endline
• Childhood diarrhea incidence was reduced from 6.3% in the
baseline to 2.2% in the endline
36.7
-73.7%
20.7
14.2
-65.9%
-43.5%
6.3
3.7
2.2
Fever
ARI
Baseline
Diarrhea
Endline
Source: UPHCP-II Project Completion Report
47
Improving Maternal Welfare
• UPHCP achieved various targets for maternal health
in project areas between baseline and endline
43.9
60.8
Target
-20%
60.6
26.1
Target
60% endline
38.5%
Modern Contraceptive
Use
86.8
Target
25%
-87.9%
Baseline
59.7
Target
60% endline
43.2%
15.2%
Knowledge on AIDS
Avoidance
ANC Coverage
3.2
Reproductive Tract
Infection
51.8
Endline
Source: UPHCP-II Project Completion Report
48
ANC, PNC and Attended Births
• Pregnant women who had at least three ANC visits 59.7% in
the endline is slightly short of the 60% project target
• The proportion of women who received PNC 50.8% in the
endline is above project target of 50%
• The proportion of attended births either increased by
51.2%, more than the project target of 10%
71.9
59.7
51.8
15.2%
50.8
33.6
51.2%
51.5%
ANC
(at least
3 visits)
47.5
PNC
Baseline
Attended Births
Endline
Source: UPHCP-II Project Completion Report
49
Breastfeeding
• Reduction in breastfeeding may be due to emerging economic
opportunities in the urban areas, as working women will less
likely to breastfeed their children
• Complimentary breastfeeding 6 months onwards increased
experienced a 98.5% surpassing the target of 52%
98.0
96.6
92.4
-5.7%
86.2
73.1
48.7
17.9%
98.5%
Ever breastfed
Breastfed within 1 day of birth
Baseline
Endline
Source: UPHCP-II Project Completion Report
Complementary breastfeeding
50