PEPFAR OVC Evaluation:
How good at doing good?
(GH TECH/USAID 2011)
Prof Lorraine Sherr UCL London UK
Acknowledgements to Miriam Zoll (Co-author), Natasha Croome,
Adele Daniels and Jaya Karira
AIDS 2012 - Turning the Tide Together
Task: Summarize and Analyze
PEPFAR OVC Evaluations
• 18 Evaluations
• 22 Programs
• 9 Countries – Kenya, Tanzania, Uganda, Mozambique, Rwanda, Namibia,
Zambia, South Africa and Haiti
• 8 basic needs categories: Food/Nutrition, Health, Education, PSS, Economic
Strengthening, Child Protection, Shelter and HIV Prevention
• Critique methodology and summarised findings.
•
http://resources.ghtechproject.net/content/pepfar-ovc-evaluation-how-good-doing-good
Evaluation – is it ever expedient?
1. Systematic reading by reviewers.
2. Coding - extract themes agreed by
two reviewers.
3. Sub-analysis of costing.
4. Methodology adequacy criteria.
5. Seven basic needs categories.
6. Recommendations for future
evaluation and evidence- based
policy.
Variable
1
2
Agreement
Refusal Rates
2
2
100%
Random
Selection
2
3
66.7%
Gender
4
4
100%
Significance
4
4
100%
Comparison
Group
3
3
100%
Cross Sectional
5
4
66.7%
Descriptive Data
7
7
100%
Double coding 6 reviews (Kappa =.83)
Evaluations Surveyed
1.
2.
3.
4.
5.
Brewster, Lee, et al, Catholic Relief Servies' OVC PEPFAR Programs: A Midterm
Multicountry Program Evaluation
Nyangara, F., Obiero, W., Kalungwa, Z., Thurman, T., Community-based Psychological
Intervention for HIV-Affected Children and their Caregivers: Evaluation of the Salvation
Army's Mama Mkubwa Program in Tanzania, USAID, AXIOS, Measure Evaluation, March
2009
Thurman, T., Hutchinson, P., Ikamari, L., Gichuhi, W., Murungaru, K., Nyangara, Community
Education and Sensitization as an OVC Care and Support Strategy: Evaluation of the
Integrated AIDS Program-Thika in Kenya, USAID, University of Nairobi Population Studies
and Research Institute, Measure Evaluation, March 2009.
Thurman, T., Rice, J., Ikamari, L., Jarabi, B., Mutuku, A., Nyangara, F., An Evaluation: the
Difference Interventions for Guardians Can Make: Evaluation of the Kilifi Orphans and
Vulnerable Children Project in Kenya, USAID, Measure Evaluation, University of Nairobi
Population Studies and Research Institute, March 2009.
Nyangara, F., Kalungwa, Z., Obiero, W., Thurman, T., Chapman, J., Promoting a Home-based
Program Model for Supporting Children Affected by HIV/AIDS: Evaluation of Tumaini
Project in Iringa Region, Tanzania, USAID, AXIOS, Measure Evaluation, March 2009
Evaluations Surveyed
6. Swarts, Brian, et al, Empowering Better Care: Report on Economic Strengthening
for OVC Caregivers in Uganda, Summary of Findings, The Salvation Army World
Service Office, WORTH, USAID and PACT, April 2010
7. Hutchinson, P., Thurman, T., Analyzing the Cost Effectiveness of Interventions to
Benefit OVC: Evidence from Kenya and Tanzania, USAID, Measure Evaluation, April
2009
8. Nyangara, F., Obiero, W., Effects of a Community-Focused Approach Supporting
the Mosts Vulnerable Children: Evaluation of SAWAKA Jali Watoto Program in
Kagera, Tanzania, USAID, Measure Evaluation, August 2009
9. Thurman, T., Hoffman, A., Chatterji, M., Brown, L., A Case Study: Kilifi Orphans and
Vulnerable Children Project, Measure Evaluation and Catholic Relief Services with
USAID, March 2007
10. Nyangara, F., Thurman, T., Hutchinson, P., Obiero, W., Effects of Programs
Supporting OVC: Key Findings, Emerging Issues and Future Directions
Evaluations Surveyed
11. Murombedzi, Cornelius, Mugumya, Firminus, Opportunity International/Habitat for Humanity:
Evaluation of Sustainable Income and Housing for Orphans and Vulnerable Children in Africa,
International NGO Training and Research Centre, August 2009
12. Larson, B., Hamazakaza, P., Costella, J., Laurence, S., Costing of OVC Service Delivery in South Africa
and Zambia, Center for Global Health and Development and Health and Development Africa,
USAID/Project SEARCH, September 10, 2010
13. Jaden Consultancies, Weaving the Safety Net: Mid-Term Evaluation, Final Report, Christian Children's
Fund and USAID, March 2008
14. Jaden Consultancies, Weaving the Safety Net: Mid-Term Evaluation, Z Scores Christian Children's Fund
and USAID, March 2008
15. Scott, N., Pfaff, M., Semrau, K., Leyenaar, J., Suojanen, A., Mugala, T., Simon, J., End of Project
Evaluation Report: Faith-based Regional Initiatives for Orphans and Vulnerable Children (FABRIC),
USAID/Project Search, boston University OVC-CARE Project, Family Health International, August 2010
16. Yates, DeeDee, Finding the Inflection Point in Health Education and Economic Strengthening: Final
Evaluation of Project HOPE's Track 1 OVC Program Sustianable Strengthening of Families and Orphans
and Vulnerable Children Mozambique and Namibia, Cooperative Agreement #: GPO-A-00-05-0001900, May 31, 2010
17. Capps, J., Bukuluki, P., Davis, S., Swarts, B., Bailey, B., Wakubona, S., Sustainable Community Support
for Orphans and Vulnerable Children in Tanzania and Uganda, Final Salvation Army (TSA) Uganda, June
2010.
18. RAPIDS Evaluation Final Report 2005–2009 Key Findings; Population Council, Africare & Rural Net
Associates, July 2009, Revised 2010
Overview and data abstraction
3
HIV OVC
12
PEPFAR com
18
Number of studies
7
Theory
2
Comparison groups
8
Ethical approval
12
Cross section
1
Power calc
2
Baseline data
4
Consecutive selection
3
Random selection
7
Refusal
0
5
10
15
20
Age Issues
Focus on Adolescents
•
•
•
•
•
•
•
•
8-14 (6)
0-4; 0-18 (2)
5-11 (1)
7-12; 7-15 (2)
13-18; 13-17 (2)
12-19; 12-17 (2)
N/A – (2 costing studies)
3 with unclear age
Grading methodology
• STRONG:
"strong".
No study used a RCT design in any form none graded as
• ADEQUATE: Nine studies graded "adequate" some form of before/after, case control, and post
intervention comparison design.
• 1 qualitative (with good design)
• Two costing studies graded as adequate - utilised
consistent audit/costing data.
• WEAK: Six studies were graded as "weak" and
conclusions/ findings should be viewed with caution.
Quantitative vs qualitative methodology
The war is over
•
•
•
•
13 used/included quantitative
11 used/included qualitative
Qualitative is no excuse for poor methodology.
If qualitative used, quality standards on how sampled, how
recorded, how analysed – coding and themes – not just the use of
quotes (which is what a number did)
Complex interventions
• Most programmes are complex – mechanisms
of impact difficult to align.
Critical Absence of Baseline Data
Data Handling
•
•
•
•
Vast number of data points (15 Scott)
Simplistic reporting (failure to do complex analysis)
Percentages versus statistical tests
Power calculations
Theory
• Role played (low)
• Recourse to theory
(eg knowledge does not equate to behaviour – why focus and
measure of knowledge eg 9 Nyangara looks at knowledge
gaining, some look at attendance at knowledge imparting
(not even if they got the knowledge).
• Generalisability
• Understanding mechanisms
Two forms of Evaluation –
Against Pepfar Goals or against Programme goals
Methodological Trends
• Baseline: PEPFAR wants evidence-based approaches but most evaluations had no baseline,
which is the only way to monitor and track impact of dollars and labor invested.
• Quantitative: Several of the evaluations were able to tell us how many OVC received
services, how many caregivers had taken out a loan and repaid it.
• Standardized Indicators – good methodological reasons for using validated indicators
(See Tulane group for good example).
• Outcome •
How do you operationalize efficacy? (eg Yates 16 – use attendance at Health
education as the indicator, but what if you attended and slept through?
Gender: many mention few analyze
• Sampling – good methodological reasons for using excellent sampling and
reporting refusal rates.
• Staff training in each year were a relatively large share of program costs. If such
expenses are considered investments in human capital, the training would provide
services to the project over more than one year. If there is substantial staff turnover,
prior investments in training would not provide services to the project in the future
• Dose Issue - Exposure dose seems to be an emerging issue.
Home visiting is not
a standardised commodity. Need to harmonise concepts in description and quantity
standards
Findings by Basic Needs:
Food/Nutrition - 14 / 18
• 9/14 evaluations focused on food/nutrition found OVC
households to be moderately or severely food insecure.
• Improvements shown, but questions raised about
dependence and sustainability.
Health Care ––11/18
• 8/11 studies demonstrated improved health
outcomes
• Specifically notable when combined with economic
strengthening and high dose/follow through
Education–– 15/18
• Enrollment effect
• Supplies positive mental health effect.
• 0 / 14 identified gender imbalances in school
enrollment, although one reported that more girls
than boys miss more than two days of school.
• Cost – resentment
• Ongoing funding need until government shoulders
this.
Psychosocial––13/18
• 10/ 13 home- visits clearly benefited OVC but
not always guardians
• Negatives associated with infrequent visits, low meeting
attendance at Kids' Clubs
• Targeting carers indirectly benefits OVC
• Volunteers overwhelmed by caseloads and ill equipped to
provide HB PSS counseling, particularly to adolescents and
caregivers.
Economic Strengthening–9/18
• PEPFAR's investments in local savings and loan schemes are
proving to be effective in raising many OVC households out of
poverty
• Economic strengthening intersects significantly in other core
areas of basic need, including food/nutrition, health,
education, psychosocial support and shelter. Thus, economic
strengthening remains one of PEPFAR's most potent and
effective strategies for improving the lives of OVC and their
families
• Costing - Great variation for example
– Kids Clubs:
$1.96 - $14.08
– Home Visiting:
$3.55 - $89.68
– Gardening
$24.00 - $89.00
– Food
$1.39 - $32.36
– Educational Support:
$17.23 - $141.32
Child Protection––16/18
• Measurable role home-based visits play in
reducing child abuse, stigma and social isolation
• Psychosocial interventions targeting guardians
often benefited OVC as well
• Children whose homes had been visited by a
volunteer > birth certificate (36% of all children
had birth certificates)
• Home visitor training associated with improved
outcome
Shelter––6/18
• Shelter was the least
prioritized basic need in all evaluations.
• Provision of shelter made significant positive
differences in multiple areas, including
protection, psychosocial support and health
HIV Prevention––15/18
• Children's HIV knowledge appeared to increase in 7 out of 13
evaluations.
• Some interventions appeared to be more thorough than
others.
• School-based HIV prevention activities were part of 5 out of
13 prevention initiatives and were estimated to be cost
effective.
Summary
• Quality Standards. Define PEPFAR’s quality standards for evaluations?
•
•
•
•
Methodology? Standardization? The very mention of "PEPFAR" should be
synonymous with quality
Costing - Great variation
Volunteers - costly cyclical patterns are troublesome to program continuity and
quality, heavy reliance, questions of quality.
Food 3 / 13 focused on food - found > 80% of household samples were severely
food insecure (86%; 86.3%; 88.5%).
Kids’ Clubs – Mixed findings. Often volunteers poor results on issues of stigma
and discrimination.
• One Stop Shop - Harmonizing and integrating complex PEPFAR-funded
OVC programs can yield better and multiple outputs
• An emergency response should be saved for emergencies
• Evidence- based programming should be a priority - There was weak
evidence of gains and impacts in the five core areas, but with a number of caveats
The way forward
• Updated PEPFAR OVC Guidance (2012) stipulates: “Allocate
at least 10% of program budgets to M&E”
• PEPFAR supporting the development and implementation of:
– Methodology to track impact of child welfare and protection systems
strengthening
– Standardized quantitative program evaluation tool to measure child
and caregiver/household outcomes across countries at population
level (Will be introduced in Nigeria, Mozambique)
Evidence-Based Policy?
How Good at Doing Good?
Reviewing the PEPFAR OVC Factor
and
PEPFAR HKID Portfolio Review:
Children of the HIV/AIDS Epidemic
Reports available at:
http://resources.ghtechproject.net