Recommendations by the
Data Working Group (DWG) to the
PEPFAR Scientific Advisory Board (SAB)
SAB Members Participating in the Data Working Group:
Ward Cates, Kevin DeCock (also USG), Mead Over, Mary Jane Rotheram, Sten Vermund,
Brian Williams, Robin Wood
US Government Participants in the Data Working Group:
Paul Bouey, Rob Lyerla
Presented October 3, 2012
The findings and recommendations in this presentation are a compilation of views discussed in the working group
and endorsed by most participants; they are neither the unanimous consensus of the DWG participants nor official
positions of the US government.
DWG’s Draft Recommendations for
Strengthening PEPFAR Data Functions
1. Establish and maintain a PEPFAR public access knowledge
portal
2. Strengthen, streamline and publicly disclose PEPFAR’s
collection and management of key program indicators
3. Establish, collect and publicly disclose activity-based budget,
expenditure and cost data
4. Require each future grantee and contractor to submit a
“Data Management Plan”
2
Outline of This Presentation
• Principles
• PEPFAR practices & recommendations
1.
Public access portal
2.
Strategic information system
3.
Activity-linked financial reporting
4.
Data management by grantees and contractors
• Summary list of recommendations
3
PRINCIPLES
Benefits of Open Government Data
• Outsource creativity to improve public services:
“most of the world’s smartest people don’t work for you”
(Sun co-founder Bill Joy)
• Improve accountability of government
• Increase trust in government through transparency
• Save time / expenditure of answering citizens’ data requests
• Enable government to use own data
• Open data is a public good and can generate jobs: e.g. $100B
market for weather data
• Show gaps in data collected
Apologies to Peter Speyer, IHME, who composed this to summarize the findings of the “International Open Government
Data Conference”, July, 2012
The DWG’s Principles of Data
Collection and Management
• Data should strengthen US government program management
• Data are a public good
• Transparency and accountability
• Standardization and comparability through a common data
“platform”
• Replicability (“Crowd sourcing”)
6
http://www.whitehouse.gov/open/documents/open-government-directive
Open Government Directive
“The three principles of transparency, participation, and collaboration form the
cornerstone of an open government. Transparency promotes accountability by
providing the public with information about what the Government is
doing. Participation allows members of the public to contribute ideas and
expertise so that their government can make policies with the benefit of
information that is widely dispersed in society. Collaboration improves the
effectiveness of Government by encouraging partnerships and cooperation
within the Federal Government, across levels of government, and between the
Government and private institutions.”
1.
2.
3.
4.
Publish Government Information Online
Improve the quality of USG information
Create and Institutionalize a Culture of Open Government
Create an Enabling Policy Framework for Open Government
7
DWG’s Recommendations Advance the Principles
DWG’s Principles
DWG’s Proposed
Recommendations
Data should
strengthen
program
management
Maintain a
knowledge
Index
Strengthen
indicators
Data are a
public good
Transparency
X
X
Comparability
(Platform)
Replicability
X
X
X
X
X
Collect and
disclose
activity costs
X
X
X
Require data
management
plans
X
X
X
X indicates that a given recommendation furthers the indicated principle.
8
PRACTICES AND
RECOMMENDATIONS
Categories of Data Supported by PEPFAR
SURVEYS & SURVEILLANCE
• Survey
– Community
– Household
– Individual (interview, biomarkers)
•
Surveillance
–
–
–
–
National
Regional/District/Community
Facility
Individual (interview, biomarkers)
PROGRAM RESULTS
• Patient Level
• Facility Aggregate
• Partner Aggregate
• Country Aggregate
• Multi-country PEPFAR Totals
PROGRAM COSTS
• Planned funds
• Estimated costs
• Expenditures
– By:
•
•
•
•
•
•
Unit
Patient Level
Facility Aggregate
Partner Aggregate
Country Aggregate
PEPFAR Totals
RESEARCH
• Community
• Household
• Individual
– Interview
– Biological
– Program/Services
10
Note: Selected categories noted here.
An Overview of PEPFAR’s Current
Practice in Four Areas
1. Public access portal
2. Strategic information system
3. Activity-linked financial reporting
4. Data management by grantees and contractors
11
PEPFAR’s Public Access Portal
• www.PEPFAR.gov provides
– Program policies, and high level budgets and
expenditure
– Fact sheets
– Partial list of relevant research papers
– Short summaries of findings from undisclosed data
• PEPFAR’s web portal does not compare favorably
with other portals for government information
1.
12
Public Access Portal
The Scope of PEPFAR’s Annual Public Reporting
Lags Behind Other Agencies…
70
Number of Pages/Indicators,
2012 Annual Report
60
60
President's Malaria Initative (PMI)
50
PEPFAR
40
30
20
14
10
7
5
0
Pages
Indicators
1.
13
Public Access Portal
…And Has Declined Over Time
250
90
200
Pages
70
60
150
50
40
Indicators
100
30
20
50
Number of Indicators
Number of Pages
80
10
0
0
2005
2006
2007
Source: Publicly available PEPFAR reports from www.PEPFAR.gov
2008
2009
2010
2011
2012
1.
14
Public Access Portal
PEPFAR Ranks #29 of 72 Donors in the
2012 Aid Transparency Index
…
…
1.
15
Public Access Portal
Other Public Access Portals Are Better
1.
17
Public Access Portal
Public Access Portal:
RECOMMENDATION #1
• Establish and maintain a PEPFAR public access
knowledge portal which:
– Expands the scope and granularity of disclosed PEPFAR
program indicators
– Provides a graphical interface to the indicators, enabling
the user to browse the data, posing her own questions
– Provides a “wiki” platform to which PEPFAR grantees,
contractors, employees, partners, etc. would be
encouraged to contribute data, “grey literature” reports
and research papers at more granular levels
1.
18
Public Access Portal
PEPFAR’s Strategic Information System
• Informed by a lengthy and detailed 2009 study: “Next
Generation Indicators Reference Guide”
(http://www.pepfar.gov/documents/organization/81097.pdf )
• To arrive at OGAC, any indicator must complete an arduous
upstream “swim”
• OGAC’s indicator management policy compares favorably with
other bilateral and multilateral agencies
2.
19
Strategic Information
PEPFAR Direct Indicators
• Essential Indicators
•
•
•
• Recommended Indicators
Considered high importance and inherently
necessary to track progress of HIV/AIDS
programs
Must be collected by all applicable programs
Subject to audit
Essential (35)
Reported to
PEPFAR HQ (30)
•
•
•
Recommended but not required for
program managers who need additional
information for program management
Not reported to headquarters
Not subject to audit
Recommended
(~30, plus component
parts of some of the
35 essential indicators)
Reported
to Congress
(7)
Source: Next Generation Indicators Reference Guide (2009). Minor revisions not captured.
2.
20
Strategic Information
ART Enrollment and Retention Indicators:
Typical data flow up from the field
Only 7 Indicators Are
Publicly Disclosed
HQ-Based TWGs, SI Advisors, and SI Liaisons
Final Data for Public Release or
Congressional Reporting
(if Applicable)
Country-Level TWGs and SI Liaisons
CDC/USAID/Etc. AgencySpecific Reporting Chains
Final Country Estimates
~30 Essential OGAC
~270 Other
Central PEPFAR CountryLevel Reporting System
Prime Partners
Sub-Partners
Facilities
Source: Correspondence with PEPFAR Officials
Country Data Entry into
FACTSINFO System
(OGAC)
•
•
•
•
•
•
•
•
Annual
Program
Review
(APR)
Reporting to headquarters is either
Information
theiscontractually
total
number of
At
this partners
stage,on
data
considered
Prime
are
Sometimes
required
onnecessary
a semi-annual
to de-duplicate
or annual
reporting
facilities
istargets
not
currently
“clean”
and
valid
for
to release
as
obligated
to
submit
to COPs,
results(SAPR
basis
across
oragencies
APR), depending on
collected
latest
Worlds
AIDS either
day
and toresults
reporton
against
targets
USGindicator
the
country teams maintain a
semi-annually or annually
centralized
In
some countries,
reporting
meetings
system with
for data
Sometimesdata
confusion
adhering
Internally,
is usedinfor
programto
collection
Global
After
data
Fund,
is reported,
DfID, WHO,
it isetc.
subject
to share
to
definition
of “direct”
support
analysis:
Why
do we
see
theseor
Activity managers
from
country-level
numbersand
cleaning
andanalysis
help validate
by Strategic
national
specific
timeframe
for
the conduct
indicator
numbers?
What
the
TWGs make
sureare
partners
results
Information
Advisors and Liaisons
programmatic
quality checks implications?
(SIAs and SILs)
2.
21
Strategic Information
Data Collection and Review:
Annual Timeline
May
Semi-Annual
Program Results
(SAPR)
January
Indicator Target
Reviews
Indicator Results
Reviews
Indicator Results
Reviews
March
November
Country
Operational Plans
(Targets)
Annual Program
Results (APR)
Source: Correspondence with PEPFAR Officials
2.
22
Strategic Information
Data Completeness:
ART Retention, Selected Countries
140,000
120,000
Initiating ART, FY2010
100,000
80,000
60,000
40,000
20,000
-
Source: FY2011 Indicator Data for ART Retention; FY2010
Indicator Data for Initiating Treatment
2.
24
Strategic Information
Data Completeness:
ART Retention, Selected Countries
140,000
% Completion
0%
Reported Denominator
120,000
73%
Initiating ART, FY2010
100,000
78%
100%
80,000
18%
79%
60,000
59%
40,000
99%
179%
20,000
133%
159%
106%
0%
-
Source: FY2011 Indicator Data for ART Retention; FY2010
Indicator Data for Initiating Treatment
2.
25
Strategic Information
Other Indicators
• Some work well: treatment enrollment, PMTCT, …
• Some of the most important indicators are challenging to
collect, i.e. key population coverage
2.
28
Strategic Information
500,000
450,000
Number Reached
400,000
350,000
300,000
Key Population Coverage:
Number of MARP Reached
Country Operational
Plan (COP) Target
Annual Performance
Result (APR), Total
APR Result (MSM,
CSW, & IDU Only)
250,000
200,000
150,000
100,000
50,000
0
2.
29
Strategic Information
Strategic Information:
RECOMMENDATION #2
• Strengthen, streamline and publicly disclose PEPFAR’s
collection and management of key program indicators by:
– Conducting an analytical study of all 300+ program indicators
requested of its in-country partners
• Study should measure the benefits (epidemiological and managerial) and the costs
(impediments, time and money) of each indicator at each level of the reporting ladder in each
of several recipient country contexts
– Ranking the indicators separately by their benefits and costs
• Aspects of indicator benefits: Proximity to a health outcome , completeness , accuracy,
epidemiological relevance of denominator
• Aspects of indicator costs: Man hours of effort, difficulty of hiring qualified data personnel,
negotiations with partners, data cleaning and reconciliation, audits
– Using these rankings to guide a process of strengthening and
streamlining the indicators, with a view toward maximizing the benefit
per dollar of the retained set of indicators
2.
30
Strategic Information
Financial Reporting
• PEPFAR has taken aggressive steps to better understand the
cost-effectiveness of its programs: (http://www.pepfar.gov/smart/index.htm)
– PEPFAR’s overall budget is reported to the public annually
(http://www.pepfar.gov/documents/organization/183974.pdf )
– Through its ART Costing Project, PEPFAR has released periodic reports
on estimated costs of treatment (most recently in February 2012)
(http://www.pepfar.gov/documents/organization/188493.pdf )
– In July/August of this year, PEPFAR released its “Report on Pilot
Expenditure Analysis of PEPFAR Programs in Six Countries”
(http://www.pepfar.gov/documents/organization/195700.pdf )
– Pilot currently expanding, with institutionalization of expenditure
analysis into routine annual reporting expected in 2013 (ibid.)
3.
31
Financial Reporting
Financial Reporting
Examples of questions the user should be
able to explore in an open data base:
• What are the components of
“program management expenses”?
Overhead percentages?
• What is the variation in program
management expenses across
partners?
• Can we link variation in expenditure
on program management or other
cost components to variation in
performance on key indicators, i.e.
retention?
3.
32
Financial Reporting
Financial Reporting:
RECOMMENDATION #3
• Establish, collect and publicly disclose activity-based budget,
expenditure and cost data by:
– Regularly repeating its surveys of the activity-linked unit cost of antiretroviral treatment and disclose the properly anonymized underlying
data
– Expanding its activity-linked unit cost analysis to other PEPFAR
supported HIV/AIDS services
– Releasing the properly anonymized activity-linked budget, expenditure
and cost data at the unit of observation, which is the partner or
program
– Disclosing the anonymized activity-linked financial data on PEPFAR’s
open web portal, freely browsable by the public
3.
33
Financial Reporting
Elements of a Data Management Plan
Element
Description
Data Description
Nature, scope and scale of data collected or generated
Access & Sharing
How and why data will be shared and archived
Metadata
Discussion of metadata standards used for data collection
Intellectual
Property Rights
Establish the legal right to access and use data
Ethics and Privacy
Privacy protection and ethical issues (IRB)
Format
Data generation, maintenance and access format(s)
Archive &
Preservation
Procedures for long-term archive and preservation, including life
cycle succession plans for outdated technology
Storage & Back-up
Physical and cyber resources and facilities
Security
Technical and procedural protections with enforcement protocol
Responsibility
Names of individuals responsible the project data management
…
A number of optional elements may also be included
4.
34
Data Management
Data Management Plans:
RECOMMENDATION #4
• Require each future grantee and contractor to
submit a “Data Management Plan” by:
– Developing a prototype of a Data Management Plan, including a
model plan for each of two or three “typical” potential grantees or
contractors
– Drafting guidelines to suggest how the applicant should develop a data
management plan and how the applicant could request an exception
to or a postponement of any specific requirement
– Consider piloting prototype DM on existing PEPFAR funded research
projects
4.
36
Data Management
SUMMARY OF
RECOMMENDATIONS
DWG’s Draft Recommendations for
Strengthening PEPFAR Data Functions
1. Establish and maintain a PEPFAR public access knowledge
portal
2. Strengthen, streamline and publicly disclose PEPFAR’s
collection and management of key program indicators
3. Establish, collect and publicly disclose activity-based budget,
expenditure and cost data
4. Require each future grantee and contractor to submit a
“Data Management Plan”
38