M&E for Accountability
&
Global Progress Tracking
Monitoring, Operations Research and Evaluation Team
UNAIDS/Geneva
April 2007
ORIENTATION
2008 UNGASS Country Progress Report
- Benefits of UNGASS for M&E strengthening
- UNGASS Guidelines
 Purpose and use
 Core indicators
 Data sources & data vetting
 Reporting process
INTRODUCTION
• Where does UNGASS fit in M&E system?
• UNGASS Reporting Timetable
• Use of 2006 UNGASS data
– Global use
– Country use
WHERE DOES UNGASS FIT IN
A COMPREHENSIVE M&E SYSTEM?
Benefits of UNGASS reporting for M&E strengthening:
• Brings partners together around a core set of
indicators
• Helps identify data gaps & data quality issues
• Stimulates the establishment of a centralised
database of HIV indicator data
• Provides a de facto bi-annual report on the epidemic
and the response
WHERE DOES UNGASS FIT IN
A COMPREHENSIVE M&E SYSTEM?
Program
Improvement
Share
Data with
Partners
Reporting/
Accountability
[UNGASS, other]
UNGASS REPORTING TIMETABLE
REPORT REQUIRED?
2003
Yes
2004
No
2005
Interim report from a subset of countries
2006
Yes
2007
No
2008
Yes
2009
No
2010
Yes
GLOBAL USE OF 2006 UNGASS DATA
• Global Progress Report
• Report to the Secretary General (as required in the
Declaration of Commitment)
• Used for tracking the Millennium Development
Goals
COUNTRY USE OF 2006 UNGASS DATA
1. National report on the epidemic and response
• informs programmatic decisions
• guides reviews of the National Strategic Plan
• informs resource mobilisation efforts (e.g.
GFATM proposal development)
• is used as an advocacy tool
• is used to report progress on Declaration of
Commitment to Senior Government Officials
COUNTRY USE OF 2006 UNGASS DATA
2. M&E system strengthening
• convenes relevant partners to address a
particular M&E need
• identifies data gaps
• assists in planning data collection efforts
• identifies data weaknesses
• helps mobilise resources for system
strengthening
• stimulates evaluation questions
LESSONS LEARNED FROM
2006 UNGASS REPORTING
• Challenges in 2006 reporting
• 2006 reporting from MENA
• Lessons learned from 2006 reporting
– UNAIDS/Geneva-level
– Country-level
CHALLENGES IN 2006 REPORTING
137 countries submitted a Country Progress Report
• 10% reported on all indicators relevant to epidemic
• 51% used CRIS to report indicator data
Some basic data quality issues including
– incorrect indicator or incorrect indicator calculation
– incomplete data (e.g. only 20% of indicators reported
included the required disaggregation)
– illogical values
– discrepancies between data in CRIS and Report
Percent Range and Median of Indicators Reported
by Countries by Region - UNGASS 2006 Report
IMPORTANCE OF DISAGGREGATED DATA
Safe Injecting & Sexual Practice in IDU by Sex;
2006 UNGASS Report
80
70
Males
Females
60
Percent
50
40
30
20
10
0
Bangladesh
Indonesia
Thailand
Arm enia
Belarus
Moldova,
Republic of
Macedonia
LESSONS LEARNED FROM 2006 REPORTING
UNAIDS/Geneva-level
The following inputs are paramount for improving
the quality of the data obtained from countries:
– Revise UNGASS indicators, where necessary
– Provide clear and specific Guidelines
– Provide user-friendly, flexible CRIS software
– Provide training on UNGASS indicators &
reporting process
– Provide technical support, where needed
LESSONS LEARNED FROM 2006 REPORTING
Country-level
• Ensure correct understanding of indicator definition/operands
• Report all required indicators
• Identify and document data sources
• Provide all data and information needed for each indicator
• Cross-validate data (comparison with previous years,
comparison with alternate sources, etc.)
• Use CRIS for indicator reporting
• Double-check for data entry errors
• Submit timely
LESSONS
LESSONS LEARNED
LEARNED FROM
FROM 2006
2006 REPORTING
REPORTING
Country-level
1.
2.
3.
4.
Address data gaps early
Seek to enhance data quality
Follow all the steps in the reporting process
Double-check accuracy and obtain consensus
on all indicator values at country level before
submission
ORIENTATION ON
THE UNGASS GUIDELINES
• Purpose
• Target audience
• Contents
• Principles and process of Guidelines revision
• Use of Guidelines
• Provide essential information on construction
of core indicators for reporting
• Ensure transparency of the reporting process
in-country
• Ensure consistency of information across
countries for global progress analysis
• Government sector
– including the National AIDS Committee, Ministries
(e.g. Ministry of Health, Ministry of Finance, Ministry
of Education)
• Nongovernmental sector
– including nongovernmental organizations, faith-based
organizations, people living with HIV, private sector
• UN co-sponsors and donor agencies
CONTENTS OVERVIEW
• Introduction
p. 9-13
• Implementation at National Level
p. 14- 20
• Core Indicators
p. 21-82
• Appendices 1 - 9
p. 83-139
APPENDICES 1 - 9
1. Changes to Core Indicators
2. Country Progress Report template
3. Methodology used for the coverage survey
4. Consultation/preparation process for the
Country Progress Report
5. Reporting schedule for Core Indicators
6. National Funding Matrix 2007
7. National Composite Policy Index (NCPI) 2007
8. Sample check-list for Country Progress Report
9. Selected bibliography
•
•
•
•
•
•
•
Consensus-driven process
Transparency
Consistency
Comprehensiveness
Quality
Integration within the National M&E system
Feasibility
Process of Guidelines Revision
•
•
•
•
Debriefing process
2006 Indicator performance
M&E Reference Group review
Technical working groups for each indicator
• Familiarize yourself with the current Guidelines
& the changes since the previous round
• Disseminate the Guidelines and use them to
develop a collaborative in-country process
• Ensure that the correct definitions are used for
construction of the indicators & refer to Further
Information, as needed
• Use all Appendices
• Ask for clarification / assistance, as needed
CORE INDICATORS
• Overview of Core UNGASS Indicators
• Which UNGASS indicators to report
• Changes since the 2005 UNGASS Guidelines
• Core UNGASS Indicators
WHAT ARE
CORE UNGASS INDICATORS?
Quantitative variables which provide simple and
reliable ways of measuring progress towards
achieving the Declaration of Commitment on
HIV/AIDS
CORE UNGASS INDICATORS
National Indicators (N=25)
Four categories:
1. National commitment and action
2. National programmes
3. Knowledge and behaviour
4. Impact
Global Indicators (N=4)
CORE UNGASS INDICATORS:
NATIONAL INDICATORS
Four categories:
1. National commitment and action
2. National programmes
3. Knowledge and behaviour
4. Impact
CORE UNGASS INDICATORS:
NATIONAL INDICATORS
National indicators are important for two reasons:
1. They are used to evaluate the effectiveness of
the national response
2. They are used to provide information on
regional and global trends
MILLENIUM
(MDGs)
MillenniumDEVELOPMENT
Development GOALS
Goals (MDG)
There are 8 MDGs:
– Goal 6: Combat HIV/AIDS, Malaria and other
diseases
• Target: Have halted by 2015 and begun to reverse
the spread of HIV/AIDS
• UNGASS indicators are used to monitor progress
UNGASS Indicators to monitor MDG-6
17) Percentage of women and men
• Four
of attendance
the national aged
indicators
alsothan
12) Current
school
15–49 whoare
had more
among orphans and
one sexual partner in the past
among non-orphans aged 10-14
12 months who report the use
of a condom during their
last sexual intercourse
13) Percentage of young
women and men aged 15-24
who both correctly identify ways
of preventing the sexual transmission
of HIV and who reject major
misconceptions about
HIV transmission
22) Percentage of young
women and men aged
15–24 who are HIV infected
Global Indicators
1) Amount of bilateral and
multilateral financial flows
(commitments and disbursements)
for the benefit of low- and
middle-income countries
2) Amount of public funds for
research and development of
preventive HIV vaccines
and microbicides
3) Percentage of transnational
companies that are present
in developing countries and
that have workplace HIV
policies and programmes
4) Percentage of international
organizations which have
workplace HIV policies and
programmes
Which UNGASS indicators
should countries report?
• No distinction between Generalized Epidemic
Indicator Set & Concentrated /Low-prevalence
Epidemic Indicator Set
• Most national indicators are applicable for all
countries
• Countries are expected to "know their epidemic"
Which UNGASS
UNGASS indicators
indicators
Which
should
countries
report?
(continued)
should
countries
report?
When countries choose not to report on a
particular indicator, an explanation needs to be
provided:
Either,
• The indicator is not applicable to the epidemic
• The indicator is applicable, but no data is available
Which UNGASS indicators
should countries report? (continued)
The indicators have to be constructed including the
required disaggregations, as outlined in the
current UNGASS guidelines
Changes since
the 2005 UNGASS Guidelines
Why changes?
• To improve the comprehensiveness and quality
of data
What changes?
• See Appendix 1
Changes since
the 2005 UNGASS Guidelines (continued)
Principles used in revisions
• changes are based on
– input received from partners
– an analysis of indicator performance in the 2005
reporting round
– new programmatic developments
• every effort was taken to minimize changes
• every effort was taken to ensure that most
countries would be able to collect the data or
obtain it from already existing data sources
Changes since
the 2005 UNGASS Guidelines (continued)
Indicators added:
6) Percentage estimated HIV-positive incident TB
cases that received treatment for TB and HIV
7) Percentage of women and men aged 15-49
who received an HIV test in the last 12 months
and who know their results
Changes since
the 2005 UNGASS Guidelines (continued)
Indicators removed:
- Percentage of large enterprises/companies
which have HIV/AIDS workplace policies and
programmes
- Percentage of women and men with sexually
transmitted infections at health care facilities
who are appropriately diagnosed, treated and
counselled
Changes since
the 2005 UNGASS Guidelines (continued)
Indicator definition changed
1) AIDS spending
3) Blood Safety
5) Prevention of Mother-to-Child Transmission
9) Most-at-risk Populations: Prevention
Programmes
11) Life Skills-based HIV Education in Schools
16) Higher-risk Sex
17) Condom Use During Higher-risk Sex
Changes since
the 2005 UNGASS Guidelines
(continued)
Age range expanded
4) HIV Treatment: Antiretroviral Therapy
Composite indicator divided into its components
20) Injecting Drug Users: Condom Use
21) Injecting Drug Users: Safe Injecting Practices
Added questions
2) National Composite Policy Index
CORE UNGASS INDICATORS:
NATIONAL INDICATORS
National Commitment and Action
1) Domestic and International AIDS spending
by categories and financing sources
2) National Composite Policy Index (NCPI)
Questionnaire divided into two sections:
Part A
(Government)
Strategic plan; Political support;
Prevention; Treatment, care &
support; Monitoring &
Evaluation
Part B
(Non-government)
Human rights; Civil society
involvement; Prevention;
Treatment, care & support
CORE UNGASS INDICATORS:
NATIONAL INDICATORS
National Programmes
3) Percentage of donated blood units screened for
HIV in a quality-assured manner
4) Percentage of adults and children with advanced
HIV infection receiving antiretroviral therapy
5) Percentage of HIV-positive pregnant women who received
antiretrovirals to reduce the risk of mother-to-child transmission
6) Percentage estimated HIV-positive incident
TB cases that received treatment for TB and HIV
CORE UNGASS INDICATORS:
NATIONAL INDICATORS
National Programmes (continued)
7) Percentage of women and men aged 15-49 who received
an HIV test in the last 12 months and who know their results
8) Percentage of most-at-risk populations who received an
HIV test in the last 12 months and who know their results
9) Percentage of most-at-risk populations reached
with HIV prevention programmes
CORE UNGASS INDICATORS:
NATIONAL INDICATORS
National Programmes (continued)
10) Percentage of orphaned and vulnerable children aged 0-17 whose
households received free basic external support in caring for the child
11) Percentage of schools that provided life-skills based
HIV education within the last academic year
CORE UNGASS INDICATORS:
NATIONAL INDICATORS
Knowledge and Behaviour
12) Current school attendance among orphans and non-orphans aged 10–14*
13) Percentage of young women and men aged 15–24 who both correctly
identify ways of preventing the sexual transmission of HIV and
who reject major misconceptions about HIV transmission*
14) Percentage of most-at-risk populations who both correctly identify ways of
preventing the sexual transmission of HIV and who reject
major misconceptions about HIV transmission
*Millennium Development Goals indicator
CORE UNGASS INDICATORS:
NATIONAL INDICATORS
Knowledge and Behaviour (continued)
15) Percentage of young women and men
aged 15-24 who have had sexual intercourse before the age of 15
16) Percentage of women and men aged 15–49 who have
had sexual intercourse with more than one partner in the last 12 months
17) Percentage of women and men aged 15–49 who had more than
one sexual partner in the past 12 months reporting the use of a condom
during their last sexual intercourse*
*Millennium Development Goals indicator
CORE UNGASS INDICATORS:
NATIONAL INDICATORS
Knowledge and Behaviour (continued)
18) Percentage of female and male sex workers
reporting the use of a condom with their most recent client
19) Percentage of men reporting the use of a condom the
last time they had anal sex with a male partner
20) Percentage of injecting drug users reporting the use
of a condom the last time they had sexual intercourse
21) Percentage of injecting drug users reporting the use of sterile
injecting equipment the last time they injected
CORE UNGASS INDICATORS:
NATIONAL INDICATORS
Impact
22) Percentage of young women and men aged 15–24
who are HIV infected*
23) Percentage of most-at-risk populations who are HIV infected
24) Percentage of adults and children with HIV known to be on treatment
12 months after initiation of antiretroviral therapy
25) Percentage of infants born to HIV-infected mothers who are infected
*Millennium Development Goals indicator
DATA PROCESS FOR UNGASS
REPORTING
• What steps to include in the data process
• Why is the data process important
DATA PROCESS FOR
UNGASS REPORTING
Data collection, vetting, and analysis process including:
• Identification of relevant tools / sources for data
collection for each indicator
• Timeline for data collection in line with other data
collection efforts, including those via funding agencies
(e.g. GFATM, PEPFAR)
• Reporting timeline for facility-based data for national
level aggregation
• Data vetting workshops to reach consensus on the
correct value for each indicator
DATA PROCESS FOR UNGASS
REPORTING (continued)
Protocols for data processing and management including:
• Basic data cleaning and validation
• One database for analysis and reporting purposes
Analysis of relevant data
in coordination with partners from government, civil society
and the international community
- See Appendix 8 Sample Checklist -
DATA PROCESS FOR UNGASS REPORTING:
the WHAT
5. Use
of Data
4. Consensus
on Data
Have a comprehensive data analysis and
findings been included in the Country Progress
Report, endorsed by all relevant stakeholders?
Has a collaborative consultation with relevant
partners to reach consensus on indicator values
taken place and been documented?
Have the characteristics and quality
3. Understanding
of the available data been assessed and strengths and
Data
limitations understood?
2. Gathering Have data and related information been collected in a timely
manner and collated from multiple sources for each indicator?
Data
1. Planning
Has a data process for UNGASS reporting
with roles and responsibilities been established
and shared with all relevant partners?
DATA PROCESS FOR UNGASS REPORTING:
the WHY
• Completeness, integrity, credibility, and
consensus of indicator data is paramount for
successful monitoring of the HIV epidemic & for
guiding program planning, advocacy, and policy
• A clear and comprehensive data process,
involving all relevant stakeholders, will help
ensure steps are taken to obtain the best possible
data
DATA CONSIDERATIONS
• Data sources
• Numerators and Denominators
• Data disaggregation
• Recency and representativeness of data
DATA SOURCES
• Population-based surveys (e.g. DHS, AIS, MICS)
• Special surveys (e.g. BSS, 2nd generation surveillance
surveys, school surveys, health facility surveys)
• Programme monitoring (e.g. patient tracking, Health
Information System)
• Key informants
• Desk review
• HIV surveillance / estimates
NUMERATORS AND DENOMINATORS
• Most indicators have a numerator and
denominator to calculate a percentage
– Follow the instructions exactly
• It is important that the data collection period is
consistent for all the information relevant to a
specific indicator’s numerator and denominator
• Countries are expected to submit
– total values
AND
– all disaggregated values as per the instructions in the
Guidelines
• Data collected through facilities, programme
monitoring, and surveys is often disaggregated;
the challenge is to ensure the disaggregated
data is also available at national level
Why important?
• Allows for more effective tracking of resources
and programmatic responses
• Allows for monitoring trends in specified subpopulations
• Allows for monitoring equity of access to
services
IN-COUNTRY REPORTING PROCESS
• In-country reporting process
• Country Progress Report Format
• Submission Process
• Role of UNAIDS/Geneva
RECENCY & REPRESENTATIVENESS
Which data to submit?
Ideally,
• most recent data available
• nationally representative data
But,
• if data is not representative, most recent survey
data which has been reviewed and endorsed by
technical experts within the country
STEPS FOR IN-COUNTRY
UNGASS REPORTING PROCESS
Preparation
Submission
Planning
Implementation
ROLES & RESPONSIBILITIES
The overall responsibility of preparing and
submitting the Country Progress Report lies with
national governments (NAC or equivalent)
UNGASS should be routine and integrated in the
country’s M&E plans, budgets and systems
Preparation
PREPARATION
1. Assess country readiness
2. Identify stakeholders
3. Identify funding
4. Identify relevant data tools / sources
Preparation
Countries are expected
• to know their epidemic
• to determine which of the UNGASS indicators
are applicable to the country’s epidemiological
context
• to know availability of data
Preparation
Individuals and organizations who are affected by
HIV and/or are involved in the HIV response:
• Government (including NAC, Ministry of Health,
Ministry of Education; Ministry of Finance)
• Civil society groups (including nongovernmental
organizations, people living with HIV)
• Private sector
Preparation
• Strategic and tactical expertise
• Quantitative and qualitative data
• Valuable perspective on the data
Preparation
Involvement in all phases of the report preparation:
•
•
•
•
•
Identify relevant civil society organizations
Assign roles and responsibility
Plan with them data collection and analysis
Invite them to participate in workshops
Share drafts with them for review and comments
• Disseminate final report to them
Preparation
• What sources are available?
• No separate funding available for UNGASS
reporting, but should be integrated in M&E
budgets (government, GFTAM, World bank)
Preparation
DATA SOURCES
• Population-based surveys (e.g. DHS, AIS, MICS)
• Special surveys (e.g. BSS, 2nd generation surveillance
surveys, school surveys, health facility surveys)
• Programme monitoring (e.g. patient tracking, Health
Information System)
• Key informants
• Desk review
• HIV surveillance / estimates
Preparation
DATA CONSIDERATIONS
• Recency and representativeness of data
• Requirements for disaggregation
• Appropriate data process
Planning
• Develop a plan for data collection / collation,
analysis and report writing
• Designate a coordinator
• Assign roles and responsibilities for each
indicator
• Assign realistic timelines and milestones
• Secure funding for the entire process (collecting,
collating, analysing and reporting the data)
Planning
• Disseminate the plan
• Use the plan as a checklist to monitor progress
against the timeline and milestones
Planning
ACTION PLAN TEMPLATE
•
•
•
•
•
•
Steps
Activities
Timeline
Budget
Responsible person
Stakeholders involved
Implementation
• Establish a process for data collecting / collating,
vetting and analysis
• Organise workshops for vetting and triangulation
of data
• Complete the appropriate data forms
• Draft the Country Progress Report
• Disseminate it for review and comment
• Collate comments from stakeholders, including
government agencies and civil society
Implementation
• Organize a national consultation workshop to
reach consensus with stakeholders on the final
report
• Enter agreed data into CRIS (or equivalent data
management system)
• Validate the indicator data against the report
narrative (consistency check)
• Finalize and submit the report to UNAIDS
• Identify focal point for communication between
UNAIDS Secretariat and country
• Monitor progress throughout the implementation
process using the action plan
• Plan a debriefing meeting
– to assess success of report (timeliness,
comprehensiveness, quality)
– to agree actions to undertake to strengthen
the national M&E system
OVERVIEW OF THE REPORTING PROCESS
Identify data needs, data sources, stakeholders, funds
Establish plan for data collection, analysis and reporting writing
Secure funds, collect/collate and analyse data, complete data forms
Draft Country Progress Report, share draft with stakeholders
Enter data in CRIS
Validate the narrative report against the data in CRIS
Reach consensus with stakeholders on final Report & submit timely
Submission
of report
• Narrative part (see Appendix 2)
– Structured to ensure consistent information
across countries
– Provide an in-depth analysis of data regarding
the HIV epidemic and response
UNGASS COUNTRY PROGRESS REPORT
[Country Name]
Reporting period: January 2006–December 2007
Submission date: […]
I.
II.
III.
IV.
V.
VI.
VII.
VIII.
Table of Contents
Status at a glance
Overview of the AIDS epidemic
National response to the AIDS epidemic
Best practices
Major challenges and remedial actions
Support from country’s development partners
Monitoring and evaluation environment
Submission
of report
ANNEXES
ANNEX 1
Consultation/preparation process for the Country
Progress Report
ANNEX 2
National Composite Policy Index Questionnaire
COUNTRY PROGRESS REPORT
Indicator Data
Submission
of report
BENEFITS OF USING THE COUNTRY
RESPONSE INFORMATION SYSTEM (CRIS)
CRIS is a monitoring system which greatly
facilitates UNGASS reporting:
• It supports all six UN languages [no need for
translation]
• It uses the current Guidelines [no confusion]
• It uses the correct indicator definitions [no
confusion]
• It facilitates compilation and management of all
UNGASS data [saves time]
Submission
of report
BENEFITS OF USING CRIS (continued)
• It has built-in validity checks [avoids logical errors]
• It stores available data from population-based
surveys [saves time]
• It facilitates trend analysis as previous UNGASS
data is stored [saves time]
• Data is seamlessly integrated into the global
database [no transcription errors, no delays]
Submission
of report
BENEFITS OF USING CRIS (continued)
In addition to use for UNGASS reporting and
reporting to donors
CRIS can be used for the management of all
monitoring information at the national
and sub-national levels
Submission
of report
What?
Country Progress Report consisting of
– Narrative section (including Annex 1, Annex 2)
– CRIS data file (or excel data file, see template on CD)
When?
– 15 January 2008 if not using CRIS
– 31 January 2008 if using CRIS [preferred]
How?
• E-mail to ungassindicators@unaids.org
• UNAIDS and its partners are responsible for
calculating the global indicators
• A process for collating, analysing and reporting
data in coordination with co-sponsors and global
partners, including civil society occurs
Country Report Findings + Global indicators
=
Report on the Global AIDS Epidemic
• The Report on the Global AIDS Epidemic is
prepared by May 2008 and submitted to the
United Nations General Assembly
• The Country Progress Reports and the Global
Report will be available on the UNAIDS website
at http://www.unaids.org
Provides answers to important questions:
• What is the status of the epidemic in the
country?
• What are the basic trends in HIV transmission
and service coverage?
• What are the main obstacles to accessing HIV
prevention, care and treatment services?
• What is the quality of services being delivered?
Provides answers to important questions:
• Are services being delivered equitably and
effectively?
• What exacerbates these problems? (e.g.
policies, laws, resources, politics, customs,
organizations, individuals)
• Who can change this situation? (e.g. elected
leaders, bureaucrats, religious leaders,
community leaders, traditional leaders, donors,
international organizations, NGOs)
• What are these people/organizations currently
doing to address the problems?
This information can be used to
• evaluate the effectiveness of the national
response
• analyse linkages between policy, implementation
of programmes, behavioural change and HIV
prevalence
• identify gaps in programmes, personnel,
technology
• identify data gaps and data quality issues
• strengthen the National M&E system
•
•
•
•
Ownership & participation
Consultation & consensus
Data vetting, validation, triangulation
Timeliness: 31 January 2008
or 15 January 2008 if not submitted in CRIS
• Focus on systems and process