M&E for Accountability & Global Progress Tracking Monitoring, Operations Research and Evaluation Team UNAIDS/Geneva April 2007 ORIENTATION 2008 UNGASS Country Progress Report - Benefits of UNGASS for M&E strengthening - UNGASS Guidelines Purpose and use Core indicators Data sources & data vetting Reporting process INTRODUCTION • Where does UNGASS fit in M&E system? • UNGASS Reporting Timetable • Use of 2006 UNGASS data – Global use – Country use WHERE DOES UNGASS FIT IN A COMPREHENSIVE M&E SYSTEM? Benefits of UNGASS reporting for M&E strengthening: • Brings partners together around a core set of indicators • Helps identify data gaps & data quality issues • Stimulates the establishment of a centralised database of HIV indicator data • Provides a de facto bi-annual report on the epidemic and the response WHERE DOES UNGASS FIT IN A COMPREHENSIVE M&E SYSTEM? Program Improvement Share Data with Partners Reporting/ Accountability [UNGASS, other] UNGASS REPORTING TIMETABLE REPORT REQUIRED? 2003 Yes 2004 No 2005 Interim report from a subset of countries 2006 Yes 2007 No 2008 Yes 2009 No 2010 Yes GLOBAL USE OF 2006 UNGASS DATA • Global Progress Report • Report to the Secretary General (as required in the Declaration of Commitment) • Used for tracking the Millennium Development Goals COUNTRY USE OF 2006 UNGASS DATA 1. National report on the epidemic and response • informs programmatic decisions • guides reviews of the National Strategic Plan • informs resource mobilisation efforts (e.g. GFATM proposal development) • is used as an advocacy tool • is used to report progress on Declaration of Commitment to Senior Government Officials COUNTRY USE OF 2006 UNGASS DATA 2. M&E system strengthening • convenes relevant partners to address a particular M&E need • identifies data gaps • assists in planning data collection efforts • identifies data weaknesses • helps mobilise resources for system strengthening • stimulates evaluation questions LESSONS LEARNED FROM 2006 UNGASS REPORTING • Challenges in 2006 reporting • 2006 reporting from MENA • Lessons learned from 2006 reporting – UNAIDS/Geneva-level – Country-level CHALLENGES IN 2006 REPORTING 137 countries submitted a Country Progress Report • 10% reported on all indicators relevant to epidemic • 51% used CRIS to report indicator data Some basic data quality issues including – incorrect indicator or incorrect indicator calculation – incomplete data (e.g. only 20% of indicators reported included the required disaggregation) – illogical values – discrepancies between data in CRIS and Report Percent Range and Median of Indicators Reported by Countries by Region - UNGASS 2006 Report IMPORTANCE OF DISAGGREGATED DATA Safe Injecting & Sexual Practice in IDU by Sex; 2006 UNGASS Report 80 70 Males Females 60 Percent 50 40 30 20 10 0 Bangladesh Indonesia Thailand Arm enia Belarus Moldova, Republic of Macedonia LESSONS LEARNED FROM 2006 REPORTING UNAIDS/Geneva-level The following inputs are paramount for improving the quality of the data obtained from countries: – Revise UNGASS indicators, where necessary – Provide clear and specific Guidelines – Provide user-friendly, flexible CRIS software – Provide training on UNGASS indicators & reporting process – Provide technical support, where needed LESSONS LEARNED FROM 2006 REPORTING Country-level • Ensure correct understanding of indicator definition/operands • Report all required indicators • Identify and document data sources • Provide all data and information needed for each indicator • Cross-validate data (comparison with previous years, comparison with alternate sources, etc.) • Use CRIS for indicator reporting • Double-check for data entry errors • Submit timely LESSONS LESSONS LEARNED LEARNED FROM FROM 2006 2006 REPORTING REPORTING Country-level 1. 2. 3. 4. Address data gaps early Seek to enhance data quality Follow all the steps in the reporting process Double-check accuracy and obtain consensus on all indicator values at country level before submission ORIENTATION ON THE UNGASS GUIDELINES • Purpose • Target audience • Contents • Principles and process of Guidelines revision • Use of Guidelines • Provide essential information on construction of core indicators for reporting • Ensure transparency of the reporting process in-country • Ensure consistency of information across countries for global progress analysis • Government sector – including the National AIDS Committee, Ministries (e.g. Ministry of Health, Ministry of Finance, Ministry of Education) • Nongovernmental sector – including nongovernmental organizations, faith-based organizations, people living with HIV, private sector • UN co-sponsors and donor agencies CONTENTS OVERVIEW • Introduction p. 9-13 • Implementation at National Level p. 14- 20 • Core Indicators p. 21-82 • Appendices 1 - 9 p. 83-139 APPENDICES 1 - 9 1. Changes to Core Indicators 2. Country Progress Report template 3. Methodology used for the coverage survey 4. Consultation/preparation process for the Country Progress Report 5. Reporting schedule for Core Indicators 6. National Funding Matrix 2007 7. National Composite Policy Index (NCPI) 2007 8. Sample check-list for Country Progress Report 9. Selected bibliography • • • • • • • Consensus-driven process Transparency Consistency Comprehensiveness Quality Integration within the National M&E system Feasibility Process of Guidelines Revision • • • • Debriefing process 2006 Indicator performance M&E Reference Group review Technical working groups for each indicator • Familiarize yourself with the current Guidelines & the changes since the previous round • Disseminate the Guidelines and use them to develop a collaborative in-country process • Ensure that the correct definitions are used for construction of the indicators & refer to Further Information, as needed • Use all Appendices • Ask for clarification / assistance, as needed CORE INDICATORS • Overview of Core UNGASS Indicators • Which UNGASS indicators to report • Changes since the 2005 UNGASS Guidelines • Core UNGASS Indicators WHAT ARE CORE UNGASS INDICATORS? Quantitative variables which provide simple and reliable ways of measuring progress towards achieving the Declaration of Commitment on HIV/AIDS CORE UNGASS INDICATORS National Indicators (N=25) Four categories: 1. National commitment and action 2. National programmes 3. Knowledge and behaviour 4. Impact Global Indicators (N=4) CORE UNGASS INDICATORS: NATIONAL INDICATORS Four categories: 1. National commitment and action 2. National programmes 3. Knowledge and behaviour 4. Impact CORE UNGASS INDICATORS: NATIONAL INDICATORS National indicators are important for two reasons: 1. They are used to evaluate the effectiveness of the national response 2. They are used to provide information on regional and global trends MILLENIUM (MDGs) MillenniumDEVELOPMENT Development GOALS Goals (MDG) There are 8 MDGs: – Goal 6: Combat HIV/AIDS, Malaria and other diseases • Target: Have halted by 2015 and begun to reverse the spread of HIV/AIDS • UNGASS indicators are used to monitor progress UNGASS Indicators to monitor MDG-6 17) Percentage of women and men • Four of attendance the national aged indicators alsothan 12) Current school 15–49 whoare had more among orphans and one sexual partner in the past among non-orphans aged 10-14 12 months who report the use of a condom during their last sexual intercourse 13) Percentage of young women and men aged 15-24 who both correctly identify ways of preventing the sexual transmission of HIV and who reject major misconceptions about HIV transmission 22) Percentage of young women and men aged 15–24 who are HIV infected Global Indicators 1) Amount of bilateral and multilateral financial flows (commitments and disbursements) for the benefit of low- and middle-income countries 2) Amount of public funds for research and development of preventive HIV vaccines and microbicides 3) Percentage of transnational companies that are present in developing countries and that have workplace HIV policies and programmes 4) Percentage of international organizations which have workplace HIV policies and programmes Which UNGASS indicators should countries report? • No distinction between Generalized Epidemic Indicator Set & Concentrated /Low-prevalence Epidemic Indicator Set • Most national indicators are applicable for all countries • Countries are expected to "know their epidemic" Which UNGASS UNGASS indicators indicators Which should countries report? (continued) should countries report? When countries choose not to report on a particular indicator, an explanation needs to be provided: Either, • The indicator is not applicable to the epidemic • The indicator is applicable, but no data is available Which UNGASS indicators should countries report? (continued) The indicators have to be constructed including the required disaggregations, as outlined in the current UNGASS guidelines Changes since the 2005 UNGASS Guidelines Why changes? • To improve the comprehensiveness and quality of data What changes? • See Appendix 1 Changes since the 2005 UNGASS Guidelines (continued) Principles used in revisions • changes are based on – input received from partners – an analysis of indicator performance in the 2005 reporting round – new programmatic developments • every effort was taken to minimize changes • every effort was taken to ensure that most countries would be able to collect the data or obtain it from already existing data sources Changes since the 2005 UNGASS Guidelines (continued) Indicators added: 6) Percentage estimated HIV-positive incident TB cases that received treatment for TB and HIV 7) Percentage of women and men aged 15-49 who received an HIV test in the last 12 months and who know their results Changes since the 2005 UNGASS Guidelines (continued) Indicators removed: - Percentage of large enterprises/companies which have HIV/AIDS workplace policies and programmes - Percentage of women and men with sexually transmitted infections at health care facilities who are appropriately diagnosed, treated and counselled Changes since the 2005 UNGASS Guidelines (continued) Indicator definition changed 1) AIDS spending 3) Blood Safety 5) Prevention of Mother-to-Child Transmission 9) Most-at-risk Populations: Prevention Programmes 11) Life Skills-based HIV Education in Schools 16) Higher-risk Sex 17) Condom Use During Higher-risk Sex Changes since the 2005 UNGASS Guidelines (continued) Age range expanded 4) HIV Treatment: Antiretroviral Therapy Composite indicator divided into its components 20) Injecting Drug Users: Condom Use 21) Injecting Drug Users: Safe Injecting Practices Added questions 2) National Composite Policy Index CORE UNGASS INDICATORS: NATIONAL INDICATORS National Commitment and Action 1) Domestic and International AIDS spending by categories and financing sources 2) National Composite Policy Index (NCPI) Questionnaire divided into two sections: Part A (Government) Strategic plan; Political support; Prevention; Treatment, care & support; Monitoring & Evaluation Part B (Non-government) Human rights; Civil society involvement; Prevention; Treatment, care & support CORE UNGASS INDICATORS: NATIONAL INDICATORS National Programmes 3) Percentage of donated blood units screened for HIV in a quality-assured manner 4) Percentage of adults and children with advanced HIV infection receiving antiretroviral therapy 5) Percentage of HIV-positive pregnant women who received antiretrovirals to reduce the risk of mother-to-child transmission 6) Percentage estimated HIV-positive incident TB cases that received treatment for TB and HIV CORE UNGASS INDICATORS: NATIONAL INDICATORS National Programmes (continued) 7) Percentage of women and men aged 15-49 who received an HIV test in the last 12 months and who know their results 8) Percentage of most-at-risk populations who received an HIV test in the last 12 months and who know their results 9) Percentage of most-at-risk populations reached with HIV prevention programmes CORE UNGASS INDICATORS: NATIONAL INDICATORS National Programmes (continued) 10) Percentage of orphaned and vulnerable children aged 0-17 whose households received free basic external support in caring for the child 11) Percentage of schools that provided life-skills based HIV education within the last academic year CORE UNGASS INDICATORS: NATIONAL INDICATORS Knowledge and Behaviour 12) Current school attendance among orphans and non-orphans aged 10–14* 13) Percentage of young women and men aged 15–24 who both correctly identify ways of preventing the sexual transmission of HIV and who reject major misconceptions about HIV transmission* 14) Percentage of most-at-risk populations who both correctly identify ways of preventing the sexual transmission of HIV and who reject major misconceptions about HIV transmission *Millennium Development Goals indicator CORE UNGASS INDICATORS: NATIONAL INDICATORS Knowledge and Behaviour (continued) 15) Percentage of young women and men aged 15-24 who have had sexual intercourse before the age of 15 16) Percentage of women and men aged 15–49 who have had sexual intercourse with more than one partner in the last 12 months 17) Percentage of women and men aged 15–49 who had more than one sexual partner in the past 12 months reporting the use of a condom during their last sexual intercourse* *Millennium Development Goals indicator CORE UNGASS INDICATORS: NATIONAL INDICATORS Knowledge and Behaviour (continued) 18) Percentage of female and male sex workers reporting the use of a condom with their most recent client 19) Percentage of men reporting the use of a condom the last time they had anal sex with a male partner 20) Percentage of injecting drug users reporting the use of a condom the last time they had sexual intercourse 21) Percentage of injecting drug users reporting the use of sterile injecting equipment the last time they injected CORE UNGASS INDICATORS: NATIONAL INDICATORS Impact 22) Percentage of young women and men aged 15–24 who are HIV infected* 23) Percentage of most-at-risk populations who are HIV infected 24) Percentage of adults and children with HIV known to be on treatment 12 months after initiation of antiretroviral therapy 25) Percentage of infants born to HIV-infected mothers who are infected *Millennium Development Goals indicator DATA PROCESS FOR UNGASS REPORTING • What steps to include in the data process • Why is the data process important DATA PROCESS FOR UNGASS REPORTING Data collection, vetting, and analysis process including: • Identification of relevant tools / sources for data collection for each indicator • Timeline for data collection in line with other data collection efforts, including those via funding agencies (e.g. GFATM, PEPFAR) • Reporting timeline for facility-based data for national level aggregation • Data vetting workshops to reach consensus on the correct value for each indicator DATA PROCESS FOR UNGASS REPORTING (continued) Protocols for data processing and management including: • Basic data cleaning and validation • One database for analysis and reporting purposes Analysis of relevant data in coordination with partners from government, civil society and the international community - See Appendix 8 Sample Checklist - DATA PROCESS FOR UNGASS REPORTING: the WHAT 5. Use of Data 4. Consensus on Data Have a comprehensive data analysis and findings been included in the Country Progress Report, endorsed by all relevant stakeholders? Has a collaborative consultation with relevant partners to reach consensus on indicator values taken place and been documented? Have the characteristics and quality 3. Understanding of the available data been assessed and strengths and Data limitations understood? 2. Gathering Have data and related information been collected in a timely manner and collated from multiple sources for each indicator? Data 1. Planning Has a data process for UNGASS reporting with roles and responsibilities been established and shared with all relevant partners? DATA PROCESS FOR UNGASS REPORTING: the WHY • Completeness, integrity, credibility, and consensus of indicator data is paramount for successful monitoring of the HIV epidemic & for guiding program planning, advocacy, and policy • A clear and comprehensive data process, involving all relevant stakeholders, will help ensure steps are taken to obtain the best possible data DATA CONSIDERATIONS • Data sources • Numerators and Denominators • Data disaggregation • Recency and representativeness of data DATA SOURCES • Population-based surveys (e.g. DHS, AIS, MICS) • Special surveys (e.g. BSS, 2nd generation surveillance surveys, school surveys, health facility surveys) • Programme monitoring (e.g. patient tracking, Health Information System) • Key informants • Desk review • HIV surveillance / estimates NUMERATORS AND DENOMINATORS • Most indicators have a numerator and denominator to calculate a percentage – Follow the instructions exactly • It is important that the data collection period is consistent for all the information relevant to a specific indicator’s numerator and denominator • Countries are expected to submit – total values AND – all disaggregated values as per the instructions in the Guidelines • Data collected through facilities, programme monitoring, and surveys is often disaggregated; the challenge is to ensure the disaggregated data is also available at national level Why important? • Allows for more effective tracking of resources and programmatic responses • Allows for monitoring trends in specified subpopulations • Allows for monitoring equity of access to services IN-COUNTRY REPORTING PROCESS • In-country reporting process • Country Progress Report Format • Submission Process • Role of UNAIDS/Geneva RECENCY & REPRESENTATIVENESS Which data to submit? Ideally, • most recent data available • nationally representative data But, • if data is not representative, most recent survey data which has been reviewed and endorsed by technical experts within the country STEPS FOR IN-COUNTRY UNGASS REPORTING PROCESS Preparation Submission Planning Implementation ROLES & RESPONSIBILITIES The overall responsibility of preparing and submitting the Country Progress Report lies with national governments (NAC or equivalent) UNGASS should be routine and integrated in the country’s M&E plans, budgets and systems Preparation PREPARATION 1. Assess country readiness 2. Identify stakeholders 3. Identify funding 4. Identify relevant data tools / sources Preparation Countries are expected • to know their epidemic • to determine which of the UNGASS indicators are applicable to the country’s epidemiological context • to know availability of data Preparation Individuals and organizations who are affected by HIV and/or are involved in the HIV response: • Government (including NAC, Ministry of Health, Ministry of Education; Ministry of Finance) • Civil society groups (including nongovernmental organizations, people living with HIV) • Private sector Preparation • Strategic and tactical expertise • Quantitative and qualitative data • Valuable perspective on the data Preparation Involvement in all phases of the report preparation: • • • • • Identify relevant civil society organizations Assign roles and responsibility Plan with them data collection and analysis Invite them to participate in workshops Share drafts with them for review and comments • Disseminate final report to them Preparation • What sources are available? • No separate funding available for UNGASS reporting, but should be integrated in M&E budgets (government, GFTAM, World bank) Preparation DATA SOURCES • Population-based surveys (e.g. DHS, AIS, MICS) • Special surveys (e.g. BSS, 2nd generation surveillance surveys, school surveys, health facility surveys) • Programme monitoring (e.g. patient tracking, Health Information System) • Key informants • Desk review • HIV surveillance / estimates Preparation DATA CONSIDERATIONS • Recency and representativeness of data • Requirements for disaggregation • Appropriate data process Planning • Develop a plan for data collection / collation, analysis and report writing • Designate a coordinator • Assign roles and responsibilities for each indicator • Assign realistic timelines and milestones • Secure funding for the entire process (collecting, collating, analysing and reporting the data) Planning • Disseminate the plan • Use the plan as a checklist to monitor progress against the timeline and milestones Planning ACTION PLAN TEMPLATE • • • • • • Steps Activities Timeline Budget Responsible person Stakeholders involved Implementation • Establish a process for data collecting / collating, vetting and analysis • Organise workshops for vetting and triangulation of data • Complete the appropriate data forms • Draft the Country Progress Report • Disseminate it for review and comment • Collate comments from stakeholders, including government agencies and civil society Implementation • Organize a national consultation workshop to reach consensus with stakeholders on the final report • Enter agreed data into CRIS (or equivalent data management system) • Validate the indicator data against the report narrative (consistency check) • Finalize and submit the report to UNAIDS • Identify focal point for communication between UNAIDS Secretariat and country • Monitor progress throughout the implementation process using the action plan • Plan a debriefing meeting – to assess success of report (timeliness, comprehensiveness, quality) – to agree actions to undertake to strengthen the national M&E system OVERVIEW OF THE REPORTING PROCESS Identify data needs, data sources, stakeholders, funds Establish plan for data collection, analysis and reporting writing Secure funds, collect/collate and analyse data, complete data forms Draft Country Progress Report, share draft with stakeholders Enter data in CRIS Validate the narrative report against the data in CRIS Reach consensus with stakeholders on final Report & submit timely Submission of report • Narrative part (see Appendix 2) – Structured to ensure consistent information across countries – Provide an in-depth analysis of data regarding the HIV epidemic and response UNGASS COUNTRY PROGRESS REPORT [Country Name] Reporting period: January 2006–December 2007 Submission date: […] I. II. III. IV. V. VI. VII. VIII. Table of Contents Status at a glance Overview of the AIDS epidemic National response to the AIDS epidemic Best practices Major challenges and remedial actions Support from country’s development partners Monitoring and evaluation environment Submission of report ANNEXES ANNEX 1 Consultation/preparation process for the Country Progress Report ANNEX 2 National Composite Policy Index Questionnaire COUNTRY PROGRESS REPORT Indicator Data Submission of report BENEFITS OF USING THE COUNTRY RESPONSE INFORMATION SYSTEM (CRIS) CRIS is a monitoring system which greatly facilitates UNGASS reporting: • It supports all six UN languages [no need for translation] • It uses the current Guidelines [no confusion] • It uses the correct indicator definitions [no confusion] • It facilitates compilation and management of all UNGASS data [saves time] Submission of report BENEFITS OF USING CRIS (continued) • It has built-in validity checks [avoids logical errors] • It stores available data from population-based surveys [saves time] • It facilitates trend analysis as previous UNGASS data is stored [saves time] • Data is seamlessly integrated into the global database [no transcription errors, no delays] Submission of report BENEFITS OF USING CRIS (continued) In addition to use for UNGASS reporting and reporting to donors CRIS can be used for the management of all monitoring information at the national and sub-national levels Submission of report What? Country Progress Report consisting of – Narrative section (including Annex 1, Annex 2) – CRIS data file (or excel data file, see template on CD) When? – 15 January 2008 if not using CRIS – 31 January 2008 if using CRIS [preferred] How? • E-mail to ungassindicators@unaids.org • UNAIDS and its partners are responsible for calculating the global indicators • A process for collating, analysing and reporting data in coordination with co-sponsors and global partners, including civil society occurs Country Report Findings + Global indicators = Report on the Global AIDS Epidemic • The Report on the Global AIDS Epidemic is prepared by May 2008 and submitted to the United Nations General Assembly • The Country Progress Reports and the Global Report will be available on the UNAIDS website at http://www.unaids.org Provides answers to important questions: • What is the status of the epidemic in the country? • What are the basic trends in HIV transmission and service coverage? • What are the main obstacles to accessing HIV prevention, care and treatment services? • What is the quality of services being delivered? Provides answers to important questions: • Are services being delivered equitably and effectively? • What exacerbates these problems? (e.g. policies, laws, resources, politics, customs, organizations, individuals) • Who can change this situation? (e.g. elected leaders, bureaucrats, religious leaders, community leaders, traditional leaders, donors, international organizations, NGOs) • What are these people/organizations currently doing to address the problems? This information can be used to • evaluate the effectiveness of the national response • analyse linkages between policy, implementation of programmes, behavioural change and HIV prevalence • identify gaps in programmes, personnel, technology • identify data gaps and data quality issues • strengthen the National M&E system • • • • Ownership & participation Consultation & consensus Data vetting, validation, triangulation Timeliness: 31 January 2008 or 15 January 2008 if not submitted in CRIS • Focus on systems and process