M&E for Accountability & Global Progress Tracking Monitoring, Operations Research and Evaluation Team UNAIDS/Geneva April 2007
ORIENTATION 2008 UNGASS Country Progress Report - Benefits of UNGASS for M&E strengthening - UNGASS Guidelines Purpose and use Core indicators Data sources & data vetting Reporting process
INTRODUCTION Where does UNGASS fit in M&E system? UNGASS Reporting Timetable Use of 2006 UNGASS data Global use Country use
WHERE DOES UNGASS FIT IN A COMPREHENSIVE M&E SYSTEM? Benefits of UNGASS reporting for M&E strengthening: Brings partners together around a core set of indicators Helps identify data gaps & data quality issues Stimulates the establishment of a centralised database of HIV indicator data Provides a de facto bi-annual report on the epidemic and the response
WHERE DOES UNGASS FIT IN A COMPREHENSIVE M&E SYSTEM? Program Improvement Share Data with Partners Reporting/ Accountability [UNGASS, other]
UNGASS REPORTING TIMETABLE REPORT REQUIRED? 2003 Yes 2004 No 2005 Interim report from a subset of countries 2006 Yes 2007 No 2008 Yes 2009 No 2010 Yes
GLOBAL USE OF 2006 UNGASS DATA Global Progress Report Report to the Secretary General (as required in the Declaration of Commitment) Used for tracking the Millennium Development Goals
COUNTRY USE OF 2006 UNGASS DATA 1. National report on the epidemic and response informs programmatic decisions guides reviews of the National Strategic Plan informs resource mobilisation efforts (e.g. GFATM proposal development) is used as an advocacy tool is used to report progress on Declaration of Commitment to Senior Government Officials
COUNTRY USE OF 2006 UNGASS DATA 2. M&E system strengthening convenes relevant partners to address a particular M&E need identifies data gaps assists in planning data collection efforts identifies data weaknesses helps mobilise resources for system strengthening stimulates evaluation questions
LESSONS LEARNED FROM 2006 UNGASS REPORTING Challenges in 2006 reporting 2006 reporting from MENA Lessons learned from 2006 reporting UNAIDS/Geneva-level Country-level
CHALLENGES IN 2006 REPORTING 137 countries submitted a Country Progress Report 10% reported on all indicators relevant to epidemic 51% used CRIS to report indicator data Some basic data quality issues including incorrect indicator or incorrect indicator calculation incomplete data (e.g. only 20% of indicators reported included the required disaggregation) illogical values discrepancies between data in CRIS and Report
Percent Range and Median of Indicators Reported by Countries by Region - UNGASS 2006 Report 100% 100% 100% 100% 90% 89% 82% 85% 88% 88% 80% 70% 60% 50% 40% 30% 20% 10% 0% Developed Lat in Asia & East er n Middle E ast West & E ast & Caribbean America Pacific Europe & Nort h Cent ral South Af rica Af rica Af rica Regi on
IMPORTANCE OF DISAGGREGATED DATA 80 70 60 Safe Injecting & Sexual Practice in IDU by Sex; 2006 UNGASS Report Males Females 50 Percent 40 30 20 10 0 Bangladesh Indonesia Thailand Armenia Belarus Moldova, Republic of Macedonia
LESSONS LEARNED FROM 2006 REPORTING UNAIDS/Geneva-level The following inputs are paramount for improving the quality of the data obtained from countries: Revise UNGASS indicators, where necessary Provide clear and specific Guidelines Provide user-friendly, flexible CRIS software Provide training on UNGASS indicators & reporting process Provide technical support, where needed
LESSONS LEARNED FROM 2006 REPORTING Country-level Ensure correct understanding of indicator definition/operands Report all required indicators Identify and document data sources Provide all data and information needed for each indicator Cross-validate data (comparison with previous years, comparison with alternate sources, etc.) Use CRIS for indicator reporting Double-check for data entry errors Submit timely
LESSONS LEARNED FROM 2006 REPORTING Country-level 1. Address data gaps early 2. Seek to enhance data quality 3. Follow all the steps in the reporting process 4. Double-check accuracy and obtain consensus on all indicator values at country level before submission
ORIENTATION ON THE UNGASS GUIDELINES Purpose Target audience Contents Principles and process of Guidelines revision Use of Guidelines
PURPOSE Provide essential information on construction of core indicators for reporting Ensure transparency of the reporting process in-country Ensure consistency of information across countries for global progress analysis
TARGET AUDIENCES Government sector including the National AIDS Committee, Ministries (e.g. Ministry of Health, Ministry of Finance, Ministry of Education) Nongovernmental sector including nongovernmental organizations, faith-based organizations, people living with HIV, private sector UN co-sponsors and donor agencies
CONTENTS OVERVIEW Introduction p. 9-13 Implementation at National Level p. 14-20 Core Indicators p. 21-82 Appendices 1-9 p. 83-139
APPENDICES 1-9 1. Changes to Core Indicators 2. Country Progress Report template 3. Methodology used for the coverage survey 4. Consultation/preparation process for the Country Progress Report 5. Reporting schedule for Core Indicators 6. National Funding Matrix 2007 7. National Composite Policy Index (NCPI) 2007 8. Sample check-list for Country Progress Report 9. Selected bibliography
PRINCIPLES OF GUIDELINES REVISION Consensus-driven process Transparency Consistency Comprehensiveness Quality Integration within the National M&E system Feasibility
Process of Guidelines Revision PROCESS OF GUIDELINES REVISION Debriefing process 2006 Indicator performance M&E Reference Group review Technical working groups for each indicator
HOW TO USE THE GUIDELINES Familiarize yourself with the current Guidelines & the changes since the previous round Disseminate the Guidelines and use them to develop a collaborative in-country process Ensure that the correct definitions are used for construction of the indicators & refer to Further Information, as needed Use all Appendices Ask for clarification / assistance, as needed
CORE INDICATORS Overview of Core UNGASS Indicators Which UNGASS indicators to report Changes since the 2005 UNGASS Guidelines Core UNGASS Indicators
WHAT ARE CORE UNGASS INDICATORS? Quantitative variables which provide simple and reliable ways of measuring progress towards achieving the Declaration of Commitment on HIV/AIDS
CORE UNGASS INDICATORS National Indicators (N=25) Four categories: 1. National commitment and action 2. National programmes 3. Knowledge and behaviour 4. Impact Global Indicators (N=4)
CORE UNGASS INDICATORS: NATIONAL INDICATORS Four categories: 1. National commitment and action 2. National programmes 3. Knowledge and behaviour 4. Impact
CORE UNGASS INDICATORS: NATIONAL INDICATORS National indicators are important for two reasons: 1. They are used to evaluate the effectiveness of the national response 2. They are used to provide information on regional and global trends
MILLENIUM DEVELOPMENT GOALS (MDGs( MDGs) Millennium Development Goals (MDG) There are 8 MDGs: Goal 6: Combat HIV/AIDS, Malaria and other diseases Target: Have halted by 2015 and begun to reverse the spread of HIV/AIDS UNGASS indicators are used to monitor progress
UNGASS Indicators to monitor MDG-6 Four of the national indicators are also 12) Current school attendance among orphans and among non-orphans aged 10-14 17) Percentage of women and men aged 15 49 who had more than one sexual partner in the past 12 months who report the use of a condom during their last sexual intercourse 13) Percentage of young women and men aged 15-24 who both correctly identify ways of preventing the sexual transmission of HIV and who reject major misconceptions about HIV transmission 22) Percentage of young women and men aged 15 24 who are HIV infected
Global Indicators 1) Amount of bilateral and multilateral financial flows (commitments and disbursements) for the benefit of low- and middle-income countries 2) Amount of public funds for research and development of preventive HIV vaccines and microbicides 3) Percentage of transnational companies that are present in developing countries and that have workplace HIV policies and programmes 4) Percentage of international organizations which have workplace HIV policies and programmes
Which UNGASS indicators should countries report? No distinction between Generalized Epidemic Indicator Set & Concentrated /Low-prevalence Epidemic Indicator Set Most national indicators are applicable for all countries Countries are expected to "know their epidemic"
Which UNGASS indicators should should countries countries report? report? (continued) When countries choose not to report on a particular indicator, an explanation needs to be provided: Either, The indicator is not applicable to the epidemic The indicator is applicable, but no data is available
Which UNGASS indicators should countries report? (continued) The indicators have to be constructed including the required disaggregations, as outlined in the current UNGASS guidelines
Why changes? Changes since the 2005 UNGASS Guidelines To improve the comprehensiveness and quality of data What changes? See Appendix 1
Changes since the 2005 UNGASS Guidelines (continued) Principles used in revisions changes are based on input received from partners an analysis of indicator performance in the 2005 reporting round new programmatic developments every effort was taken to minimize changes every effort was taken to ensure that most countries would be able to collect the data or obtain it from already existing data sources
Changes since the 2005 UNGASS Guidelines (continued) Indicators added: 6) Percentage estimated HIV-positive incident TB cases that received treatment for TB and HIV 7) Percentage of women and men aged 15-49 who received an HIV test in the last 12 months and who know their results
Changes since the 2005 UNGASS Guidelines (continued) Indicators removed: - Percentage of large enterprises/companies which have HIV/AIDS workplace policies and programmes - Percentage of women and men with sexually transmitted infections at health care facilities who are appropriately diagnosed, treated and counselled
Changes since the 2005 UNGASS Guidelines (continued) Indicator definition changed 1) AIDS spending 3) Blood Safety 5) Prevention of Mother-to-Child Transmission 9) Most-at-risk Populations: Prevention Programmes 11) Life Skills-based HIV Education in Schools 16) Higher-risk Sex 17) Condom Use During Higher-risk Sex
Changes since the 2005 UNGASS Guidelines (continued) Age range expanded 4) HIV Treatment: Antiretroviral Therapy Composite indicator divided into its components 20) Injecting Drug Users: Condom Use 21) Injecting Drug Users: Safe Injecting Practices Added questions 2) National Composite Policy Index
CORE UNGASS INDICATORS: NATIONAL INDICATORS National Commitment and Action 1) Domestic and International AIDS spending by categories and financing sources 2) National Composite Policy Index (NCPI) Questionnaire divided into two sections: Part A (Government) Strategic plan; Political support; Prevention; Treatment, care & support; Monitoring & Evaluation Part B (Non-government) Human rights; Civil society involvement; Prevention; Treatment, care & support
CORE UNGASS INDICATORS: NATIONAL INDICATORS National Programmes 3) Percentage of donated blood units screened for HIV in a quality-assured manner 4) Percentage of adults and children with advanced HIV infection receiving antiretroviral therapy 5) Percentage of HIV-positive pregnant women who received antiretrovirals to reduce the risk of mother-to-child transmission 6) Percentage estimated HIV-positive incident TB cases that received treatment for TB and HIV
CORE UNGASS INDICATORS: NATIONAL INDICATORS National Programmes (continued) 7) Percentage of women and men aged 15-49 who received an HIV test in the last 12 months and who know their results 8) Percentage of most-at-risk populations who received an HIV test in the last 12 months and who know their results 9) Percentage of most-at-risk populations reached with HIV prevention programmes
CORE UNGASS INDICATORS: NATIONAL INDICATORS National Programmes (continued) 10) Percentage of orphaned and vulnerable children aged 0-17 whose households received free basic external support in caring for the child 11) Percentage of schools that provided life-skills based HIV education within the last academic year
CORE UNGASS INDICATORS: NATIONAL INDICATORS Knowledge and Behaviour 12) Current school attendance among orphans and non-orphans aged 10 14* 13) Percentage of young women and men aged 15 24 who both correctly identify ways of preventing the sexual transmission of HIV and who reject major misconceptions about HIV transmission* 14) Percentage of most-at-risk populations who both correctly identify ways of preventing the sexual transmission of HIV and who reject major misconceptions about HIV transmission *Millennium Development Goals indicator
CORE UNGASS INDICATORS: NATIONAL INDICATORS Knowledge and Behaviour (continued) 15) Percentage of young women and men aged 15-24 who have had sexual intercourse before the age of 15 16) Percentage of women and men aged 15 49 who have had sexual intercourse with more than one partner in the last 12 months 17) Percentage of women and men aged 15 49 who had more than one sexual partner in the past 12 months reporting the use of a condom during their last sexual intercourse* *Millennium Development Goals indicator
CORE UNGASS INDICATORS: NATIONAL INDICATORS Knowledge and Behaviour (continued) 18) Percentage of female and male sex workers reporting the use of a condom with their most recent client 19) Percentage of men reporting the use of a condom the last time they had anal sex with a male partner 20) Percentage of injecting drug users reporting the use of a condom the last time they had sexual intercourse 21) Percentage of injecting drug users reporting the use of sterile injecting equipment the last time they injected
CORE UNGASS INDICATORS: NATIONAL INDICATORS Impact 22) Percentage of young women and men aged 15 24 who are HIV infected* 23) Percentage of most-at-risk populations who are HIV infected 24) Percentage of adults and children with HIV known to be on treatment 12 months after initiation of antiretroviral therapy 25) Percentage of infants born to HIV-infected mothers who are infected *Millennium Development Goals indicator
DATA PROCESS FOR UNGASS REPORTING What steps to include in the data process Why is the data process important
DATA PROCESS FOR UNGASS REPORTING Data collection, vetting, and analysis process including: Identification of relevant tools / sources for data collection for each indicator Timeline for data collection in line with other data collection efforts, including those via funding agencies (e.g. GFATM, PEPFAR) Reporting timeline for facility-based data for national level aggregation Data vetting workshops to reach consensus on the correct value for each indicator
DATA PROCESS FOR UNGASS REPORTING (continued) Protocols for data processing and management including: Basic data cleaning and validation One database for analysis and reporting purposes Analysis of relevant data in coordination with partners from government, civil society and the international community - See Appendix 8 Sample Checklist -
DATA PROCESS FOR UNGASS REPORTING: the WHAT 5. Use of Data Have a comprehensive data analysis and findings been included in the Country Progress Report, endorsed by all relevant stakeholders? 4. Consensus on Data 3. Understanding Data Has a collaborative consultation with relevant partners to reach consensus on indicator values taken place and been documented? Have the characteristics and quality of the available data been assessed and strengths and limitations understood? 2. Gathering Data 1. Planning Have data and related information been collected in a timely manner and collated from multiple sources for each indicator? Has a data process for UNGASS reporting with roles and responsibilities been established and shared with all relevant partners?
DATA PROCESS FOR UNGASS REPORTING: the WHY Completeness, integrity, credibility, and consensus of indicator data is paramount for successful monitoring of the HIV epidemic & for guiding program planning, advocacy, and policy A clear and comprehensive data process, involving all relevant stakeholders, will help ensure steps are taken to obtain the best possible data
DATA CONSIDERATIONS Data sources Numerators and Denominators Data disaggregation Recency and representativeness of data
DATA SOURCES Population-based surveys (e.g. DHS, AIS, MICS) Special surveys (e.g. BSS, 2 nd generation surveillance surveys, school surveys, health facility surveys) Programme monitoring (e.g. patient tracking, Health Information System) Key informants Desk review HIV surveillance / estimates
NUMERATORS AND DENOMINATORS Most indicators have a numerator and denominator to calculate a percentage Follow the instructions exactly It is important that the data collection period is consistent for all the information relevant to a specific indicator s numerator and denominator
DATA DISAGGREGATION Countries are expected to submit total values AND all disaggregated values as per the instructions in the Guidelines Data collected through facilities, programme monitoring, and surveys is often disaggregated; the challenge is to ensure the disaggregated data is also available at national level
DATA DISAGGREGATION (continued) Why important? Allows for more effective tracking of resources and programmatic responses Allows for monitoring trends in specified subpopulations Allows for monitoring equity of access to services
IN-COUNTRY REPORTING PROCESS In-country reporting process Country Progress Report Format Submission Process Role of UNAIDS/Geneva
RECENCY & REPRESENTATIVENESS Which data to submit? Ideally, most recent data available nationally representative data But, if data is not representative, most recent survey data which has been reviewed and endorsed by technical experts within the country
STEPS FOR IN-COUNTRY UNGASS REPORTING PROCESS Preparation Planning Planning Submission Implementation
ROLES & RESPONSIBILITIES The overall responsibility of preparing and submitting the Country Progress Report lies with national governments (NAC or equivalent) UNGASS should be routine and integrated in the country s M&E plans, budgets and systems
Preparation PREPARATION 1. Assess country readiness 2. Identify stakeholders 3. Identify funding 4. Identify relevant data tools / sources
ASSESS COUNTRY READINESS Preparation Countries are expected to know their epidemic to determine which of the UNGASS indicators are applicable to the country s epidemiological context to know availability of data
IDENTIFY STAKEHOLDERS Preparation Individuals and organizations who are affected by HIV and/or are involved in the HIV response: Government (including NAC, Ministry of Health, Ministry of Education; Ministry of Finance) Civil society groups (including nongovernmental organizations, people living with HIV) Private sector
Preparation IMPORTANCE OF CIVIL SOCIETY Strategic and tactical expertise Quantitative and qualitative data Valuable perspective on the data
HOW AND WHEN TO INCLUDE CIVIL SOCIETY Preparation Involvement in all phases of the report preparation: Identify relevant civil society organizations Assign roles and responsibility Plan with them data collection and analysis Invite them to participate in workshops Share drafts with them for review and comments Disseminate final report to them
IDENTIFY SOURCE OF FUNDING Preparation What sources are available? No separate funding available for UNGASS reporting, but should be integrated in M&E budgets (government, GFTAM, World bank)
DATA SOURCES Preparation Population-based surveys (e.g. DHS, AIS, MICS) Special surveys (e.g. BSS, 2 nd generation surveillance surveys, school surveys, health facility surveys) Programme monitoring (e.g. patient tracking, Health Information System) Key informants Desk review HIV surveillance / estimates
Preparation DATA CONSIDERATIONS Recency and representativeness of data Requirements for disaggregation Appropriate data process
PLANNING Planning Planning Develop a plan for data collection / collation, analysis and report writing Designate a coordinator Assign roles and responsibilities for each indicator Assign realistic timelines and milestones Secure funding for the entire process (collecting, collating, analysing and reporting the data)
Planning Planning PLANNING (continued) Disseminate the plan Use the plan as a checklist to monitor progress against the timeline and milestones
ACTION PLAN TEMPLATE Planning Planning Steps Activities Timeline Budget Responsible person Stakeholders involved
IMPLEMENTATION Implementation Establish a process for data collecting / collating, vetting and analysis Organise workshops for vetting and triangulation of data Complete the appropriate data forms Draft the Country Progress Report Disseminate it for review and comment Collate comments from stakeholders, including government agencies and civil society
IMPLEMENTATION (continued) Implementation Organize a national consultation workshop to reach consensus with stakeholders on the final report Enter agreed data into CRIS (or equivalent data management system) Validate the indicator data against the report narrative (consistency check) Finalize and submit the report to UNAIDS Identify focal point for communication between UNAIDS Secretariat and country
MONITOR PROGRESS & DEBRIEF Monitor progress throughout the implementation process using the action plan Plan a debriefing meeting to assess success of report (timeliness, comprehensiveness, quality) to agree actions to undertake to strengthen the national M&E system
OVERVIEW OF THE REPORTING PROCESS Identify data needs, data sources, stakeholders, funds Establish plan for data collection, analysis and reporting writing Secure funds, collect/collate and analyse data, complete data forms Draft Country Progress Report, share draft with stakeholders Enter data in CRIS Validate the narrative report against the data in CRIS Reach consensus with stakeholders on final Report & submit timely
FORMAT OF THE COUNTRY PROGRESS REPORT Narrative part (see Appendix 2) Structured to ensure consistent information across countries Provide an in-depth analysis of data regarding the HIV epidemic and response Submission of of report report
UNGASS COUNTRY PROGRESS REPORT [Country Name] Reporting period: January 2006 December 2007 Submission date: [ ][ I. Table of Contents II. III. IV. Status at a glance Overview of the AIDS epidemic National response to the AIDS epidemic V. Best practices VI. VII. VIII. Major challenges and remedial actions Support from country s development partners Monitoring and evaluation environment
ANNEXES Submission of of report report ANNEX 1 Consultation/preparation process for the Country Progress Report ANNEX 2 National Composite Policy Index Questionnaire
COUNTRY PROGRESS REPORT Indicator Data
BENEFITS OF USING THE COUNTRY RESPONSE INFORMATION SYSTEM (CRIS) CRIS is a monitoring system which greatly facilitates UNGASS reporting: It supports all six UN languages [no need for translation] It uses the current Guidelines [no confusion] It uses the correct indicator definitions [no confusion] It facilitates compilation and management of all UNGASS data [saves time] Submission of of report report
BENEFITS OF USING CRIS (continued) It has built-in validity checks [avoids logical errors] It stores available data from population-based surveys [saves time] It facilitates trend analysis as previous UNGASS data is stored [saves time] Data is seamlessly integrated into the global database [no transcription errors, no delays] Submission of of report report
BENEFITS OF USING CRIS (continued) Submission of of report report In addition to use for UNGASS reporting and reporting to donors CRIS can be used for the management of all monitoring information at the national and sub-national levels
SUBMISSION PROCESS Submission of of report report What? Country Progress Report consisting of Narrative section (including Annex 1, Annex 2) CRIS data file (or excel data file, see template on CD) When? 15 January 2008 if not using CRIS 31 January 2008 if using CRIS [preferred] How? E-mail to ungassindicators@unaids.org
WHAT HAPPENS AT UNAIDS/GENEVA? UNAIDS and its partners are responsible for calculating the global indicators A process for collating, analysing and reporting data in coordination with co-sponsors and global partners, including civil society occurs
WHAT HAPPENS AT UNAIDS/GENEVA? Country Report Findings + Global indicators = Report on the Global AIDS Epidemic
WHAT HAPPENS AT UNAIDS/GENEVA? The Report on the Global AIDS Epidemic is prepared by May 2008 and submitted to the United Nations General Assembly The Country Progress Reports and the Global Report will be available on the UNAIDS website at http://www.unaids.org
BENEFITS OF UNGASS REPORTING Provides answers to important questions: What is the status of the epidemic in the country? What are the basic trends in HIV transmission and service coverage? What are the main obstacles to accessing HIV prevention, care and treatment services? What is the quality of services being delivered?
BENEFITS OF UNGASS REPORTING (continued) Provides answers to important questions: Are services being delivered equitably and effectively? What exacerbates these problems? (e.g. policies, laws, resources, politics, customs, organizations, individuals) Who can change this situation? (e.g. elected leaders, bureaucrats, religious leaders, community leaders, traditional leaders, donors, international organizations, NGOs) What are these people/organizations currently doing to address the problems?
BENEFITS OF UNGASS REPORTING (continued) This information can be used to evaluate the effectiveness of the national response analyse linkages between policy, implementation of programmes, behavioural change and HIV prevalence identify gaps in programmes, personnel, technology identify data gaps and data quality issues strengthen the National M&E system
SUMMARY Ownership & participation Consultation & consensus Data vetting, validation, triangulation Timeliness: 31 January 2008 or 15 January 2008 if not submitted in CRIS Focus on systems and process