Draft Inception Report

Transcription

1 The Second Independent Evaluation of UNAIDS Draft Inception Report Submitted by 5 th September 2008

2

3 Table of contents 1 INTRODUCTION WIDER CONTEXT... 1 Evolving and increasingly complex aid environment... 1 Coordination challenges... 2 Sustaining political and financial commitment... 3 Challenging AIDS exceptionalism... 3 Growing emphasis on health systems and health sector response... 3 Supporting comprehensive and effective national AIDS responses... 4 Increasing involvement of civil society and people with HIV... 4 Addressing gender and human rights... 4 Strengthening the governance and administration of UNAIDS... 5 Implementing the Division of Labour... 5 Delivering as One SCOPE OF THE EVALUATION... 6 EVALUATION PURPOSE... 6 ECOSOC OBJECTIVES... 8 OPERATIONAL OBJECTIVES OF UNAIDS... 8 EVALUATION CRITERIA EVALUATION DESIGN EVALUATION FRAMEWORK APPROACH METHODOLOGY Summary of methods EVALUATION PROGRAMME COUNTRY VISITS Sample size Country selection Schedule of country visits Team composition Country visit preparation, logistics and duration Country visit methodology COUNTRY VISIT PROGRAMME COUNTRY VISIT REPORTS REGIONAL MEETINGS AND STAKEHOLDERS GLOBAL MEETINGS AND STAKEHOLDERS WORK PLAN ORGANISATION AND MANAGEMENT Roles and responsibilities Team structure Partnerships ACTIVITIES AND OUTPUTS WORK PLAN ANNEX 1 HIV AND AIDS DEVELOPMENT ASSISTANCE: KEY MILESTONES ANNEX 2 EXAMPLE QUESTIONS ON GOVERNANCE ANNEX 3 TABLES OF DATA FOR COUNTRY SELECTION i

4 ANNEX 4 DESCRIPTION OF ITAD, HLSP AND PARTNERS ii

5 Abbreviations and acronyms ART ARV ASAP BP CBO CCA CCM CCO CDC CDF CERPOD CGIAR CPA EC ECOSOC EPI ESP ET FGD GAVI GEF GFATM GIST GNP+ GTT GTZ HQ ICPA ICT IDU ILO IPAA IR LICUS M&E MAP MDG MERG MOE MOH MSF MSM MTCT NAC NACP Anti Antiretroviral Therapy Antiretroviral Aids Strategy Action Programme Best practice Community based organisation Common Country Assessment Country Coordination Mechanism (Global Fund) Committee of Cosponsoring Organisations Centers for Disease Control and Prevention Comprehensive Development Framework Centre d Etudes et de Recherche sur la Population pour le Developpement Consultative Group for International Agricultural Research Country Programme Advisor (UNAIDS) European Commission Economic and Social Council of the United Nations Expanded Programme on Immunisation Evaluation Supervisory Panel for the Five-Year Evaluation of UNAIDS Evaluation team Focus Group Discussion Global Alliance for Vaccines and Immunisation Global Environment Facility Global Fund for AIDS, TB and Malaria Global Joint Problem solving & Implementation Support Team Global Network of PLHIV Global Task Team Gesellschaft für Technische Zusammenarbeit (German Technical Cooperation) Headquarters Inter-Country Programme Advisor Inter-Country Team (UNAIDS Secretariat) Injecting drug user International Labour Organisation International Programme for Aids in Africa Inception Report Low Income Country Under Stress Monitoring & Evaluation Multi-country HIV/AIDS Programme (World Bank) Millennium Development Goals Monitoring and Evaluation Reference Group of UNAIDS Ministry of Education Ministry of Health Medecins Sans Frontieres Men who have sex with men Mother to child transmission National AIDS Commission/Council National AIDS Control Programme iii

6 OC OCP OECD DAC PCB PEPFAR PLHIV RFP SCF STI/STD SWOT TB TDR TOR TSF UBW UCC UN UNAIDS UNDAF UNDP UNESCO UNFPA UNGA UNHCR UNICEF UNODC UNOPS UNTG USAID WB WFP WHO Oversight Committee Onchocerciasis Control Programme Organisation for Economic Cooperation and Development Development Assistance Committee Programme Coordinating Board (UNAIDS) President s Emergency Plan for AIDS Relief People Living with HIV Request for proposals Save the Children Fund Sexually transmitted infection/disease Strengths, weaknesses, opportunities, threats (a management tool) Tuberculosis Tropical Disease Research Programme Terms of Reference Technical Support Facility Unified Budget and Workplan UNAIDS Country Coordinator United Nations United Nations Joint Programme on HIV/AIDS United Nations Development Assistance Framework United Nations Development Programme United Nations Educational, Scientific and Cultural Organisation United Nations Population Fund United Nations General Assembly United Nations High Commissioner for Refugees United Nations Children s Fund United Nations Office on Drugs and Crime United Nations Office for Project Services United Nations Theme Group United States Agency for International Development World Bank World Food Programme World Health Organisation iv

7 1 Introduction 1.1 This report sets out the objectives and design of the second independent evaluation of UNAIDS, based on the terms of reference and the approach presented by the evaluation team (ET) in their technical proposal. Following a brief overview of the context within which UNAIDS is operating, the report summarises the purpose and scope of the evaluation, describes the evaluation design and provides details about the proposed plan of work. 1.2 The report has benefited from a process of consultation with stakeholders. A first draft was circulated to members of the UNAIDS Programme Coordinating Board (PCB) and discussed at a workshop held in Geneva on 15 th and 16 th September Additional comments were also received in writing. Errors of fact have been corrected and improvements made to the evaluation design as far as practicable within the terms of the evaluation mandate and budgeted resources. [Text to be finalised after the stakeholders workshop.] 2 Wider context 2.1 This section highlights issues and challenges that influence the role of UNAIDS, the scope for UNAIDS to achieve its mandate and the way in which UNAIDS operates. All have a bearing on the evaluation. It sets the scene for the discussion of the evaluation design and methodology, rather than being an exhaustive or definitive context analysis. Evolving and increasingly complex aid environment 2.2 The nature of development assistance for HIV is complex and rapidly evolving. Recent years have seen a proliferation of organisations and funding channels and an increase in targeted funding for AIDS. Developments include the significant resources provided through channels such as the President s Emergency Programme for AIDS Relief (PEPFAR) and the Global Fund to Fight AIDS, TB and Malaria (Global Fund) (see Figure 1), the emergence of new partners such as the Bill and Melinda Gates, and Clinton Foundations and the launch of new aid instruments such as UNITAID (see timeline in Annex 1). 2.3 Strategic engagement with the global aid architecture is a critical issue for UNAIDS. For example, the evaluation will need to explore the relationship between UNAIDS and the Global Fund, since Global Fund resources are expected to increase substantially in the next few years. In addition, a considerable proportion of UNAIDS country staff time is spent on technical support for Global Fund processes, and duplication of coordination mechanisms specifically National Aids Commission/Councils (NACs) and Country Coordinating Mechanisms (CCMs) remains an issue of concern in certain contexts. 1

8 Figure 1: Global spending for HIV/AIDS (US$ millions) * * Projected funding Source: UNAIDS, Avg Price of ARVs $7,944-20,224/ Person per year World Bank MAP 3 by 5 GFATM PEPFAR G8 Avg Price of ARVs $50-200/ Person per year 2 Coordination challenges 2.4 While the considerable increase in financial resources available for the AIDS response is welcomed, the resultant increase in complexity of the aid environment has made it imperative to deliver better coordination, harmonisation and alignment. The Paris Declaration on Aid Effectiveness represents international donor and multilateral commitment to reforming the ways in which they deliver and manage aid. In the context of AIDS, international recognition to use resources and coordinate partnerships more effectively has led to the development of a number of initiatives that are based on the Paris Declaration principles. These initiatives have been led by UNAIDS in conjunction with country partners and include the Three Ones Principles (the Three Ones), the Global Task Team on Improving AIDS Coordination among Multilateral Institutions and International Donors (GTT) and follow-up actions to these principles and international commitments. 2.5 The GTT recommended actions to strengthen harmonisation and alignment of AIDS support with country priorities. The evaluation will draw on the GTT independent assessment conducted by HLSP in 2007, which reviewed progress in technical support provision to the national AIDS response brokered by the UN system, and harmonisation and alignment of international partners. The evaluation will assess follow up on its recommendations. 2.6 The Three Ones encompasses one national AIDS plan or framework for action, one national AIDS coordinating authority and one national AIDS monitoring & evaluation (M&E) framework. The Three Ones and support to national AIDS coordinating authorities are fundamental to UNAIDS country support. However, progress with implementation of the Three Ones is mixed, as is progress in promoting coordination among partners outside the UN system. In many countries, national authorities report dealing with multiple donors, projects, processes and procedures. Often new AIDS money is off budget outside government planning and budgeting processes resulting in separate plans, operations and monitoring. The evaluation will explore factors influencing progress with the Three Ones and harmonisation and alignment. 2

9 Sustaining political and financial commitment 2.7 Ambitious global goals have been set for HIV and AIDS. The 2001 UNGASS Declaration and 2006 High Level Meeting on AIDS, which led to international commitment to move towards Universal Access to HIV prevention, treatment, care and support by 2010, have helped to keep AIDS high on the political agenda and to mobilise significant funding for the response. 2.8 Despite progress, there is still much to be done if global targets are to be met, and UNAIDS forecasts a growing funding gap. Continued global leadership will be essential to ensure donors and national governments meet existing commitments and provide additional resources required to move towards Universal Access goals. Consideration of the future role of UNAIDS in advocacy for increased and more predictable long-term financing will be central to the evaluation. Challenging AIDS exceptionalism 2.9 The concept of AIDS exceptionalism that AIDS is exceptional in terms of the threat it poses, its complexity and the response required has been fundamental to UNAIDS and to its success in generating political commitment and resource mobilisation, but is currently an issue of debate. Although most observers are not arguing against the exceptional nature of the epidemic, especially in areas of very high prevalence, some have argued for an end to exceptionalism and for the response to be normalised. Those taking the latter view have used the lower revised estimates of global HIV infections in the UNAIDS 2007 AIDS Epidemic Update to argue that the epidemic is no longer a global emergency and that there is no longer a case for AIDS exceptionalism At the same time, evidence is growing to inform the linkages and synergies between HIV, other development goals and broader socioeconomic variables. For example, evidence now indicates that HIV is a disease of inequality, often associated with economic transition, rather than a disease of poverty. This has challenged the fundamental belief, that did underpin UNAIDS approach that poverty increases vulnerability to HIV infection. The evaluation will need to explore the implications of these debates for UNAIDS, including the implications for resource mobilisation to support Universal Access goals at a time when AIDS exceptionalism is being challenged. Growing emphasis on health systems and health sector response 2.11 There is growing recognition that strengthening health systems is essential to achievement of Universal Access. The High Level Forum on Health MDGs noted that without increased support to help build health system capacity in almost all developing countries, the resources mobilised by global partnerships are unlikely to achieve their full potential. The increased focus on health is reflected in, for example, the recent establishment of the International Health Partnership 2.12 There is also growing debate about the impact of vertical funding for AIDS on the health system. In some aid dependent countries, external financing for AIDS exceeds the health sector budget and there are concerns about distortion, crowding out of other health priorities and parallel systems that undermine the public health sector. The extent to which AIDS funding can be leveraged to strengthen health systems, as well as to accelerate progress towards Universal Access, is critical and UNAIDS is currently considering its position on this issue At the same time, while clear gains have been made in Antiretroviral Therapy (ART) provision through the health sector, progress in mainstreaming the response by other sectors has been slow. Rigorous evaluation demonstrating the added value of a multisectoral response has also been limited. 3

10 2.14 These developments have raised concerns about a shift away from a multisectoral to a health sector response to the epidemic and a reduction in non-health sector funding for AIDS. This is a clearly a challenge for UNAIDS, which is mandated to lead and coordinate a multisectoral response to HIV and AIDS. Supporting comprehensive and effective national AIDS responses 2.15 Increased funding is providing opportunities to scale up prevention, treatment and care services but limited national capacity to absorb additional resources is a significant constraint. Obstacles to scaling up HIV prevention, treatment, care and support identified during country consultations in 2006 included: lack of credible, budgeted national AIDS plans; lack of predictable and sustainable financing; weak health systems and shortages of human resources; inadequate supplies of affordable drugs and commodities; and challenges associated with stigma, discrimination, gender and human rights. The evaluation will need to focus on ways in which UNAIDS is assisting countries to address these obstacles, including through technical support Strengthening coordination and provision of technical support by UNAIDS and other international development partners has received increased attention in recent years. Technical support mechanisms such as the Global Joint Problem Solving and Implementation Support Team (GIST), AIDS Strategy Action Programme (ASAP), WHO Knowledge Hubs, and Technical Support Facilities (TSFs) have increased the range of expertise available at country and regional levels The 2007 GTT independent assessment identified a range of challenges related to technical support including limited national capacity to identify technical support needs, lack of progress in assisting countries to develop budgeted technical support plans, supply-driven UN technical support plans, lack of clarity about the respective roles of different technical support mechanisms and limited awareness of these mechanisms and how to access technical support among national partners. The evaluation will need to explore progress in addressing these challenges including UNAIDS initiatives to assess access to and effectiveness of technical support for Global Fund processes and to support development of national technical support plans Consideration of technical support should also assess the extent to which UNAIDS is assisting countries to know their epidemic and respond accordingly. Available evidence indicates that many countries lack adequate data to design an appropriate response and that resource allocation does not always reflect the epidemiology of the epidemic. Refocusing attention on HIV prevention is acknowledged as a major achievement of UNAIDS, but more remains to be done to identify effective approaches to behaviour change and to tackle the social, economic and cultural drivers of HIV infection. Increasing involvement of civil society and people with HIV 2.19 Another major achievement of UNAIDS has been to increase the involvement of civil society and people with HIV in advocacy and service delivery and, through participation in global and national coordinating mechanisms, in policy making and governance. This has been more successful in some countries than others. The evaluation will need to explore the challenges to meaningful involvement and the contribution of civil society and organisations of people with HIV to tackling the epidemic. Addressing gender and human rights 2.20 In 2005, reported data on the use of ARVs failed to detect any notable gender inequities, although there are concerns about the reliability of this data and about inadequate efforts to address barriers that prevent women from accessing treatment. Positive women s organisations 4

11 have highlighted neglect of wider treatment and care needs of positive women and of the specific needs of young women as critical issues. The extent to which UNAIDS has supported policy and practical measures to address gender inequalities and the gender dimensions of the epidemic will need to be considered by the evaluation Stigma and discrimination are a significant constraint to scale up, deterring people from seeking information and services. UNAIDS has made some progress, but more needs to be done to develop and enforce supportive legal frameworks to eliminate AIDS-related stigma and discrimination and protect the rights of people with HIV and those most vulnerable to infection. The needs of sub-populations identified as at particular risk of HIV infection have been neglected. As of 2006, UNAIDS reported that coverage with HIV prevention services only reached 36% of sex workers, 9% of men who have sex with men and 5% of injecting drug users, with few prisoners having access to HIV services. The evaluation will explore issues relating to human rights and the extent to which UNAIDS has succeeded in encouraging low and middleincome countries to focus greater attention on most at risk populations. Strengthening the governance and administration of UNAIDS 2.22 UNAIDS effectiveness in fulfilling its mandate is limited by its governance structure. Cosponsors are accountable to their own structures. Mechanisms for accountability at country level are at an early stage of development. The need to review the mandate of UNAIDS and the roles, responsibilities and modus operandi of governance and accountability structures has been identified by earlier reviews and will be addressed by the evaluation. The evaluation will also explore the business practices of UNAIDS, including steps taken to strengthen planning, reporting and performance management, and the respective roles of the Secretariat and Cosponsors at global, regional and country levels. Implementing the Division of Labour 2.23 To ensure that the UN system responds to requests in a coordinated manner that builds on the comparative advantage of each agency, UNAIDS and its Cosponsors are implementing their Division of Labour initiative. Lead organisations have been identified in 17 technical areas to act as the entry point for the provision and coordination of UN support requested by national partners. UN Joint Teams on AIDS have been established at country level to implement the Division of Labour, develop Joint Programmes of Support that are aligned with national priorities, and harmonise the UN response The imperative for the Division of Labour and Joint Teams came from the GTT recommendations. The recent GTT assessment identified a number of challenges relating to implementation of these recommendations. These include: the efficiency of adaptation and application of the Division of Labour by UN agency country offices; the extent to which the Division of Labour is understood by stakeholders outside the UN system; the process of accessing technical support under the Division of Labour and lack of awareness of the process among national partners; and the extent to which the Division of Labour is bringing about rationalisation of working practices. The evaluation will consider subsequent progress since the GTT assessment and the links between the Division of Labour and Joint Teams and Programmes of Support and wider UN reform. Delivering as One 2.25 Alongside the Division of Labour, changes in the UN system, in particular reforms intended to enable UN agencies to deliver as one, clearly have implications for the UNAIDS Secretariat and Cosponsors. The evaluation will need to take account of progress with UN reform 5

12 in the eight pilot countries and lessons learned from UNAIDS experience of establishing Joint UN Teams and Programmes of Support. UNAIDS response to the challenges to joint working, identified by the recent review of progress with implementing the GTT recommendations, will also be considered The issues listed above will be taken into account alongside development of the questions in the evaluation framework which is set out in Section 4. 3 Scope of the evaluation Evaluation purpose 3.1 The purpose of this evaluation is to assess the efficacy, effectiveness and outcomes of UNAIDS (including UNAIDS Secretariat, the PCB and Cosponsors) at the global, regional and country levels. 3.2 The terms of reference (TOR) explain that the evaluation should relate to how UNAIDS has met its ECOSOC mandate for an internationally coordinated response to the HIV pandemic, the extent to which UNAIDS has met its objectives and the continuing relevance of its mandate and objectives in the current global environment. The TOR set out the focus of the evaluation through a series of questions. These are listed in Table 1 and developed into a more detailed evaluation framework in Section 4. Table 1 Questions for the evaluation from the TOR a) The evolving role of UNAIDS within a changing environment Given the changing global, regional and country environments, the evolving role and priorities of the Joint Programme needs to be clearly defined, especially concerning working relationships with institutions like the Global Fund, PEPFAR, UNITAID, bilateral donors, private sector, civil society, regional organizations and others, all of which have grown in importance since the Five Year Evaluation. To what extent does UNAIDS generate and take advantage of synergies with its partners including HIV vaccine and other appropriate technologies, advocacy, and development partners and organizations of vulnerable populations and people living with HIV? A special focus will be placed upon the role of UNAIDS in monitoring and evaluation of different interventions, policies and strategies implemented across many partners. b) Governance of UNAIDS This evaluation should involve a review of the governance and accountability structures of UNAIDS (Program Coordinating Board, Committee of Cosponsoring Organizations and the Unified Budget and Workplan), and its relationships with the Cosponsors and other UN bodies on a wide range of issues, especially given the organization s expansion, the entry of new partners into the field, and the growing range of activities being undertaken. The evaluation should consider the progress on recommendations of the Global Task Team (GTT), review and the Review of NGO/Civil Society Participation in the Programme Coordinating Board. c) The response to the Five Year Evaluation of UNAIDS Assessing the extent to which UNAIDS has been able to respond to the recommendations and proposed activities that emerged from the Five Year Evaluation based on the PCB decisions is important. It is also necessary to identify any factors, which may have facilitated or limited UNAIDS implementation of these recommendations such as national capacities, availability of resources and resource gaps. Implementation will also have to be evaluated at headquarters, regional and country levels to determine the overall effectiveness, efficiency, equity and acceptability of the Programme. 6

13 Table 1 Questions for the evaluation from the TOR d) The Division of Labour between the Secretariat, Cosponsors, Agencies and Countries The components of UNAIDS, and the operational relationships between Secretariat, Cosponsors and other institutions, like the Global Fund, at headquarters, regional and country levels need to be reviewed. This should also involve evaluating the efficiency of UNAIDS in terms of coordination, consistency and compatibility of activities and programmatic strategies and, how the Division of Labour has affected working relationships in country, taking into account the perspective of national governments. Does UNAIDS fulfill its global coordination role on AIDS? e) Strengthening Health systems The Evaluation should include an assessment of UNAIDS role in strengthening health systems and determine what improvements could be made to strengthen health systems in ways that support UNAIDS objectives. f) The administration of the Joint Programme This involves evaluating how the administration and business practice of the UNAIDS Secretariat has evolved since its creation, including its institutional relationships with WHO and UNDP, and whether it has been flexible and creative enough to keep up with the changing pace and types of demands that have emerged over time, including transfer of resources to countries. Patterns and processes of staff deployment and management will need to be examined. g) Delivering as One UN Reform, Global Task Team (GTT) and the Paris Declaration on Aid Effectiveness all influence the context in which UNAIDS operates. The impact of these changes on how UNAIDS is viewed (by countries, co-sponsors donors and staff) and on how it works to meet its mandate (particularly in countries), should be assessed. Implications and choices for the future should be identified. h) Involving and working with civil society The extent to which UNAIDS has been able to, support, include, engage and incorporate in a meaningful and measurable way the concerns and capacities of civil society, and what types of functional relationships and partnerships have evolved at different operational levels should be reviewed and should be an integral part of all questions to be addressed by this Independent Evaluation. i) Gender dimensions of the epidemic The extent to which gender equality has been incorporated as an integral part of the work of UNAIDS at the global and national levels and the extent to which these issues have been incorporated in national strategies and actions. This must include the degree to which UNAIDS has supported countries in their efforts to address the gender dimensions of the epidemic. The measurement of impact on the gender equality must include: analysis of the development of policy guidance; monitoring of gender-differentiated impact of programmes; systematic disaggregation of data by sex and integration of gender and equality indicators in monitoring and evaluation frameworks; internal capacity for gender analysis and policy guidance. Work on gender norms, work with sexual minorities, including men who have sex with men and transgender communities, should also be examined. j) Technical support to national AIDS responses The outcome of the technical support rendered by UNAIDS through an examination of activities in, and the needs and priorities of affected countries, and the quantity and quality of support rendered, including transaction costs, accessibility of funding, coordination mechanisms such as Joint UN Teams and others designed to enhance service delivery. To what extent does UNAIDS allow for flexible procedures that are adaptable to different national or regional situations? k) Human rights 7

14 Table 1 Questions for the evaluation from the TOR How UNAIDS programmes and policies have contributed to strengthening the rights of vulnerable populations, have addressed issues of gender inequality, stigma and discrimination, the empowerment of vulnerable populations among its priorities, and ensures that programme objectives reflect the priorities expressed by vulnerable populations themselves. This should include mechanisms to enable meaningful participation of vulnerable populations in policy and programme development. l) The greater and meaningful involvement of People living with HIV The extent to which UNAIDS has enabled the active and meaningful engagement of people living with HIV through the: transparent and democratic selection processes and choices of representatives; involvement in the design of policy making; involvement in the implementation of programmes; involvement on the monitoring and evaluation of UNAIDS programmes. ECOSOC objectives 3.3 The six original ECOSOC objectives are the cornerstone of the evaluation design and are reproduced here. The notation E1, E2 is used in Table 2 to number the objectives for ease of cross-reference in later sections. Table 2 ECOSOC objectives for UNAIDS E1. To provide global leadership in response to the epidemic E2. To achieve and promote global consensus on policy and programme approaches E3. To strengthen the capacity to monitor trends and ensure that appropriate and effective policies and strategies are implemented at the country level E4. To strengthen the capacity of national governments to develop comprehensive national strategies and implement effective HIV/AIDS activities E5. To promote broad-based political and social mobilization to prevent and respond to HIV/AIDS E6. To advocate greater political commitment at the global and country levels including the mobilization and allocation of adequate resources Operational objectives of UNAIDS 3.4 UNAIDS is an innovative joint venture of the United Nations family, bringing together the efforts and resources of ten UN system organisations in the AIDS response to help the world prevent new HIV infections, care for people living with HIV, and mitigate the impact of the epidemic. UNAIDS helps mount and support an expanded response to AIDS one that engages the efforts of many sectors and partners from government and civil society. 1 The operational objectives for UNAIDS are expressed in the MDGs, especially Goal 6, the 2001 Declaration of Commitment and the 2006 Political Declaration of Committment. 2 All are reproduced in Table About UNAIDS 2 In 2001, Member States unanimously embraced a series of time-bound targets in the Declaration of Commitment on HIV/AIDS (General Assembly resolution S-26/2, annex). In the 2006 Political Declaration on HIV/AIDS (General Assembly resolution 60/262, annex), Member States restated their commitment to achieve the time-bound targets agreed on in 2001 and to move towards universal access to HIV prevention, treatment, care and support by

15 Table 3 Millennium Development Goals and Declaration of Commitment Millennium Development Goal 6: Combat HIV/AIDS, malaria and other diseases Target 1: Halt and begin to reverse the spread of HIV/AIDS Target 2: Achieve, by 2010, universal access to treatment for HIV/AIDS for all those who need it In 2001 Heads of State and Government Representatives of 189 nations gathered at the firstever Special Session of the United Nations General Assembly on HIV/AIDS. They unanimously adopted the Declaration of Commitment on HIV/AIDS, acknowledging that the AIDS epidemic constitutes a global emergency and one of the most formidable challenges to human life and dignity. The Declaration of Commitment covers ten priorities, from prevention to treatment to funding. It was designed as a blueprint to meet the Millennium Development Goal of halting and beginning to reverse the spread of HIV/AIDS by Evaluation criteria 3.5 In addition to these specific topics, the evaluation will follow the established OECD DAC evaluation criteria in the way that the design is structured and questions are organised. Table 4 summarises those criteria. Table 4 Summary of Definitions of DAC Criteria for Evaluating Development Assistance Relevance: The extent to which UNAIDS reflects stakeholder priorities and policy objectives, is consistent with beneficiaries requirements, country needs, global priorities, partners and donors, policies. Effectiveness: The extent to which the programme has achieved its objectives or are expected to be achieved, taking into their account their relative importance. Efficiency: Have the objectives been achieved through use of the least costly resources possible? How economically resources/inputs (funds, expertise, time etc.) are converted to results? Impact: The positive and negative changes produced by the programme, directly or indirectly intended or unintended. Sustainability: The continuation of benefits from a development intervention after major development assistance has been completed. The probability of long-term benefits. The resilience of the risk of the net benefit flows over time. 9

16 4 Evaluation design Box 1 Summary of the proposed approach and methodology The main building block is an Evaluation Framework, which develops detailed questions, indicators, data sources and methods from the evaluation question topics in the TOR and our understanding of contextual issues. We put forward a conceptual framework that provides a structure for analysis and reporting, linking the follow-up to the Five-year Evaluation through the current issues to a forward-looking perspective. The approach is issue-driven, led from a country perspective and participatory. Data collection is based on proven tools and standardised methods. A set of criteria for country selection are presented and 12 countries proposed. The work plan prioritises a country perspective with a pilot country visit early on. We also provide time for visits to headquarters of all Cosponsors. The proposed country visits are structured to enable comparative analysis in the final report. Our team has a strong regional perspective, good gender balance and sound technical expertise in evaluation, HIV and AIDS and health systems. Evaluation framework 4.1 To translate the questions for the evaluation and the contextual issues identified in the preceding section into a programme of work, we have developed an Evaluation Framework that structures the issues and questions as indicators that can be measured or assessed during the evaluation. The Evaluation Framework also identifies the range of documents to be reviewed and key informants to interview for each question. We see the framework as being part of a process rather than simply an end product. We will finalise the framework in consultation with the Oversight Committee during the Inception Stage to ensure there is clarity and agreement about what is required and how the evaluation structure and methodology are derived from that. 4.2 The tables overleaf set out the framework structured around questions a) to l). 10

17 Evaluation Framework Question: a) The evolving role of UNAIDS within a changing environment Hypothesis: UNAIDS has evolved in response to the changing global environment and has remained relevant and effective Issues & detailed questions Indicators Sources of data Methods 3 The evolving role and priorities of the Joint Programme Clear and unambiguous policy statements in PCB documents and public information Review of governing body and operational documents Document review against structured checklist Policy timeline Working relationships with institutions like the Global Fund, PEPFAR, UNITAID, bilateral donors, private sector, civil society, regional organizations and others To what extent does UNAIDS generate and take advantage of synergies with its partners including HIV vaccine and other appropriate technologies, advocacy, and development partners and organizations of vulnerable populations and people living with HIV? A special focus will be placed upon the role of UNAIDS in monitoring and evaluation of different interventions, policies and strategies implemented across many partners. Existence of a written statement about working relationships Perceptions of UNAIDS/Cosponsors and other institutions about the working relationship Review of governing body and operational documents Interviews with policy advisors, officials and representatives Document review against structured checklist Topic-list semi-structured interviews Examples of synergies Interviews with key informants Appreciative enquiry Extent to which UNAIDS M&E programmes cover these issues Review of planning documents and M&E reports for Secretariat and Cosponsors Document review against structured checklist 3 An explanation of specific techniques that we plan to use follows later 11

18 Question: b) Governance of UNAIDS Hypothesis: Governance of UNAIDS has responded to the recommendations of analytical reports, and has enabled the development of improved accountability and performance in line with the objectives of UN Reform Issues & detailed questions Indicators Sources of data Methods A review of the governance and accountability structures of UNAIDS (Program Coordinating Board, Committee of Cosponsoring Organizations and the Unified Budget and Workplan), and its relationships with the Cosponsors and other UN bodies Clear and transparent arrangements known to staff and governing body Accountability meets emerging UN standards: following proper processes, delivering planned and modified outputs with approved resources, measuring the planned outcomes of interest, demonstrating the contribution being made by the cosponsor to the accomplishment of the planned outcomes, and demonstrating what was learned in delivering the outputs, and what changes were made as a result. Analysis of governing body documents Interviews with Executive Director, senior management and governing body members Interviews with CCO members and Cosponsor focal points Interviews with representative of client groups, PLHIV and networks Document review against structured checklist Topic-list semi-structured interviews RACI matrix analysis Progress on recommendations of the Global Task Team (GTT) review and the Review of NGO/Civil Society Participation in the Programme Coordinating Board Number of recommendations adopted Progress towards full implementation of adopted recommendations Evidence of changes in practice and performance Analysis of governing body documents Interviews with senior management and governing body members Interviews with CCO members and Cosponsor focal points Interviews with representative of client groups, PLHIV and networks Document review against structured checklist Topic-list semi-structured interviews 12

19 Question: c) The response to the Five Year Evaluation of UNAIDS Hypothesis: UNAIDS responded satisfactorily to recommendations from the Five Year Evaluation and has improved performance within the framework of its ECOSOC objectives Issues & detailed questions Indicators Sources of data Methods Assessing the extent to which UNAIDS has been able to respond to the recommendations and proposed activities that emerged from the Five Year Evaluation based on the PCB decisions 4 Number of recommendations adopted Development of a responsive workplan Reports of progress against the workplan Evidence of performance implementing recommendations UBW and planning documents of the Secretariat and Cosponsors Interviews with past and present senior managers in the Secretariat and Cosponsors Document review against structured checklist Topic-list semi-structured interviews Review of factors which may have facilitated or limited UNAIDS implementation of these recommendations such as national capacities, availability of resources and resource gaps Reports of progress against the workplan Perceptions of officials about constraints to implementation Progress reports to the PCB Interviews with key informants Document review against structured checklist Topic-list semi-structured interviews 4 See a list of the recommendations at Annex X 13

20 Question: d) The Division of Labour between the Secretariat, Cosponsors, Agencies and Countries Hypothesis: Division of labour has led to a more focused relationship by the Secretariat and Cosponsors and reduced duplication and overlap of activities and functions Issues & detailed questions Indicators Sources of data Methods Evaluating the efficiency of the Joint Programme in terms of coordination, consistency and compatibility of activities and programmatic strategies and, how the Division of Labour has affected working relationships in country, taking into account the perspective of national governments. Follow-up on GTT review recommendations Establishment and effective functioning of Joint UN Teams on AIDS Number of funded joint programmes aligned with national priorities in operation Nature of funding mechanisms; use of Spanish Millennium Challenge Fund Evidence of parties changing programmes to respond to division of labour Extent of overlap or contested areas of programmes Cosponsor commitment and resources allocated to joint working; accountability mechanisms in place UNAIDS and Cosponsor biennial work plans at country level RC, UCC, CD/RR at country level Directors of NACs or equivalent bodies Donor representatives in country including GFATM and PEPFAR etc. NGO and HIV organisations in country Topic-list semi-structured interviews Focus group meetings of donors, and other organisations Follow up on GTT recommendations Partnership assessment Analysis of joint programmes and programming against recommendations set out in: UNDG (May 2006) Proposed Working Mechanisms for Joint UN Teams on AIDS at Country Level - Guidance Paper UNAIDS (2008) Second Guidance Paper: Joint UN programmes and teams on AIDS Practical guidelines on implementing effective and sustainable joint teams and programmes of support Does UNAIDS fulfil its global coordination role on AIDS? Extent of agreement among Cosponsors and other key stakeholders about UNAIDS role Existence of a clear statement on UNAIDS coordination role Effective functioning of IATTs Allocation of UNAIDS staff & resources Implementation of Three Ones and Paris Declaration commitments Roll-out and implementation of CHAT Documents of the Secretariat and PCB Interviews with senior managers in Cosponsors and other parties Document review against structured checklist Topic-list semi-structured interviews 14

21 Question: d) The Division of Labour between the Secretariat, Cosponsors, Agencies and Countries Hypothesis: Division of labour has led to a more focused relationship by the Secretariat and Cosponsors and reduced duplication and overlap of activities and functions Issues & detailed questions Indicators Sources of data Methods Perceptions by stakeholders of UNAIDS performance 15

22 Question: e) Strengthening Health systems Hypothesis: UNAIDS has taken a constructive approach to incorporating health system issues in overall strategy and in the support it provides at country level Issues & detailed questions Indicators Sources of data Methods To what extent have the Secretariat and Cosponsors strengthened health systems through their work; and to what extent has others work on health system strengthening supported the achievement of UNAIDS objectives? UNAIDS approach to health system issues clearly articulated Agreements with WHO and other relevant Cosponsors Clearly articulated UNAIDS role vis-à-vis international health initiatives and key actors; and plan of action Existence of Health System issues in national HIV strategies Major donors fund programmes and activities for health systems strengthening Existence of cross linkages between health sector and HIV strategies Mechanisms in place to track use of HIV funding for health systems strengthening Documents of the Secretariat and PCB Documents from NAC or equivalent and health sector strategy/prsp Senior officials in health sector agencies Document review against structured checklist Policy timeline Topic-list semi-structured interviews 16

23 Question: f) The administration of the Joint Programme Hypothesis: The Joint Programme has evolved to take account of the changing environment and develop improved ways of managing its institutional relationship with the Cosponsors Issues & detailed questions Indicators Sources of data Methods How the administration and business practice of the UNAIDS Secretariat has evolved since its creation, including its institutional relationships with WHO and UNDP, and whether it has been flexible and creative enough to keep up with the changing pace and types of demands that have emerged over time, including transfer of resources to countries. Analysis of changing organisation and management Business practices in the Secretariat simplified and harmonised with a focus on implementation of IPSAS and ERP Arrangements for the relationship with WHO and UNDP (committee; designated focal point etc) Performance and evaluation framework in place Perceptions of key informants on the relationship Efficient mechanisms for transfer of resources to countries (UBW and PAF) Snapshots of organisational structure and management arrangements from before and during the evaluation period Present and past policy advisers and managers Review of organisation charts and flow diagrams Document review against structured checklist Topic-list semi-structured interviews The revised Working Arrangement between UNDP and UNAIDS (2008, superseding WA of 1996) for provision of administrative services by UNDP Patterns and processes of staff deployment and management will be examined Staffing complement; grade and technical composition and deployment Effective recruitment, development, deployment and performance systems in place UNAIDS Secretariat HR records before and during the evaluation period Quantitative and qualitative analysis of staffing Results of the 2007 evaluation of UNAIDS recruitment process Increase in positions at country level from 2003 to 2005 reflected breakdown shown in Directions for the Future: Unifying and Intensifying Country Support (p. 65) and the 2006/07 and 2008/2009 biennial rollout plans. 17

24 Question: g) Delivering as One Hypothesis: The broader UN reform process and implementation of the Paris Declaration on Aid Effectiveness has fostered greater acceptance of UNAIDS approach both within the UN and by the wider range of stakeholders at country level. Issues & detailed questions Indicators Sources of data Methods Assess the impact of UN Reform, Global Task Team (GTT) and the Paris Declaration on Aid Effectiveness on how UNAIDS is viewed (by countries, co-sponsors donors and staff) and on how it works to meet its mandate (particularly in countries). Implications and choices for the future should be identified. Extent of specific provisions in UNAIDS programmes and policy statements Joint programme integrated into the One UN pilot Actions taken in response by Secretariat and Cosponsors Perceptions of stakeholders on how UNAIDS is viewed Secondary data: UNAIDS Evaluation of Progress in Implementing GTT Recommendations; Reports from the evaluation of the Paris Declaration; working documents on UN Reform Documents of the Secretariat and PCB Documents from NAC or equivalent and PRSP Interviews with senior managers in Cosponsors and other parties Document review against structured checklist Topic-list semi-structured interviews Question: h) Involving and working with civil society Hypothesis: Inclusion of civil society has improved the orientation and effectiveness of the national response and helped guide regional and global policy-making Issues & detailed questions Indicators Sources of data Methods The extent to which UNAIDS has been able to support, include, engage and incorporate in a meaningful and measurable way the concerns and capacities of civil society, and what types of functional relationships and partnerships have evolved at different operational levels A plan for working with civil society Progress against that plan globally, regionally and at country level Funding allocated by donors and governments for CS organisations to implement programmes and services Civil society representation on global and national policy-making bodies A process to learn and share information Documents of the Secretariat and PCB Documents of the Cosponsors Interviews with representatives of civil society and umbrella bodies globally, regionally and at country level e.g. GNP+, ICW, UK NGO AIDS and Development Consortium, International HIV/AIDS Alliance Document review against structured checklist Focus group discussions with representatives of civil society at country level Web-based survey of representatives of civil society at regional and global level 18

25 Question: i) Gender dimensions of the epidemic Hypothesis: UNAIDS has identified gender issues and worked to respond to gender dimensions in programming and relations with countries Issues & detailed questions Indicators Sources of data Methods The degree to which UNAIDS has supported countries in their efforts to address the gender dimensions of the epidemic Analysis of the development of policy guidance; Monitoring of gender-differentiated impact of programmes; Systematic disaggregation of data by sex and integration of gender and equality indicators in monitoring and evaluation frameworks; Internal capacity for gender analysis and policy guidance. Documents of the Secretariat and PCB Documents of the Cosponsors Documents from NAC or equivalent and PRSP HR records of staffing complement of Secretariat and Cosponsors for gender analysis and policy guidance Secondary data: recent evaluations of Gender at UNDP (2006) and UNICEF (2007) Managers and policy advisers; NAC; representatives of civil society and gender-based groups Document review against structured checklist Topic-list semi-structured interviews Focus group discussion with stakeholders and gender action groups at country level Work on gender norms, work with sexual minorities, including men who have sex with men and transgender communities, should also be examined Existence of programmes for working on gender norms and sexual minorities at Secretariat and Cosponsors Progress against plans Documents of the Secretariat and PCB Documents of the Cosponsors Documents from NAC or equivalent and PRSP Managers and policy advisers; NAC; representatives of civil society and gender-based groups Document review against structured checklist Topic-list semi-structured interviews Focus group discussion with sexual minorities and gender action groups at country level 19