This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike License. Your use of this material constitutes acceptance of that license and the conditions of use of materials on this site. Copyright 2006, The Johns Hopkins University and Duff Gillespie. All rights reserved. Use of these materials permitted only in accordance with license rights granted. Materials provided AS IS ; no representations or warranties provided. User assumes all responsibility for use, and all liability related thereto, and must independently review all materials for accuracy and efficacy. May contain materials owned by others. User is responsible for obtaining permissions for use from third parties as needed.
Monitoring and Evaluation IN ACTION! Duff Gillespie
CASE STUDIES Monitoring for Problem Identification and Resolution Program Evaluation Selected Issues
GLOBAL EPI Problem Testing of vaccine potency found millions of doses that had lost their potency due to cumulative heat exposure Immunization campaigns being compromised because: - children receiving worthless vaccines - questionable vaccines destroyed - lower coverage rates - millions dollars wasted
SOLUTION Vaccine Vial Monitors (VVM) for: Oral polio Measles Hib BCG Yellow fever Hepatitis B Diphtheria Tetanus Pertussis VVM color coded labels allow for easy identification of vaccines over-exposed to heat
MONITORING CONDOM SHIPMENTS Problem Shipments of condoms to Sub-Saharan Africa were declining, despite rise of HIV infections Declines not counterbalanced by other donors Significant condom gap
CONDOM GAP IN AFRICA Pieces Per Man Per Year (1999) 35.0 30.0 25.0 20.0 15.0 10.0 5.0 0.0 Average Availability of Top 6 Countries = 16.9 Additional Condoms Needed to Get All SSA Countries to the level of 16.9 = 2 billion per year Countries In Sub-Saharan Africa Top Six Horizontal line represents average procurement of top 6 countries - Botswana, South Africa, Zimbabwe, Togo, Congo and Kenya). Overall provision of condoms is 4.6 per man per year.
AMELIORATIVE ACTIONS Creation of Condom Fund Formation of Reproductive Health Supplies Coalition Demand Creation ABC battles make corrective actions difficult
TRACKING IMMUNIZATION Problem Stagnation or decline in immunization coverage in Africa Problem identified through surveys, well after serious degrading of EPI Tracking of vaccine shipments or funding levels could have served as early warning system
AMELIORATIVE ACTIONS Reallocation of Funds UNICEF/USAID Taskforce Partnership for Maternal, Newborn and Child Health Field Guidance Global Alliance for Vaccines and Immunization (GAVI) and GAVI Funds (Private-Public Partnership)
SYSTEM EVALUATIONS Problem Very Poor Access to Family Planning, especially in rural areas -services offered by medical personnel -rigid screening -fixed facilities Low Contraceptive Prevalence
SOLUTION Series of quasi-experimental studies -areas with lay CBD providers vs. standard of care -conducted in all geographical regions -over 25 studies
RESULTS Without exception, dramatic increases in treatment areas with modest or no increase in control areas
IMPACT Change in policies and programs CBD is now the norm Dramatic increase in prevalence
SELECTED M&E ISSUES
SELECTED M&E ISSUES Monitoring System must be Timely Utilized Accurate
SELECTED M&E ISSUES Project vs. Impact Evaluation Project: Did the contractor do what you paid them to do? Impact: Did you ask them to do the right thing?
SELECTED M&E ISSUES Program Evaluation from the Get Go IMCI Multi-country Evaluation
SELECTED M&E ISSUES M&E Indicators Frame Program Inputs and Activities Contraceptive Prevalence vs. Continuation Rates
SELECTED M&E ISSUES Many Decisions are Not Evidence Based Recent examples: FDA EC Decision Abstinence Condom Degradation Needle Exchange Programs
SELECTED M&E ISSUES Critical Mass of Evidence Often Needed for Fundamental Changes CBD Family Planning ORT Vitamin A Supplementation
SELECTED M&E ISSUES RESULTS ARE NOT ENOUGH! The Case of Vitamin A Supplementation
SELECTED M&E ISSUES Diffusion of Vitamin A 11 field trials 6 meta analyses
g Key y Events in Diffusion of Vitamin A Number of Medline Citations for MeSH subject Vitamin A limited to humans 900 800 700 600 500 400 300 200 100 0 1975: IVACG established 1975: Sugar fortification in Central America 1972: Secretary of State, Kissinger pledges support for VA to first World Conference on Food in Rome 1972: U.S. devotes $3 million VA 1972: 5 countries begin VA supplementation 1965: U.S. begins fortifying nonfat dry milk for Food for Peace Program 1987: WHO and UNICEF recommend VA for treatment of measles 1984: Congressional Hunger Committee 1984: Jim Grant (UNICEF) epiphany 1983: Sommer article in Lancet 1993: World Bank Development Report highlights higher cost-benefits of VA 1993: Beaton/Fawzi Meta-analysis Meetings setting VA goals 1990: World Summit for Children 1990: UNICEF/WHO Montreal Conference 1992: International Conference on Nutrition 1992: Bellagio 1985: UNICEF focuses on VA 1985: U.S. Congress creates VA funding mandate 1996: CIDA commits to VA capsule procurement 1999: USAID s Global VITA Alliance 1986-1992: Confirmational Studies 1985-1987: USAID funds research confirming VA saves children s lives 1998: WHO/ UNICEF endorse integrating VA and EPI/NIDS 2002: Vitamin A supplementation in over 80 countries 1962 1964 1966 1968 1970 1972 1974 1976 1978 1980 1982 1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 Adapted from EM Rogers. Diffusion of Innovations. Free Press, 2005.