Managing for the World M&E: Tool for Internal Management

Transcription

1 Managing for the World M&E: Tool for Internal Management MIT Sloan School of Management Global Health Delivery Lab Team: Yuko Amizaki, Terry Hu, Shirley Li, Jeongyeon Shim April, 2011

2 [MODULE OVERVIEW] M&E: TOOL FOR INTERNAL MANAGEMENT Many healthcare delivery organizations in resource-constrained setting focus on M&E for donors, collecting and reporting what the donors specify. However, many organizations do not pay much attention on what they need to measure and how they should interpret the data collected for internal management purposes; nor do they scrutinize the data collected over time to track progress. In one fieldwork example, we have developed a dashboard-style management tool that collect key performance indicators longitudinally to facilitate the use of M&E output in day-to-day management. This module provides a high-level overview of M&E, how they are used by nonprofit healthcare delivery organizations and the challenges they face (especially in the context of the fieldwork organization), how indicators were selected for monthly longitudinal dashboard, and an example of the managerial questions to be asked to use the data for operational improvement. Note that a summary of open-source EMR systems available is included at the end of the document for organizations that aspire to upgrade their patient data collection system as well. 2

3 Agenda What is Monitoring and Evaluation (M&E)? How are M&E outcomes used in nonprofit organizations? What are the common challenges in using M&E results in day-to-day operation? How can an internal dashboard be created to help utilize data in operation monitoring and improvement? 3

4 What is monitoring and evaluation (M&E)? Monitoring Evaluation Definition Benefits Limitation A continuing function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds Regularly tracks indicators for inputs, activities, outputs, outcomes and impacts Facilitates daily management of operation Does not provide an exhaustive view of the project or program, as often focuses on what is easily measureable The process of determining the worth or significance of a development activity, policy or program.. to determine the relevance of objectives, the efficacy of design and implementation, the efficiency or resource use, and the sustainability of results Enables an exhaustive and balanced understanding of the project Facilitates derivation of lessons from the program and their inclusion in decisionmaking for different parties including partner and donor Requires significant resource and time investment, limiting the frequency of implementation World Bank, ~pagePK: ~piPK: ~theSitePK: ,00.html, 4 accessed March 8, 2011.

5 Monitoring and evaluation are related but have slightly different focus Monitoring Evaluation Activities Method Focus Value Timeframe Tracking and documentation of progress Oversight and ongoing analyses Assessment by internal stakeholders such as program lead, managers, and related stakeholders such as community stakeholders and donors Implemented activities Results are achieved Likely to focus on output and outcome Early warning of problems Generation of corrective action or alternative tactics Continuous, as part of daily operation UNICEF, WFP, UNFPA M&A Toolkit 5 In-depth analyses of implication Comparison of performance against predetermined goals In addition to assessment by internal and related stakeholders, can also include assessment by external evaluators Reasons why and mechanism through which the result is achieved Organization s model of change Focus on impact Feedback to strategy Generation of alternative policies Periodic, such as midpoint or end of program review

6 Different types of data collection methods can be used in M&E, depending on the objective of M&E, data availability, and resource availability Use Examples Quantitative methods Measurement of the frequency and extent of change, by sub-categories of target group Reporting of indicators Standardized interviews or surveys with closed-ended questions Qualitative methods Qualitative assessment of the reason and mechanism of change Analyses of reported indicators Free/guided interviews or surveys with open-ended questions Strength Weakness Objective and undisputable data with a precise hard number Possibility to test statistical correlation between a problem and potential drivers Representative of the target population Possibility of comparisons across different groups Proxy of the impact to be measured, rather than actual measurement Limited explanation of reasons and drivers of status quo or change UNICEF, 1991; Gosling, USAID TIPS Number 2, 1996; UNFPA M&A Toolkit 6 Interpretation and understanding of the program background and quantitative data Understanding of the psychosocial and behavioral factors Relatively low cost given small sample size Strength in collecting stakeholder feedback and planning for social change Small subset of the target population, limiting representativeness Potentially disputable objectivity, given the possibility of bias from those involved in the data collection

7 Agenda What is Monitoring and Evaluation (M&E)? How are M&E outcomes used in nonprofit organizations? What are the common challenges in using M&E results in day-to-day operation? How can an internal dashboard be created to help utilize data in operation monitoring and improvement? 7

8 M&E programs and their results can be used for multiple objectives Strategic decision-making Which strategies warrant more investment? How should we revise a specific strategy? How should future programs be formulated? Stakeholder management Why should a donor, partner, or other stakeholder be interested in the given program? How do stakeholders feel about the program? How is the program held accountable to the stakeholders? Operation management What is the quality of the project and how can it be enhanced? How is the project managed and how can it be improved?? Organizational impact and learning What is the impact of the intervention? What are the lessons learned from the project? What components of the strategy were successful and why? UNFPA M&A Toolkit 8

9 Externally, presentations can be used to enhance the organization s reputation as well as financial and intellectual resources Media Report to donors Usage Ask for feedback from experts Brochure of organization Increase presence among international healthcare community Presentation at international conferences Raise follow-up funding Team analysis 9

10 Internally, results of M&E can be used to facilitate organizational development at multiple levels What are the areas of our operation that need improvement? What should be our program objectives and targets? How efficient are we in using available resources? Which areas should we allocate more resources in the future? How successful are our programs and why are they successful? What is our progress against our strategic plans and targets? What is the impact of our programs in the community? Team analysis 10

11 Agenda What is Monitoring and Evaluation (M&E)? How are M&E outcomes used in nonprofit organizations? What are the common challenges in using M&E results in day-to-day operation? How can an internal dashboard be created to help utilize data in operation monitoring and improvement? 11

12 M&E resources are deployed unevenly and often overstretched, especially given differing donor requirements EXAMPLE Organization background Impact on M&E Areas of operation Resource allocation Description Operational focus is on all areas of health, including, but not limited to, HIV/AIDS Areas of operation also include socioeconomic development to improve health outcome of HIV orphans and vulnerable population Program-specific funding drives resource allocation within the organization HIV is a major focus of multiple partners, who also require result and process reporting in specific formats A large share of M&E resource devoted to donor-focused areas, e.g., HIV o Close to 10 different reports generated for HIV operation only o Over 350 health outcome indicators required, many of which further track by gender and age group breakdowns o Further breakdown categories inconsistent with one another Areas with limited donor interest with limited tracking of results Organization interview, Team analysis 12

13 Not all M&E results are useful in daily operation monitoring, calling for careful selection of indicators (1/2) Use baseline data to identify program targets and understand their needs Improve program design Focus of daily operation Program design Improve future program content by compiling lessons learned Incorporate lessons on program process and management learned from previous phases Program follow-up Program implementation Improve program implementation through early identification of gaps and issues Articulate and modify program objectives, and assess progress against activities and outcome targets in the plan Family Health International, Evaluating Programs for HIV/AIDS Prevention and Care in Developing Countries; Team analysis 13

14 Effectiveness indicators Process indicators Not all M&E results are useful in daily operation monitoring, calling for careful selection of indicators (2/2) VCT EXAMPLE Indicator example Program output Service aspects Outcome % of people aware of the service # of people counseled and tested % of people counseled and tested who returned to receive results % of HIV positives referred to appropriate care % of tested and counseled who informed their partners Adequacy of counseling protocol Adequacy of testing protocol Adequacy of counseling and testing actually provided % of clients counseled who are also tested for HIV Accessibility of service Barriers to counseling and testing Change in risky behavior of VCT clients and their partners Change in STI trends in the program area Reduced stigmatization of people affected by HIV Increased community support for people living with HIV/AIDS Process indicators are more appropriate to measure progress in everyday operation Program output indicators are relatively easy to track over time Impact Change in trends in HIV/AIDS incidence and prevalence Reduced mother-to-child transmission in the women targeted Sustained changes in societal norms in the community Family Health International, Evaluating Programs for HIV/AIDS Prevention and Care in Developing Countries; Team analysis. 14

15 Agenda What is Monitoring and Evaluation (M&E)? How are M&E outcomes used in nonprofit organizations? What are the common challenges in using M&E results in day-to-day operation? How can an internal dashboard be created to help utilize data in operation monitoring and improvement? 15

16 An organizational dashboard or scorecard can facilitate monitoring of operation and outcome CONCEPTUAL EXAMPLES Breaking free of the one-page dashboard rule, accessed March 2011; Dashboard vs. Portals: What s the difference?, accessed March

17 Development and use of dashboard is a three-step process Select dashboard indicators Tracking indicators over time Asking appropriate managerial questions Select indicators which are: o Sensitive enough to show short-term results and serve as early warning system o Representative of the focus of activities o Easily measurable for tracking purpose Avoid selecting too many indicators to ensure focus Track selected indicators longitudinally e.g., monthly to easily understand trends in operation Show current data alongside historical data to ensure month-to-month comparison as well as to capture any seasonality Understand managerial implications, such as: o Reason for specific decrease or increase o Impact of any specific initiative in the given period o Expected impact of the change in the longer term Develop action plan, such as: ohow to mitigate an adverse impact ohow to sustain advantage gained Team analysis 17

18 Indicators in the dashboard should capture the health of a given project in the short run HIV/AIDS EXAMPLE Sensitization Counseling Testing Positive results Palliative care CD4 count ART Monitoring Exit Key process output indicators Number of people sensitized on HIV/AIDS per month Number of patients counseled per month Number of individuals that underwent HIV testing per month Number of positive patients per month Number of patients receiving palliative care (total) Number of patients that went through CD4 count per month Number of patients receiving ART (total) Number of ART receivers who were monitored per month Number of ART patients who transferred out per month Suggested for dashboard/ longitudinal tracking A dashboard should include a select number of key output indicators that can show the health of a given project in the short term, e.g., monthly Team analysis 18 18

19 Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Longitudinal tracking of data can facilitate understanding of performance and uncovering any issues HIV/AIDS EXAMPLE HIV tested Received palliative care M F M F HIV tested (+) Currently on ART M F M F Team analysis 19

20 Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec And asking the right questions for operational management HIV/AIDS EXAMPLE HIV tested Received palliative care What explains the surge in testing? Can this be replicated? How effective was a specific initiative? M F What explains the decline despite the increase in HIV+ tests? What % of newly diagnosed receive palliative care? M F HIV tested (+) Currently on ART Are we targeting the right population? How can we ensure that the population at risk are tested? M F What is the reason behind the decrease in 100 M ART receivers? F 50 0 What explains the gender difference in the decline? What steps need to be taken to ensure equal access? Team analysis 20

21 Reference Resources referenced in this document World Bank. What is Monitoring and Evaluation? DHL:True~menuPK: ~pagePK: ~piPK: ~theSitePK: ,00.html. Accessed March 8, World Bank. Monitoring and Evaluation: Some Tools, Methods, and Approaches UNFPA. Programme Manager s Planning Monitoring & Evaluation Toolkit Rehle, Thomas; Saidel, Tobi; Mills, Stephen; and Magnani, Robert. Evaluating Programs for HIV/AIDS Prevention and Care in Developing Countries: A Handbook for Program Managers and Decision Makers. Family Health International (year unknown). Additional resources on M&E Bertrand, Jane T., and Manuel Solís. Evaluating HIV/AIDS Prevention Projects: A Manual for Nongovernmental Organizations. MEASURE Evaluation Manual Series, No. 10. Carolina Population Center, University of North Carolina at Chapel Hill. July (Translation to English 2004). International Center for Research on Women. A Measure of Success: Building Monitoring and Evaluation Capacity in Small, Community-Based Programs New York State Department of Health AIDS Institute. Measuring Clinical Performance: A Guide for HIV Health Care Providers Avahan India AIDS Initiative. Use It or Lose It: How Avahan Used Data to Shape its HIV Prevention Efforts in India. Bill & Melinda Gates Foundation New York State Department of Health AIDS Institute. HIVQUAL Workbook: Guide for Quality Improvement in HIV Care

22 APPENDIX 22

23 Potential dashboard/longitudinal indicators: PMTCT and tuberculosis Suggested for dashboard/ longitudinal tracking PMTCT Tuberculosis Key process output indicators Key process output indicators Family planning Pregnancy Delivery Birth Number of HIV+ women that have accepted family planning Number of HIV+ women using family planning Number of HIV+ women who are pregnant Share of pregnant HIV+ women who are receiving PMTCT Number of HIV+ women giving birth at the health center Number of HIV+ women giving birth with traditional birth att nt Number of babies born with HIV and given ART Advocacy TB testing Positive X-ray Treatment Completion Number of families sensitized in TB Number of TB tests carried out Number of TB tests on HIV patients Share of positive TB test results Number of patients referred for X-ray % of patients who return with X-ray Number of TB patients in treatment Number of HIV+ patients in treatment Number of new TB patients entering treatment Number of TB patients completing treatment Number of patients cured Number of patients with resistance 23

24 Potential dashboard/longitudinal indicators: Antenatal care, immunization, and water & sanitation Suggested for dashboard/ longitudinal tracking Antenatal care, immunization, and other Water and sanitation Key process output indicators Key process output indicators Antenatal care Immunization Malaria Number of birth given at health center Number of birth assisted by traditional health attendants Number of children whose weight is monitored at birth Number of children receiving immunization oat health center oat school and outreach Number of female s receiving TT onumber of pregnant women with TT onumber of girls receiving TT Number of ITN distributed Number of mother who received ITN at birth Sanitation education Latrine Dish rack Access to clean water Number of education talks offered Number of households that attended the talks Number of households with access to latrines Number of households actually using latrines Number of households with dish racks or equivalent Number of households actually using dish racks or equivalent Number of households with access to clean water Number of households using Kyetume s water facility Number of days of interruption at Kyetume s water facility 24 24

25 Potential dashboard/longitudinal indicators: Domestic violence and OVC program Suggested for dashboard/ longitudinal tracking Domestic violence Orphan & vulnerable children (OVC) Key process output indicators Number of changemakers, monthly and cumulative orecruited by officer orecruited by existing change-makers Number of outreach and training participants Number of circles formed and member count Number of domestic violence incidents reported o% receiving counseling from Kyetume o% receiving counseling from the police * For heifer and goat separately OVCs Orphan support Agriculture Counseling /Guidance Rep. health education Training 25 Key process output indicators Number of HIV+ OVCs Number of HIV+ OVCs who receive treatment Number of OVCs that receive direct assistance Number of OVCs that attend school Number of households provided with animals* Number of first female offspring returned* Average milk yield per animal* Number of OVC beneficiaries Number of beneficiaries starting keyhole gardens Number of OVCs counseled Number of child care workers trained Total number of youth counseled Number of schools visited Number of artisan mentors trained Number of OVCs who completed vocational training Number of farmers (caregivers) trained Number of local leaders trained

26 END OF DOCUMENT 26

27 Open source tools for data collection on patient level OpenMRS RyanWhite Careware WorldVistA Patient OS Description Electronic medical records system designed for resource-constrained environment, including clinics Designed for flexibility of data and disease type Developed by Partners in Health and Regenstrief Institute Software to manage and monitor HIV care Automatically produce Ryan White program-specific reports Developed by US HHS s Health Resources and Service Admn. Developed by US VA to be used in veterans health care Includes functionalities targeting clinics and clinician offices such as billing, quality measures, and prescription Healthcare information system to manage patient visits, medications, and other clinical and administrative tasks Sites used Over 120 sites including 5 users in Uganda, e.g., ISS Clinic & Makarere Univ. 800 Ryan White-funded agencies in the US All VA medical centers and outpatient clinics Unknown Tolven OpenXData A platform providing electronic personal health record and clinician health record solution Downloadable from the Web Mobile device based data collection software (e.g., surveys) Peter Wakholi, Makerere University, as managing committee member Internet search, as of March 2011, based on ghd online postings. 27 US, Europe and Asia Partners include academic medical centers and pharmaceutical companies Three sites in Africa and one site in Asia, including Makerere University s project to track attendance across 100 schools