Assessing, by measuring and proving, what was achieved

Transcription

1 Assessing, by measuring and proving, what was achieved Februari 8th 2010 Luxembourg Richard Braam

2 Monitoring and evaluation Monitoring to measure changes over time Evaluation to check if those changes match the objectives

3 Evaluating interventions Process evaluation Effect/outcome evaluation Impact evaluation Page 3

4 Process evaluation Assesses the development and implementation of an intervention Helps to explain results of an intervention in terms of failures and successes within the project Content - Are all project activities executed as intended Scope - Is the intended target group reached Quality - Does the intervention fit the target group Page 4

5 Effect/outcome evaluation Measuring the direct results of the intervention Effects on (target) population level Short term measurement (project+) Page 5

6 Effect/outcome evaluation Designs (1) Single measurement Measurement after the implementation of the intervention No reference point No indication that changes are due to your intervention Baseline End-line design Measurements before and after the intervention show changes within the target population over time Indication for effect of intervention Page 6

7 Effect/outcome evaluation Designs (2) Control groups Treatment - no treatment Gives evidence that the effect of the intervention is exclusive for that part of the target population that was served by the intervention Regular treatment - specific treatment Gives evidence that a specific treatment has an other effect than regular treatment No treatment regular treatment specific treatment Gives stronger evidence for effect specific treatment Page 7

8 Effect/outcome evaluation Designs (3) Randomized Control Trial (RCT) Several experimental and control groups; several conditions; random and double blinded selection; scientific measurements and outcome comparison Gives strongest evidence of intervention effects on population Expensive and complex Page 8

9 Impact evaluation To measure long term impact of the overall objective on population level Long time span and several measurements over time Complex scientific research design Expensive e.g..socio-epidemiological population research.longitudinal follow-up studies Rarely done Page 9

10 Monitoring & Evaluation pipeline Number of projects: ALL: Input/Specific Objective Monitoring MOST: Processevaluation SOME: Results Outcome evaluation FEW: Overall Objective Impact evaluation Source: UNAIDS 2007 Page 10

11 Info management system for Monitoring & Evaluation Logical framework matrixes To guide project management To logically set out objectives, results and activities To describe indicators and means of verification To structure monitoring and evaluation To analyze and present results Page 11

12 Logical framework Matrix Intervention logic Objectively verifiable Indicators Means of verification Risks and assumptions Goal Purpose Outputs Activities

13 Logical framework Matrix vertical intervention logic Intervention logic Goal Purpose Outputs Activities (Horstman, 2002) Description An overall objective greater than that of the project itself. To improve public health. The specific objectives to be reached by implementing the project. The achievement contributes to the overall goal. The products or deliverables of the activities undertaken. Together they will contribute to the purpose of the project. Those things which must be done to achieve the outputs. Example Reduction in new cases of sexually transmitted infections. Decrease of unsafe sex. Increased contraceptive use. Increases safe sex education; the amount of contraceptives distributed. Training outreach workers; creating condom distribution points.

14 Logical framework Matrix horizontal structure Intervention logic Objectively verifiable Indicators Means of verification Risks and assumptions goal purpose outcome activities measure the achievement of the envisaged objectives, results, outcomes and activities are the means and sources of information needed to measure the indicator refer to risk factors and supportive factors that frustrate/help to achieve the objectives

15 Logical framework Matrix Intervention logic Objectively verifiable Indicators Means of verification Risks and assumptions Goal Reduction of sexually transmitted diseases (STD s) in country x. Incidence of STDs among young men and women, from x to y in country x over period x. Annual National behavioral and health survey. Purpose Reduced vulnerability of young man and woman to STD risks. Increase % of sexually active young people who used a condom during their last nonmarital encounter, from x to y. Program impact evaluation report, comparing baseline and endline results. Policy makers and planners support project activities. Outputs Increased knowledge and awareness of STD risk among young people Increase of % of young people who know about condoms and sources of supply, from x to y (boys) and p to q (girls). Baseline endline report. Report of interviews with key informants. Young people are motivated to use and pay for services. Activities Upgrade service outlets. Produce prevention materials. Budget in US$ Service staff $ Materials $ First annual report. List of materials. US$ is not devalued

16 Choosing indicators Indicators are used to measure to what extent the objectives are being reached Indicators are essential for monitoring and evaluation Indicators must be SMART

17 SMART indicators Specific Measurable Appropriate Realistic Time-bound

18 SMART indicators Specific A clear description of the aim: What do we want to reach? Who are involved? Where will it happen? When will it happen? The more specific the easier to work with.

19 SMART indicators Measurable Precisely defined and easily quantified How much are we going to do? How can we measure that? A smart indicator you can see, hear, smell, taste or feel.

20 SMART indicators Appropriate It should fit the local needs, capacities and culture of the target group Project partners must agree on the objectives, accept the aims and activities and be willing to cooperate

21 SMART indicators Realistic There must be a reasonable chance that the objective or aim can be reached How much is possible? Can it be done in the time available? Are the available resources sufficient? Too difficult is frustrating Too easy frustrates motivation

22 SMART indicators Time-bound It must be related to a certain time period When does it start? When is it finished? When is the objective or aim reached? Deadlines must be realistic

23 Formulation of an indicator Step Example (Horstman, 2002) Basic indicator Quantity Quality Time Target Peer educators trained Number of peer educators trained Number of peer educators trained in counselling that passed the test Number of peer educators trained in counselling that passed the test in 2010 Increase from x to y in the number of peer educators trained in counselling that passed the test in 2010

24 Types of indicators Process indicators Are related to the outputs of the project (deliverables, structures created, materials produced/published) e.g. 120 condom distribution points have been created in night life venues in the city before the end of 2009 Impact/outcome indicators Measure broader results achieved by project activities and outputs e.g. Contraceptive use by the target group has increased from 30 to 40% over the last six months after the intervention started Page 24

25 Measuring indicators: Means and methods of data collection to measure what has been achieved Routine project statistics (client records, training registers) Management information (project documents and reports) National or local health statistics Baseline-endline surveys (target group, key informants) Interviews (general or key informants) Focus groups (discussions with clients and/or providers) Observation (client-provider interactions)

26 Monitoring & Evaluation Plan What are you going to evaluate Objectives and indicators How are you going to evaluate/measure it Methods of data collection When, where and how often are you going to collect data Timeframe and milestones Who will provide or collect the data needed Human and financial resources Page 26

27 Monitoring & Evaluation Plan Try to involve a social scientist with experience on M&E Can help to develop M&E Plan Can help to formulate indicators and design logframes Can support in choosing/developing instruments Can support data collection and analyses Page 27

28 References HORSTMAN, R.G. et al., Monitoring and Evaluation of Sexual and Reproductive Health Interventions. A manual for the EC/UNFPA Initiative for Reproductive Health in Asia. London/The Hague: LSHTM/NIDI. UNAIDS, A Framework for Monitoring and Evaluating HIV Prevention Programmes for Most-At-Risk Populations. Geneva: UNAIDS WHO, Selecting Reproductive Health Indicators: A Guide for District Managers. Geneva: WHO On the ECHIM product website you can find an overview of products related to health indicators in the EU: Page 28

29 Richard Braam CVO - Addiction Research Centre Montalbaendreef LC Utrecht, the Netherlands Page 29