Program evaluation: Technical and ethics considerations. The example of the evaluation of the nutrition component of Oportunidades

Transcription

1 Program evaluation: Technical and ethics considerations The example of the evaluation of the nutrition component of Oportunidades LM Neufeld Director, Division of Nutritional Epidemiology Center for Research in Nutrition and Health National Institute of Public Health Cuernavaca, Mexico INSTITUTO NACIONAL DE SALUD PUBLICA

2 Program evaluation is independent of program activities and co-responsibilities Evaluation is a research activity and all research activities must be guided by internationally accepted code of ethics

3 Foundations for the protection of human subjects in research 1. Beneficence Never do harm Maximize benefits and minimize risks 2. Justice Fair distribution of risks and benefits 3. Respect for persons Treat individuals with autonomy Protect vulnerable groups Pregnant women, children, prisoners, sub-ordinates, individuals with decision-making impediments

4 Implementation of the ethical foundations of research 1. Beneficence Risk-benefit analysis As part of research design 2. Justice Participant selection As part of research design and implementation 3. Respect for persons Voluntary participation with informed consent

5 In practice this results in three steps: 1. Careful research design Assumes that the researcher is aware of issues and responsible On-line courses obligatory in many institutes including Clearly address potential ethical issues in research protocol 2. Review of research protocol by independent trained ethics committee Prior to initiation of research activities 3. Informed consent of research participants Responsibility of research team

6 When design and proposal complete Review of research protocol by independent trained ethics committee External to research team may be internal to organization if not exist, seek external committee review Review based on ethics principals: Beneficence, Justice, Respect for persons Consider issues discussed in protocol and others not identified by researcher Ensure potential biases and omissions are avoided

7 Informed consent 1. Information.. Complete details regarding Study objectives Procedures, time commitment Risks, benefits, compensations Where to obtain more information 2. Comprehension Individual has capacity to and has understood all details mentioned and the implications that this may have for them 3. Voluntary participation Complete liberty to refuse and/or continue in research without any repercussion

8 Why such a fuss??!!! Many examples of unethical research in history (distant and recent!) Experimentation with humans Nazi German Obligatory Mutilation, death Discrimination Nuremberg Military Trial 8 August 1945 Arrest of scientists Nuremberg Code

9 The main stipulations of the Nuremberg code 1. Voluntary participation and informed consent 2. Experimentation in humans permissible only with sufficient evidence of no harm (animal, laboratory studies etc.) 3. Scientific evidence clearly justified research 4. Avoid all physical and emotional suffering 5. Research permissible only when death or physical damage are not expected Not the beginning or end of ethics problems in research

10 Technical consideration in evaluation design. Example: Evaluation of the nutrition component of Oportunidades

11 The evaluation of nutrition outcomes presents specific problems for the design of program evaluations 1. Delay between program implementation and detectable impact and magnitude of impact 2. Critical periods for potential impact in certain outcomes 3. Variability and measurement error

12 1. Delay between program implementation and detectable impact and magnitude of impact The expected improvement in many nutritional outcomes is small and takes considerable time before it can be detected E.g.. Impact in child length versus child weight Depends on prevalence and severity of problem in population ability of the intervention to address true nutritional problem (e.g., anemia) program function (e.g., actual intake of micronutrients during observation period)

13 2. Critical periods for potential impact in certain outcomes Improvement in many nutritional outcomes is feasible only if the intervention is implemented during the appropriate critical period E.g., child growth Others nutritional problems may be resolved at later ages, but with lasting implications for outcomes E.g., anemia and child development

14 3. Variability and measurement error For some nutritional outcomes, there are no precise, accurate indicators at an individual level E.g., zinc deficiency For others the variation and problems of measurement error cause difficulty for accurate estimation E.g., dietary intake

15 Solutions are feasible although not always simple 1. Ideally, evaluation should be designed as integral part of program design E.g., If improving child growth is an objective of the program, interventions must be targeted to pregnant and lactating women and children <2 yrs of age AND the evaluation must be conducted in THIS age group Careful evaluation design using evidence from scientific literature, e.g., minimum time to detect impact

16 Solutions are feasible although not always simple 2. Include intermediate variable as well as principal outcomes Know the etiology of the nutritional problems being assessed Know the potential routes by which the program may influence outcomes

17 The direct causes of child malnutrition Malnutrition Poor diet Frequent and/severe Infectious disease Poor access To food Poor care for women and children Poor health and sanitary services Modified from UNICEF, 1998

18 The direct causes of child malnutrition Malnutrition Poor diet Frequent and/severe Infectious disease Poor access To food Poor care for women and children Poor health and sanitary services Monetary transfers Fortified food supplements Nutrition education Health education Components of the Oportunidades program Health services

19 Potential mechanisms of impact of the Oportunidades poverty alleviation program Better nutritional status Better diet Less frequent and less Severe infectious disease Better access to food Better child care practices Improved environment and health care Monetary transfers Fortified food supplements Nutrition education Health education Components of the Oportunidades program Health services

20 Potential mechanisms of impact of the Oportunidades poverty alleviation program Better nutritional status Better diet Less frequent and less Severe infectious disease Better access to food Better child care practices Improved environment and health care Monetary transfers Fortified food supplements Nutrition education Health education Components of the Oportunidades program Health services

21 Solutions are feasible although not always simple 3. Have extensive knowledge of prevalence, severity and direct causes of nutritional problems in the target population Potential for impact Sample size requirements Detection of unexpected and/or undesirable outcomes

22 The prevalence of overweight and obesity are increasing at an alarming rate in Mexico Adult women Obesity Overweight 40 % Year of national nutrition survey Source: Rivera & Sepulveda, Sal Pub Mex 2003

23 All socio-economic strata y geographical regions are affected Adult women Rural Urban 60 % Socio-economic status (deciles) Source: National Nutrition Survey, 1999

24 The program is targeted to the poorest 20% of the Mexican population Adult women Rural Urban 60 % Socio-economic status (deciles) Source: National Nutrition Survey 1999

25 Height and weight status of children 2 to 6 yrs of age in rural Mexico Z-score Height for age 0.42 Weight for height Age (mo) Source: Neufeld et al, in preparation

26 Oportunidades had a positive impact on child linear growth in urban areas Difference in height, Oportunidades-control in 2004 (cm) n = 437 p<0.05 n = 206 p<0.05 n = 96 p=0.05 <24 <12 <6 Child age in 2002 (mo) Paired on: propensity score, age, height in 2002, sex and maternal height Source: Neufeld et al, In preparation

27 Tendency towards higher weight-for-height among Oportunidades beneficiary children Difference in weight for height, Oportunidades-control in p<0.10 NS NS <24 <12 <6 Child age in 2002 (mo) Paired on: propensity score, age, weight for height in 2002, sex Source: Neufeld et al, In preparation

28 In two years, the prevalence of obesity increased by 5% - no impact of Oportunidades Overweight Obese % N=1293 N= Cross-sectional sample beneficiaries and non-beneficiaries combined

29 Significantly larger increase in BMI from 2002 to 2004 among Oportunidades beneficiaries Gain in BMI from 2002 to 2004 (kg/m 2 ) Non-beneficiaries Beneficiaries Matched on propensity score and age. p<0.05

30 Trend towards greater increase in BMI among overweight and obese women incorporated in the Oportunidades program BMI change from 2002 to 2004 (kg/m2) Not overweight or obese Overweight Obese Status on baseline (2002) without Oportunidades with Oportunidades Matched on proponsity score and age. p=0.10 for interaction

31 Summary and conclusions Evaluation protocol must pass through a process of external review for compliance with ethical codes Participation of program beneficiaries is voluntary and must not affect their program benefits

32 Summary and conclusions Evaluation should, ideally be designed prior as an integral part of program design Nutrition evaluation must have solid scientific base Intermediate variables should be included in evaluation to support outcomes Evaluation should have capacity to detect unexpected outcomes