A SYSTEMATIC REVIEW OF THE IMPACT OF AGRICULTURAL INTERVENTIONS ON NUTRITIONAL STATUS OF CHILDREN Edoardo Masset 21/02/2011
ACKNOWLEDGMENTS Edoardo Masset (IDS) Lawrence Haddad (IDS) Alex Cornelius (IDS) Jairo Isaza-Castro (University of Sussex) The study was commissioned and funded by DFID with the support of 3ie
OBJECTIVES OF THE REVIEW Primary objective: systematisation of evidence on impact nutritional of agricultural interventions Secondary objective: an assessment of the state of the literature on this subject to establish its ability to provide credible evidence and guidance for policy.
A SHORT HISTORY OF AGRICULTURE AND NUTRITION Historically there have been three waves of agricultural interventions which have focused on different causes and remedies of undernutrition: 1. Cause is food availability and policy is increasing agricultural productivity and output 2. Cause is food security and policy is increasing incomes, livelihood opportunities, and reducing vulnerability 3. Cause is poverty of the diet and policy is changing the composition of the diet of the poor
AGRICULTURAL INTERVENTIONS INCLUDED IN THE REVIEW Production diversification projects: in particular those promoting dairy production, fisheries, vegetable gardens and livestock Bio-fortification projects: by conventional crop breeding or genetic engineering that increase the content of iron, zinc and vitamins in staple crops
INTERVENTIONS INCLUDED AND EXCLUDED FROM THE REVIEW Included Biofortification Home gardening Aquaculture Small scale fisheries Poultry development Animal husbandry Dairy development Excluded Irrigation Watershed development Credit and microfinance Land reforms Marketing Agricultural extension Food processing and storage
PREVIOUS SYSTEMATIC REVIEWS Review Period covered Studies Interventions Nutritional impact Berti et al. (2003) 1985-2001 30 All agriculture: home gardens, animal husbandry, cash cropping, irrigation, land reforms, credit and Mixed results World Bank (2007) extension, duck-fish promotion 1985-2007 52 All agriculture: agricultural commercialisation, horticulture, animal source food, and mixed interventions Ruel (2001) 1995-1999 14 Interventions designed to increase production and intake of micronutrient-rich food through: home gardens, animal husbandry, aquaculture and nutrition education Leroy and Frongillo (2007) Kawarazuka (2010) Not specified (but oldest study is 1987 and most recent is 2003) Not specified (but oldest study is from 2000 and most recent is in 2009) 14 Animal interventions: aquaculture, dairy production and poultry production Mixed results Some evidence of impact on vitamin A intake but evidence is scant and studies are poorly designed Some evidence of impact but few studies available and often poorly conducted 23 Aquaculture and small-fisheries Few studies available and very little evidence of impact
SUMMARY OF RESULTS FROM PREVIOUS REVIEWS Scarcity of studies: very few studies assessed the nutritional impact of agricultural interventions Mixed results: the nutritional outcomes observed were not always positive Weaknesses of research: all reviews stressed several methodological weaknesses of the studies reviewed
PATHWAYS OF IMPACT OF AGRICULTURAL INTERVENTIONS ON NUTRITION Participation in the programme Technology adoption household income Diet composition food expenditure Caloric, protein and micronutrient intake Nutritional status
STEPS OF THE PROGRAMME THEORY Farmers join the programme Farmers adopt new production technologies New technologies increase agricultural productivity and income, and change the food basked Higher incomes and better diets result in improved nutritional status
OUTCOME INDICATORS EMPLOYED IN THE REVIEW Indicator Programme participation Income Description Characteristics of targeted population and participation rates Total household income Diet diversity Consumption of food rich in calories, protein and micronutrients Micronutrient Iron and Vitamin A intake intake Nutritional status Prevalence rates of stunting, underweight and wasting among children under 5
SEARCH METHODOLOGY: EXCLUSION CRITERIA Date: if study was produced before 1990. Language: if it was not written in English. Geographic location: if it was conducted in a high income country as classified by the World Bank Intervention: if it did not investigate the impact of agricultural interventions on any of the key outcome indicators identified Study design: if it did not employ a credible methodology for assessing programme impact
WHAT IS A CREDIBLE METHODOLOGY? A study methodology was considered credible when the impact of the programme on the outcome indicators was based on a counterfactual analysis: a the programme group was compared to a control group We categorised studies in the following way: No control group; Before-after comparison of participants; Comparison of participants and non-participants ; Cross-sectional projectcontrol comparison; Cross-sectional project-control comparison over time; Panel project-control comparison over time; Randomised field trials Studies not employing a control group and before-after studies were excluded. Studies comparing participants and nonparticipants were included if they matched project and control observations. Studies based on longitudinal observations and randomised field trials were accepted.
SEARCHED DATABASES We employed 10 databases The published literature (Econlit, IBSS, PubMed and Web of Science) The unpublished literature (Agris, Eldis, IDEAS, IFPRI, Jolis and World Bank) Strategy adopted: Back referencing from key studies Forward citations of key studies in google scholar Search of studies referenced in previous reviews Contacted experts in the field for further studies
RESULTS OF THE SCREENING PROCESS 10,855 Search results 7,239 studies 3,616 duplicates excluded Removal of duplicates 6,932 studies exlcuded 307 studies included Exclusion criteria (based on title and abstract) - published before 1990 - study not in English - high income country - does not report outcome indicators 284 studies excluded 23 studies included Exclusion criteria (based on full report) - study methodology: no control group and no adjustment for selection bias
CHARACTERISTICS OF THE SELECTED Programme type Search results STUDIES 1 st stage screening 2 nd stage screening Biofortification 833 87 2 Home gardens 1,347 65 16 Fisheries 2,088 81 3 Dairy development Animal husbandry 1,709 38 1 1,262 36 1 TOTAL 7,239 307 23
IMPACT ON PROGRAMME PARTICIPATION Studies described the population targeted by the interventions in general terms: poor geographic areas, women, poor households, remote communities No study reported participation rates No study described the socio-economic characteristics of the programme participants or estimated a model identifying the determinants of participation Little is known about: the impact of these interventions on specific vulnerable groups; the targeting efficiency of the interventions; the characteristic of programme participants.
IMPACT ON INCOME Study Impact Description Hoorweg et al. (2000), Kenya Marsh et al. (1998), Bangladesh Murshed-e-Jahan (2010), Bangladesh Nielsen et al. (2003), Bangladesh Schipani et al. (2002), Thailand Positive (statistically significant at 0.001) Positive (no statistical test) Positive (no statistical test) Positive (no statistical test) Positive (no statistical test) Income among dairy farmers is 40% higher than among non dairy farmers Households with home garden have slightly higher incomes Income increases more rapidly among farmers in aquaculture programme and at the endline is 40% higher in the project group Income is 15% higher among households in a poultry promotion programme Income is 60% higher among families with home gardens
IMPACT ON INCOME The interventions have a positive impact on the production of the food item promoted by the intervention But this is an imprecise measure of income because of substitution effects in labour supply Large effects found on total household income but no statistical tests were reported Given the low response of food consumption and calories to income changes, the impact of these interventions on nutrition via income effects is unlikely
IMPACT ON THE DIET 19 studies assessed impact on diet. 2 studies undertook no statistical test, 4 found no statistically significant impact and 13 found a significant and positive impact on the consumption of food targeted by the intervention. Agricultural interventions change the diet of the beneficiary households in the expected way But substitution effects in consumption are likely and overall impact of the interventions on the diet of the poor remains unexplored Changes in the diet are independent of the unit of observations within the family (children, mothers or all household members): no support to the hypothesis that women and children are discriminated in the allocation of food
IMPACT ON MICRONUTRIENTS INTAKE (IRON) Study Iron measurement Statistical significance of the difference Impact Olney et al. (2009) Schipani et al. (2002) Roos et al. (2003) Talukder et al. (2010) hemoglobin n.s. No differences in levels among women and children under 5 Serum ferritin and hemoglobin Food consumption Anaemia prevalence n.s. n.s. ** (in Nepal) * (in Banglades h) No differences in levels among women and children under 5 No difference in household iron intake Difference found among non-pregnant women in Bangladesh and Nepal but not in Cambodia
IMPACT ON MICRONUTRIENTS INTAKE (IRON) Data on iron intake were collected and reported in different ways and cannot be summarised 5 studies undertook tests for impacts on iron intake. 4 tests showed no statistically significant difference at 5% and one showed a positive impact at 5%. One study reported a statistically significant reduction in anaemia prevalence among nonpregnant women in project areas and no change in non intervention areas. Poor evidence of impact on iron intake
IMPACT ON MICRONUTRIENTS INTAKE (VITAMIN A) Study name Mean difference and 95% confidence interval Faber et al. (2002) 2.30 (1.53, 3.07) Schipani (2002) 5.60 (-0.79, 11.99) Shmidt (1995) 0.57 (-9.19, 10.33) Smitasiri (1999) 3.95 (0.63, 7.27) Overall 2.42 (1.67, 3.16) 0 1 2 3 4 5 Mean difference between project and control group
IMPACT ON MICRONUTRIENTS INTAKE (VITAMIN A) Only four studies reported means and standard deviations of children of project and control areas The meta-analysis provides some support to the hypothesis that agricultural interventions improve intake of vitamin A among children under five But the number of studies available is too small to generate robust results as the summary results are very sensitive to the inclusion of one or two studies
IMPACT ON CHILDREN NUTRITIONAL STATUS Study Stunting (height-for-age) Underweight (weight-forage) Wasting (weight-forheight) Aiiga et al. (2002) n.s ** n.s Faber et al. (2002) n.s n.s n.s. Hoorweg et al. (2000) ** ** ** Makhotla et al. (2004) n.s. n.s. n.s. Low et al. (2007) n.s. ** ** Olney et al. (2009) n.s. n.s. n.s Schipani et al. (2002) n.s. n.s. n.s. Shmidt et al. (1995) n.s. n.s. n.s.
ANALYSIS OF RESULTS: CHILDREN NUTRITIONAL STATUS The studies reviewed report little or no impact of agricultural interventions on the nutritional status of children. The studies found a greater impact of the intervention on the prevalence of short term indicators of hunger (wasting and underweight) versus long-term indicators (stunting). These result confirms results of previous systematic reviews. However, unlike previous reviews, we attribute this result to the lack of statistical power of the studies reviewed rather than to the lack of efficacy of the interventions
POWER ANALYSIS Lack of significance can be the result of absence of impact as well of absence of statistical power and many studies were conducted over small samples of children We considered three hypothetical project effects: Large: the MDG target of halving malnutrition in 15 years Medium: the average impact found by impact evaluations of nutrition interventions Small: the yearly change in malnutrition rates that would bring the reduction targeted by the MDG
Study POWER ANALYSIS: RESULTS Sample size Sample s ratio (contro l/proje ct) Small (2%) Change in stunting prevalence Mediu m (10%) Large (30%) Change in underweight prevalence Small (2%) Mediu m (10%) Large (30%) Shmidt et al. (1995) 36 1 0.05 0.04 0.08 0.05 0.04 0.09 Schipani et al. (2002) 60 1 0.04 0.04 0.06 0.05 0.04 0.04 Aiiga et al. (2002) 66 1 0.03 0.06 0.35 0.03 0.03 0.12 Hoorweg et al. (2000) 102 1 0.03 0.03 0.14 0.03 0.03 0.13 Faber et al. (2002) 165 0.35 0.03 0.05 0.29 0.03 0.04 0.2 Olney et al. (2009) 445 0.44 0.03 0.14 0.8 0.03 0.09 0.54 Low et al. (2007) 741 0.33 0.04 0.49 0.99 0.03 0.16 0.89 Marsh (1998) 1,200 0.17 0.03 0.17 0.9 0.03 0.12 0.75 Makhotla et al. (2004) 2,688 0.25 0.05 0.52 1 0.04 0.25 0.98 Average power.04.15.51.04.09.42
POWER ANALYSIS: RESULTS The probability of detecting a small effect (-2%) is less than 5% The included studies would detect a medium impact (10%) in 15% of cases The included studies would detect a large impact (30%) in 50% of cases The absence of any reported statistically significant impact on nutrition should not be attributed to the inefficacy of these interventions, but to lack of power and small sample sizes
SAMPLE SIZES REQUIRED TO DETECT IMPACTS ON PREVALENCE RATES OF STUNTING AND UNDERWEIGHT Study Power Alpha Samples ratio (control /project ) Sample required to detect a change in stunting prevalence 2% 10% 30% Stunting.80.05 1 117,832 4,722 520 Underweight.80.05 1 182,752 7,240 778
VALIDITY ASSESSMENT We conducted an assessment of the methodology employed by the studies reviewed to judge internal and external validity Internal validity is the ability to establish causality. Indicators are: sample size and counterfactual analysis External validity is the ability to extrapolating results to other context. Indicators are: intermediate outcomes and impact heterogeneity
VALIDITY ASSESSMENT Low Low Medium Medium High HIgh 0 5 10 15 20 25 Counterfactual analysis 0 5 10 15 20 25 Power Low Low Medium Medium HIgh HIgh 0 5 10 15 20 25 Intermediate outcomes 0 5 10 15 20 25 Heterogeneity
VALIDITY ASSESSMENT: RESULTS Few studies performed a rigorous counterfactual analysis of the impact of the interventions Most studies neglected the analysis of the characteristics of programme participants Power calculations for determining sample size were rarely performed or presented Good conceptual framework and analysis of intermediate outcomes, but inappropriate indicators ignoring substitution effects in production and consumption Neglect of heterogeneity of impact and impossibility of extrapolating the results
RECOMMENDATIONS The very important question of whether agricultural interventions have a positive impact on nutritional status of children does not currently have an answer The following are needed: Evaluations of agricultural interventions and of their impact on nutrition More precise measurement of nutritional outcomes and use of appropriate sample size Investigation of determinants of programme participation
BIOFORTIFICATION Biofortification is the use of traditional crop breeding practices or modern biotechnology to produce micronutrient-dense staple crops to reduce micronutrients deficiencies (iron, zinc and vitamin A) Biofortification can reach poor populations in remote areas that rely on consumption of staple foods and have no access to fortified food biofortification is a cost-effective intervention ranked fifth among cost-effective interventions to tackle malnutrition and hunger by the panel of experts of the Copenhagen consensus of 2008
THEORY OF CHANGE OF BIOFORTIFICATION PLANT BREEDING: increased concentration of micronutrients in fortified staples FARMERS' RESPONSE: farmers adopt biofortification technology CONSUMERS' RESPONSE: consumers purchase and consume biofortified food BIOAVAILABILITY: micronutrients are succesfullyabsorbed by the body NUTRITIONAL IMPACT: bio fortified food improves nutritional status
STEPS OF THE THEORY OF CHANGE plants are developed that retain a large amount of micronutrients in their edible parts; farmers adopt biofortification technologies consumers (often the producers themselves) buy and consume staples from biofortified crops the micronutrients contents of biofortified foods are absorbed by the human body (bioavailability) finally consumption of biofortified food results in improved nutritional status.
EVIDENCE ON IMPACT OF BIOFORTIFICATION INTERVENTIONS PLANT BREEDING FARMERS' RESPONSE CONSUMERS' RESPONSE BIOAVAILABILITY NUTRITIONAL IMPACT Evidence: FAIR Evidence: VERY POOR Evidence: GOOD Evidence: GOOD Evidence: VERY POOR
CONCLUSIONS ON IMPACT OF BIOFORTIFICATION INTERVENTIONS There is insufficient evidence to assess the programme theory of biofortification interventions and only few of the causal links have been successfully explored Little is known about farmers acceptance of biofortified crops and no evidence is available on impact on yields and farm profits Only two studies report evidence on the nutritional impact of biofortification programmes and the results are positive Impact evaluations of biofortification programmes under different climatic and socioeconomic conditions are needed in order to assess their acceptance by farmers and their effectiveness