EVIDENCE SYNTHESIS FOR GLOBAL HEALTH: ANIMAL STUDIES UNDER THE LENS Paul Garner On behalf of the Cochrane infectious Diseases Group Editors And the Effective Health Care Research Programme Consortium
Competing interests Funded by the UK Aid Programme to increase the number of decisions based on reliable evidence in health for the poor in developing countries
previously UK Aid policies were well meaning, but based on expert opinion Example from the British Colonial Service for the Punjab
1993
Systematic reviews 1. Responses from our stakeholders over 20 yr. 2. Findings from a ASR of a new TB vaccine 3. Personal reflections on the agenda
Funders Synthesis Researchers Policy maker
Stakeholders Our funders
Research governance framework for health and social care (2001)
Global policy makers guidelines panels/advocates Category 1. Those that are happy
Amodiaquine (Lancet 1996) 50 studies Better than chloroquine WH O reintroduced in Africa 1998 Impregnated mosquito nets for malaria (Cochrane 1997) Low osmolarity ORS (BMJ/Cochrane 2001) 14 studies Reduced child mortality Helped underpin investment 1998-10 studies Better Changed formula 2003 Artemisinin combination Rx (Lancet 2004) 30 studies 70% increase in cure ACTs first line 2007 Artesunate severe malaria (Cochrane 2007) 8 studies 40% fewer deaths WHO rec 2008
Global policy makers guidelines panels/advocates Category 2. Those that are cross
Direct observation for TB (BMJ/Cochrane 1997) Deworming for schoolchildren (Cochrane 1996/2011) Primaquine single dose to block transmission (Cochrane 2012/4) 5 studies No difference Controversy Modified early 2000 50 studies Exaggerated claims Investment continued until recently 15 studies Inadequate data Policy remains; research continues
Belief disconfirmation bias Belief-confirming evidence: Belief-disconfirming evidence: taken at face value intense critical evaluation
Belief disconfirmation bias Standard responses: Original research methods is flawed Synthesis cannot be used here Data incomplete-studies missed You have applied the methods wrongly Synthesis authors are uninformed/ incompetent/trouble makers
Cochrane review Does routine deworming improve children s health, performance at school and contribute to economic development?
Deworming review: accountability in question One trial of 22,000 children in 60 clusters showed increased weight gain because it did not adjust for clustering A trial from Vietnam of over 300 children that looked at school performance has never been published One trial of 2 million children remained unpublished 6 years after recruitment ended Re-analysis of Miguel and Kremer study (Nov 2014) the claimed externality effects vanished & no effect on nutrition Plus one more
Increasingly Academic competing interests causes substantive biases and distortion Academic behaviour often challenges the principles of scientific integrity
Systematic review of prime boost TB vaccine Rufaro Kashangura, Emily S. Sena, Taryn Young, Paul Garner
Protocol driven study Comparison: vaccine + BCG vs. BCG Animals: any Design: TB challenge given Outcomes: Mortality/euthanasia, pathology in lungs, and bacterial loads in lung
Methods Comprehensive search Double application of inclusion criteria Study quality assessed Cumulative meta-analysis
quality
Criteria Were sample size and power calculations given? Was the number of animals clear? Was competing interest declared? Was there randomisation? Was there baseline comparability? Was the outcome assessor blind? Humane endpoint Pathology Bacteriol Williams Williams Goonetille Romano Tchilian Verrick Vorderme Sharpe 2005 (a) 2005 (b) ke 2003 2006 2008 2009 ier 2009 2010 No No No No No No No No Yes Yes No Yes No Yes Yes Yes No COI statement Not reported All 250-300g female pigs Not reported Not reported but objective Not reported No COI statement Not reported Same species pigs Not reported No COI statement Not reported All 8-10wk old female mice. Not reported No COI statement Not reported All 8-10 weeks old mice. Not reported No COI statement Not reported All 6-8wk old female mice. Not reported Contains COI with conflict Yes Same age weight, male macaques. Not reported Contains COI with conflict Not Reported All six month old cattle. No COI statemen t Not reported All 4 yr old macaque s. Not reported Yes Yes Only lesion volume blinded Not reported Not reported Not reported
Outcomes
Bacterial load
Pathology summary Authors conclusion Our interpretation William 2005 Guinea pigs Not able to discriminate No data provided (b) 14 William 2005 (b) 14 Guinea pigs Not significantly better than BCG No data provided Verrick 2009 c 18 Macaques BCG/MVA85A less severe disease and fewer lesions than BCG only Vordermeier 2009 1 Calves Sharpe 2010 12 Macaques significantly reduced after MVA85A vaccination compared to control f No statistical difference between MVA85A vaccinated and BCG vaccinated animals for lesions in lung, lung pathology and gross pathology scoring Scores in both BCG alone and MVA85A boost much lower than controls No direct statistical comparison between MVA85A and BCG. No statistical difference but more lesions in BCG and MVA85A groups. Other parameters show no trends
Mortality
Implications for strategy We as a group have a duty to develop the science and reviews in this area It has to be independent of the researchers The so called scientists will be sceptical (watch for belief disconfirmation bias ) We have to get FUNDERS and the PUBLIC onside