EVIDENCE SYNTHESIS FOR GLOBAL HEALTH: ANIMAL STUDIES UNDER THE LENS

Transcription

1 EVIDENCE SYNTHESIS FOR GLOBAL HEALTH: ANIMAL STUDIES UNDER THE LENS Paul Garner On behalf of the Cochrane infectious Diseases Group Editors And the Effective Health Care Research Programme Consortium

2 Competing interests Funded by the UK Aid Programme to increase the number of decisions based on reliable evidence in health for the poor in developing countries

3

4

5 previously UK Aid policies were well meaning, but based on expert opinion Example from the British Colonial Service for the Punjab

6

7

8

9

10 1993

11 Systematic reviews 1. Responses from our stakeholders over 20 yr. 2. Findings from a ASR of a new TB vaccine 3. Personal reflections on the agenda

12 Funders Synthesis Researchers Policy maker

13 Stakeholders Our funders

14 Research governance framework for health and social care (2001)

15

16 Global policy makers guidelines panels/advocates Category 1. Those that are happy

17 Amodiaquine (Lancet 1996) 50 studies Better than chloroquine WH O reintroduced in Africa 1998 Impregnated mosquito nets for malaria (Cochrane 1997) Low osmolarity ORS (BMJ/Cochrane 2001) 14 studies Reduced child mortality Helped underpin investment studies Better Changed formula 2003 Artemisinin combination Rx (Lancet 2004) 30 studies 70% increase in cure ACTs first line 2007 Artesunate severe malaria (Cochrane 2007) 8 studies 40% fewer deaths WHO rec 2008

18 Global policy makers guidelines panels/advocates Category 2. Those that are cross

19 Direct observation for TB (BMJ/Cochrane 1997) Deworming for schoolchildren (Cochrane 1996/2011) Primaquine single dose to block transmission (Cochrane 2012/4) 5 studies No difference Controversy Modified early studies Exaggerated claims Investment continued until recently 15 studies Inadequate data Policy remains; research continues

20 Belief disconfirmation bias Belief-confirming evidence: Belief-disconfirming evidence: taken at face value intense critical evaluation

21 Belief disconfirmation bias Standard responses: Original research methods is flawed Synthesis cannot be used here Data incomplete-studies missed You have applied the methods wrongly Synthesis authors are uninformed/ incompetent/trouble makers

22 Cochrane review Does routine deworming improve children s health, performance at school and contribute to economic development?

23

24

25

26

27

28 Deworming review: accountability in question One trial of 22,000 children in 60 clusters showed increased weight gain because it did not adjust for clustering A trial from Vietnam of over 300 children that looked at school performance has never been published One trial of 2 million children remained unpublished 6 years after recruitment ended Re-analysis of Miguel and Kremer study (Nov 2014) the claimed externality effects vanished & no effect on nutrition Plus one more

29 Increasingly Academic competing interests causes substantive biases and distortion Academic behaviour often challenges the principles of scientific integrity

30

31 Systematic review of prime boost TB vaccine Rufaro Kashangura, Emily S. Sena, Taryn Young, Paul Garner

32 Protocol driven study Comparison: vaccine + BCG vs. BCG Animals: any Design: TB challenge given Outcomes: Mortality/euthanasia, pathology in lungs, and bacterial loads in lung

33 Methods Comprehensive search Double application of inclusion criteria Study quality assessed Cumulative meta-analysis

34

35 quality

36 Criteria Were sample size and power calculations given? Was the number of animals clear? Was competing interest declared? Was there randomisation? Was there baseline comparability? Was the outcome assessor blind? Humane endpoint Pathology Bacteriol Williams Williams Goonetille Romano Tchilian Verrick Vorderme Sharpe 2005 (a) 2005 (b) ke ier No No No No No No No No Yes Yes No Yes No Yes Yes Yes No COI statement Not reported All g female pigs Not reported Not reported but objective Not reported No COI statement Not reported Same species pigs Not reported No COI statement Not reported All 8-10wk old female mice. Not reported No COI statement Not reported All 8-10 weeks old mice. Not reported No COI statement Not reported All 6-8wk old female mice. Not reported Contains COI with conflict Yes Same age weight, male macaques. Not reported Contains COI with conflict Not Reported All six month old cattle. No COI statemen t Not reported All 4 yr old macaque s. Not reported Yes Yes Only lesion volume blinded Not reported Not reported Not reported

37 Outcomes

38 Bacterial load

39 Pathology summary Authors conclusion Our interpretation William 2005 Guinea pigs Not able to discriminate No data provided (b) 14 William 2005 (b) 14 Guinea pigs Not significantly better than BCG No data provided Verrick 2009 c 18 Macaques BCG/MVA85A less severe disease and fewer lesions than BCG only Vordermeier Calves Sharpe Macaques significantly reduced after MVA85A vaccination compared to control f No statistical difference between MVA85A vaccinated and BCG vaccinated animals for lesions in lung, lung pathology and gross pathology scoring Scores in both BCG alone and MVA85A boost much lower than controls No direct statistical comparison between MVA85A and BCG. No statistical difference but more lesions in BCG and MVA85A groups. Other parameters show no trends

40 Mortality

41

42 Implications for strategy We as a group have a duty to develop the science and reviews in this area It has to be independent of the researchers The so called scientists will be sceptical (watch for belief disconfirmation bias ) We have to get FUNDERS and the PUBLIC onside