Assessing quality of evidence The GRADE approach

Transcription

1 Assessing quality of evidence The GRADE approach Elie A. Akl, MD, MPH, PhD Department of Internal Medicine, AUB Department of Clinical Epidemiology & BiostaAsAcs McMaster University, Canada

2 Disclosure Member of the GRADE working group No financial conflict of interest

3 This session First of two sessions concerning the use of GRADE methodology in developing recommendaaons for pracace guidelines: How to grade the quality of evidence How to move from evidence to recommendaaons

4 Goals and ObjecAves 1. Define the quality of evidence 2. Discuss the factors that affect the raang of the quality of evidence reported in a systemaac review

5 Grades of Recommendation Assessment, Development and Evaluation GRADE Working Group CMAJ 2003, BMJ 2004, BMC 2004, BMC 2005, AJRCCM 2006, Chest 2006, BMJ 2008, Lancet ID 2007, PLOS Medicine BMJ series 2011 JCE series

6 70+ OrganizaAons

7 Goals and ObjecAves 1. Define the quality of evidence 2. Discuss the factors that affect the raang of the quality of evidence reported in a systemaac review

8 What is quality of evidence? Extent to which we have confidence in the estimate of effect The quality is assessed by outcome (and not by study)

9 Example A new medicaaon X is reported to reduce mortality from heart disease by 30%

10

11 GRADE levels of Evidence The confidence in effect estimate is a continuum (from 0% to 100%) For practical reasons, it is categorized into 4 levles: High: Moderate: Low: Very low:

12 GRADE levels of Evidence High: considerable confidence in estimate of effect Moderate: further research likely to have impact on confidence in estimate, may change estimate Low: further research is very likely to impact on confidence, likely to change the estimate Very low: any estimate of effect is very uncertain

13 Goals and ObjecAves 1. Define the quality of evidence 2. Discuss the factors that affect the of the quality of evidence reported in a systema@c review

14 Quality of evidence What are the factors that determine the raang of the quality of evidence reported in a systemaac review?

15 Determinants of quality

16 Factors that lower quality 1. Risk of bias ú Inappropriate sequence generaaon ú Lack of allocaaon concealment ú Inadequate blinding ú IntenAon to treat principle violaaon ú Loss to follow- up ú Early stopping for benefit

17 RandomizaAon Why is randomizaaon important? How do you ensure an adequate randomizaaon process?

18 RandomizaAon Adequate sequence generaaon Table of random number Computer- generated list of random number Roll of a dice; flip of a coin AllocaAon concealment Central randomizaaon office Pre- numbered or coded containers SequenAally numbered, sealed, opaque envelopes

19 Blinding Who do you blind? Why do you blind them?

20 Blinding PaAents Providers Data collectors Outcome adjudicators Data analysts Data and safety monitoring board (DSMB)

21 Analysis of missing outcome data Can paaents with missing outcome data (lost to follow- up) introduce bias? How do you assess whether the number of paracipants with missing outcome data introduces bias?

22 Analysis of missing outcome data missing data 100 à ASA 10 events 1 missing data 100 à Placebo 20 events missing data 100 à ASA 10 events 20 missing data 100 à Placebo 20 events

23 Analysis of missing outcome data missing data 100 à ASA 10 events 1 missing data 100 à Placebo 20 events missing data 100 à ASA 1 event 1 missing data 100 à Placebo 2 event

24 Analysis of non- adherence Can handling of non- adherence paaents in the analysis introduce bias? How do you avoid bias?

25 Analysis of non- adherence to treat analysis: analyze all those randomized in the arm to which they were randomized (irrespecave of adherence) As treated: analyze all those randomized according to what they actually received Per protocol analysis: analyze only those who were adherent

26 Analyzing non- adherents 20 non adherent 10 events à ASA 100 à placebo 80 adherent 100 adherent 10 events 20 events RR = (Risk on ASA) / (Risk on Placebo) ITT: RR= Per protocol: RR = As treated RR =

27 Analyzing non- adherents 20 non adherent 10 events à ASA 100 à placebo 80 adherent 100 adherent 10 events 20 events RR = (Risk on ASA) / (Risk on Placebo) ITT: RR= (20/100) / (20/100) = 1 Per protocol: RR = As treated RR =

28 Analyzing non- adherents 20 non adherent 10 events à ASA 100 à placebo 80 adherent 100 adherent 10 events 20 events RR = (Risk on ASA) / (Risk on Placebo) ITT: RR= (20/100) / (20/100) = 1 Per protocol: RR = (10/80) / (20/100) = 0.63 As treated RR =

29 Analyzing non- adherents 20 non adherent 10 events à ASA 100 à placebo 80 adherent 100 adherent 10 events 20 events RR = (Risk on ASA) / (Risk on Placebo) ITT: RR= (20/100) / (20/100) = 1 Per protocol: RR = (10/80) / (20/100) = 0.63 As treated RR = (10/80) / (30/120) = 0.5

30 Stopping study early for benefit Is there a risk of bias associated with stopping a study early?

31 Studies stopped early because of benefit

32 Factors that lower quality 32

33 Factors that lower quality 2. Inconsistency ú Assess for inconsistency (Heterogeneity) ú If inconsistency à look for explanaaon paaents, intervenaon, outcome, methods ú If unexplained inconsistency à downgrade quality

34 Factors that lower quality 3. Indirectness of Evidence ú Differences in populaaons/paaents ú Differences in intervenaons ú Differences in outcomes

35 Factors that lower quality 4. Imprecision ú If 95% CI includes both negligible effect and appreciable benefit or appreciable harm à rate down

36 1.0% 0

37 1.0% 0

38 1.0% 0

39 Factors that lower quality 5. Publication bias Faster and multiple publication of positive trials Fewer and slower publication of negative trials

40 0 Funnel plot Standard Error 1 2 Asymmetrical: PublicaAon bias? Odds raao 10 Egger M, Cochrane Colloquium Lyon

41 ISIS-4 Lancet 1995 Meta- analysis contradicted by mega- trials Publication bias Meta-analysis Yusuf S.Circulation 1993 Egger M, Smith DS. BMJ 1995;310:

42 Factors that raise quality 1. Large magnitude of effect 2. Dose response relation 3. All plausible confounding may be working to reduce the demonstrated effect 42

43 Factors that raise quality 1. Large magnitude of effect large (RRR 50%) can raise by one level very large (RRR 80%) can raise by two levels common criteria: everyone used to do badly and now almost everyone does well.

44 BMJ 2003 BMJ, 2003

45 RelaAve risk reducaon:.> 99.9 % (1/100,000) U.S. Parachute AssociaAon reported 821 injuries and 18 deaths out of 2.2 million jumps in 2007 BMJ 2003

46 Factors that raise quality 2. Dose response relaaon ú (higher INR increased bleeding) ú childhood lymphoblasac leukemia; risk for CNS malignancies 15 years aqer cranial irradiaaon ú no radiaaon: 1% (95% CI 0% to 2.1%) ú 12 Gy: 1.6% (95% CI 0% to 3.4%) ú 18 Gy: 3.3% (95% CI 0.9% to 5.6%)

47 Determinants of quality

48 Questions? 48

49 Exercise Grade the quality of evidence for a systemaac review on the use o parenteral anacoagulaaon in paaents with cancer