Copyright GRADE ING THE QUALITY OF EVIDENCE AND STRENGTH OF RECOMMENDATIONS NANCY SANTESSO, RD, PHD

Transcription

1 GRADE ING THE QUALITY OF EVIDENCE AND STRENGTH OF RECOMMENDATIONS NANCY SANTESSO, RD, PHD ASSISTANT PROFESSOR DEPARTMENT OF CLINICAL EPIDEMIOLOGY AND BIOSTATISTICS, MCMASTER UNIVERSITY

2 Nancy Santesso, RD, PhD Registered Dietitian Assistant Professor, Department of Clinical Epidemiology and Biostatistics, McMaster University Deputy Director, Cochrane Canada Perform systematic reviews Facilitate guideline development for national and international organisations Research the communication of guidelines, specifically to patients and the public

3 Declarations of Interest Member of the GRADE Working Group ( based at the MacGRADE Centre at McMaster University Member of Cochrane Cochrane Grading Methods Group Cochrane Consumers and Communication Review Group Consultant and Trainer for GRADE and Cochrane (Income into a shared research account at McMaster University)

4 Agenda Overview of use of GRADE in guideline development How to GRADE the evidence Review of the 8 criteria to GRADE Hands-on work to GRADE evidence using an example Stretch How to move from evidence to recommendation Mock guideline panel using the GRADE Framework Wording recommendations

5 Overview of GRADE

6 Typical guideline process not different with GRADE

7 Schünemann et al. Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise. CMAJ Feb 18;186(3):E

8 > 90 Organizations

9

10 Why move to GRADE?

11 Why move to GRADE?

12 Why move to GRADE?

13 What are the key changes to this process? Clearer identification of critical outcomes Additional criteria to assess quality of evidence Clearer move from evidence to recommendation Wording changes

14

15 Treatment of condition with drug X 11 RCTs 1600 women Well conducted ~1000 women showed improvement Risk Ratio 1.45 (1.24 to 1.65) for improvement Consistent effects across studies certain uncertain

16 Treatment of condition with drug X 3 RCTs 100 women Well conducted ~40 women improved Risk Ratio 1.45 (0.40 to 2.45) for improvement Inconsistent effects across studies certain uncertain

17 Treatment of condition with drug X 3 RCTs 100 women (mix of similar conditions) NOT well conducted ~40 people improved Risk Ratio 1.45 (0.40 to 2.45) for improvement Inconsistent effects across studies 2 other protocols but authors did not send data certain uncertain

18

19 Assessment of evidence needed to be improved

20 Domains of GRADE 5 factors to evaluate the certainty of the evidence 1. Risk of bias 2. Inconsistency 3. Indirectness 4. Imprecision 5. Publication bias Plus additional factors for non-randomised studies Dose response, size of effect, residual confounding

21 GRADE domains Domains determine where on scale is certainty or quality Certain - HIGH Uncertain VERY LOW

22 Level of evidence High quality: We are very confident that the true effect lies close to that of the estimate of the effect Moderate quality: We are moderately confident in the effect estimate: The true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different Low quality: Our confidence in the effect estimate is limited: The true effect may be substantially different from the estimate of the effect Very low quality: We have very little confidence in the effect estimate: The true effect is likely to be substantially different from the estimate of effect

23 What would this look like in SOGC guidelines?

24 Currently Fetal Health Surveillance: Antepartum and Intrapartum Consensus Guideline

25 Fetal movement counting in pregnancies with risk factors Outcomes Effects of antenatal fetal assessment intervention No of Participants (studies) Mortality Apgar score Need for emergency delivery Extra alarms [Number] studies showed increased mortality in women with decreased fetal movements. [Number] studies showed Apgar <7 at 5 minutes in women with decreased fetal movements. [Number] studies showed increased need for emergency delivery in women with decreased fetal movements. [Number] case-control studies showed increased frequency of extra alarms in women with decreased fetal movements No (# study) No (# study)) No (# study) No (# study) Quality of the evidence High 1 Low 1,2 Low 1,2 Low 1,2 Clarifications 1 Examples only; 2 Evidence is based on observational studies.

26 With GRADE Fetal Health Surveillance: Antepartum and Intrapartum Consensus Guideline Quality of evidence? Due to? Quality of evidence? Due to?

27

28 GRADE is not just about evidence for benefits and harms

29 GRADE includes other criteria to judge the strength of recommendation Balance of benefits and harms Quality of the Evidence Values and preferences Recommendation Feasibility, acceptability, equity issues Resources

30 GRADE: Recommendations 1. Direction: For or Against an option 2. Strength: Strong or Weak/Conditional

31 With GRADE, consider more than the evidence for benefits and harms

32 Currently

33 With GRADE Strong or Conditional, Level of evidence

34 How to GRADE the evidence

35 How to GRADE the evidence Scenarios 1. Systematic review available quality of evidence already assessed using GRADE with/without tables

36

37 How to GRADE the evidence Scenarios 1. Systematic review available quality of evidence already assessed using GRADE with/without tables 2. Systematic review available but GRADE not used

38

39 Evidence in systematic review is a meta-analysis or narrative summary Use GRADE

40 Evidence is from a meta-analysis The RR is 1.38 (0.96, 2.00) for deaths for interventionist care compared to expectant care What does this mean?

41 The chance of deaths with interventionist care is 1.38 times the chance with expectant care (from 0.96 to 2.00)

42 Misinterpretation: No effect or the same! Instead, likely not enough power to find a significant result For now, the true value may be between 0.96 and 2.00

43 What is the quality/certainty of the evidence?

44 What to consider when making conclusions? Do I believe the results from these studies? Risk of bias Are the results consistent across studies? Inconsistency How do these results apply to my question? Indirectness Is this effect size precise? Imprecision Are these all of the studies? Publication bias

45 What to consider when making conclusions Do I believe the results from these studies? Risk of bias How do these results apply? Indirectness Are the results consistent across studies? Inconsistency Are these all of the studies? Publication bias Is this effect size precise? Imprecision

46 Is the risk of bias Not serious Serious Very Serious that my confidence in the result is reduced?

47 Cochrane risk of bias tool for randomised studies Cochrane risk of bias tool for non-randomised studies ACROBAT-NRSI Risk of bias tool for diagnostic accuracy studies QUADAS Risk of bias for prognostic studies QUIPS tool Cochrane Prognosis Methods Group

48 Risk of bias for randomised controlled studies Sequence generation Allocation concealment Blinding Incomplete outcome assessment Outcome reporting Other Typically 2 people should assess each study 1 person? Put in a figure or summarise in text overall assessment of risk of bias

49 Overall Risk of Bias What is the overall risk of bias of these studies? Does the risk of bias reduce your certainty in the effect? Bigger studies more weight

50 Is the risk of bias Not serious Serious Very Serious DISCUSS that my confidence in the result is reduced?

51 Do I have no concern, serious concern, very serious concern about the risk of bias?

52 True for a narrative synthesis

53 What to consider when making conclusions Do I believe the results from these studies? Risk of bias How do these results apply? Indirectness Are the results consistent across studies? Inconsistency Are these all of the studies? Publication bias Is this effect size precise? Imprecision

54 Indirectness Consider the extent to which you are uncertain about the applicability of the evidence to your relevant question Consider PICO Population Intervention and comparison Outcome Not about whether evidence is generalisable to other populations etc.

55 Indirectness Population:??? Interventions and comparisons:??? Outcomes: stillbirths:??? Tables of included studies

56 Is the indirectness Not serious Serious Very Serious DISCUSS that my confidence in the result is reduced?

57 What to consider when making conclusions Do I believe the results from these studies? Risk of bias How do these results apply? Indirectness Are the results consistent across studies? Inconsistency Are these all of the studies? Publication bias Is this effect size precise? Imprecision

58 Interventionist versus expectant care Was the effect on stillbirths very different in the studies? Did the authors of the review indicate heterogeneity? Measure heterogeneity? Did they indicate conflicting results in different studies?

59 Visual inspection of heterogeneity Forest plot A Forest plot B

60 Statistical heterogeneity The chi-squared (c2) test tests the null hypothesis of homogeneity Significant p value = heterogeneity Yes or no answer The I 2 statistic I 2 Level of heterogeneity >30 to 40% Low or unimportant 30 to 60% Moderate 50-90% Substantial 75 to 100% High

61 The I2 statistic

62 The I2 statistic

63 What about with no meta-analysis?

64 Is the inconsistency Not serious Serious Very Serious DISCUSS that my confidence in the result is reduced?

65 What to consider when making conclusions Do I believe the results from these studies? Risk of bias How do these results apply? Indirectness Are the results consistent across studies? Inconsistency Is this effect size precise? Imprecision Are these all of the studies? Publication bias

66 How many people and how many events?

67 Imprecision Consider Sample sizes and number of events assess according to effect size, control event rates, Optimal information size (OIS) Width of confidence intervals Wide confidence intervals indicate uncertainty about INCLUDES the NO effect EFFECT!! Includes null effect and appreciable benefit or harm (rule of thumb: RR<0.75 or >1.25) NOT WHETHER IT

68 Optimal information size (OIS) if the total number of patients included in a systematic review is less than the number of patients generated by a conventional sample size calculation for a single adequately powered trial, consider rating down for imprecision

69 Rules of thumb Dichotomous outcomes events Continuous outcomes 400 people providing outcome measures

70 Imprecision Consider Sample sizes and number of events assess according to effect size, control event rates, Optimal information size (OIS) Width of confidence intervals How wide is too wide? Wide confidence intervals indicate uncertainty about the effect Includes null effect and appreciable benefit or harm (rule of thumb: RR<0.75 or >1.25)

71

72

73 What about with no meta-analysis?

74 Is the imprecision Not serious Serious Very Serious DISCUSS that my confidence in the result is reduced?

75 What to consider when making conclusions Do I believe the results from these studies? Risk of bias How do these results apply? Indirectness Are the results consistent across studies? Inconsistency Is this effect size precise? Imprecision Are these all of the studies? Publication bias

76 Publication biases dissemination of research findings is influenced by the nature and direction of results positive results more likely to be published large studies likely to be published, so evidence from all small studies most likely to be affected

77 No statistics possible? Assess for publication biases by. a comprehensive search of multiple sources? many small studies all saying positive results?

78 Is publication bias undetected Strongly suspected DISCUSS that my confidence in the result is reduced?

79 Summary: deaths Risk of bias? Yes, overall risk of bias indirectness? No (some) inconsistency? No imprecision? Yes publication bias? Undetected Low quality evidence due to risk of bias and imprecise results Interventionist care may increase risk of deaths by 38% (95 CI, 0.96 to 2.00) High moderate

80 True for a narrative synthesis

81 GRADE criteria to decide on level of evidence 5 criteria for randomised controlled trials and nonrandomised studies If serious concern with any criteria, level of evidence goes down 3 criteria usually for non-randomised studies If any criteria met, level of evidence goes up

82 How to GRADE the evidence Scenarios 1. Systematic review available quality of evidence already assessed using GRADE with/without tables 2. Systematic review available but GRADE not used 3. Systematic review PLUS additional new study??? DISCUSS

83 How to GRADE the evidence Scenarios 1. Systematic review available quality of evidence already assessed using GRADE with/without tables 2. Systematic review available but GRADE not used 3. Systematic review PLUS additional new study??? 4. Systematic review of non-randomised studies?

84 Non-randomised studies Use all the same criteria for randomised controlled trials But additional criteria to upgrade the evidence use when you have not already downgraded!!

85 Upgrading: Large effects Large magnitude Very large upgrade two levels (RR > 5 or RR < 0.2) Large upgrade 1 level (RR > 2 or RR < 0.5) everyone used to do badly but after treatment almost everyone does well Example: parachutes to prevent death when jumping from airplanes, oral contraceptives

86 Non-randomised studies

87 Upgrading: Dose response relationship A systematic review of observational studies investigating the effect of cyclooxygenase-2 (COX-2) inhibitors on cardiovascular events, found a RR with rofecoxib of 1.33 (95% CI, ) with doses less than 25 mg/d and a RR of 2.19 (95% CI, ) with doses more than 25 mg/d.

88 Upgrading: residual confounding Effects of plausible residual confounding may be working to reduce the demonstrated effect or increase the effect if no effect was observed You found no effect but the confounding you can think of would have increased it You found an effect even though all confounding should have reduced the effect

89 Upgrading: residual confounding Example: Vaccination and autism Large observational studies have failed to demonstrate an association even though clinicians would be more alert to suspecting and reporting autism and vaccination status

90 Strength of recommendations 1. Direction: For or Against an option 2. Strength: Strong or Weak/Conditional

91 Should not be tied to the study design It s possible to have RCTs that are fair evidence or cohorts that are good evidence

92 What does strong and weak mean?

93 Balancing all factors: For Against NOT JUST BENEFITS AND HARMS ALL CRITERIA CONSIDERED GREATER THE TIPPING THE STRONGER THE RECOMMENDATION

94 Cryotherapy to LEEP exercise Use the EtD to make a recommendation for Cryotherapy versus LEEP DISCUSS

95 What were your responses?

96

97 Format in guideline We suggest LEEP over cryotherapy to treat women with confirmed CIN II. Conditional recommendation, Low quality evidence

98

99 Conditional recommendation interpretation

100 Currently

101 With GRADE Strong or Conditional, Level of evidence???????

102 Best Practice Statements (GRADE guidance not yet published) Is a statement necessary? Is there large net consequences? Is there a clear rationale connecting indirect evidence?

103 With GRADE Good Practice Statement

104 Key to evidence assessment and recommendations: GRADE Certainty of evidence HIGH: We are very confident that the true effect lies close to that of the estimate of the effect MODERATE: We are moderately confident in the effect estimate: The true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different. LOW: Our confidence in the effect estimate is limited: The true effect may be substantially different from the estimate of the effect VERY LOW: We have very little confidence in the effect estimate: The true effect is likely to be substantially different from the estimate of effect Strength of recommendation STRONG FOR: Most individuals should receive the intervention. CONDITIONAL The majority of individuals in this FOR: situation would want the suggested course of action, but many would not. Clinicians should recognize that different choices will be appropriate for each individual. STRONG AGAINST: Most individuals should not receive the intervention. CONDITIONAL The majority of individuals in this AGAINST: situation would want the suggested course of action, but many would not. Clinicians should recognize that different choices will be appropriate for each individual.

105 More information Nancy Santesso