1 Why is ILCOR moving to GRADE? Associate Professor Peter Morley Director Medical Education Royal Melbourne Hospital University of Melbourne 10 min
2
3
4
5
6
7 Apart from international consensus Allows more clarity and consistency when move from evidence to recommendations
8 So is this really different to the C2010 approach?
9 Differences between processes 2010 ILCOR Identify PICO question Search for studies Clear inclusion exclusion criteria Identify studies for detailed review Identify key (critical) outcomes of relevance 2015 ILCOR (GRADE) Identify PICO question Search for studies Clear inclusion exclusion criteria Identify studies for detailed review Identify key (critical) outcomes of relevance
10 Differences between processes 2010 ILCOR Allocate study LOE: from LOE 1 to LOE 5 Allocate methodological quality for each study All studies (with outcomes) into grid according to direction for specific question 2015 ILCOR (GRADE) Allocate study type: RCTs or observational Assess risk of bias for each study All studies (with outcomes) into risk of bias table (including overall risk of bias for each study)
11 Assessment of methodological quality (risk of bias) of RCTs: similar 2010: ILCOR Quality RCTs (not clearly documented) Adequacy of randomization Allocation concealment Blinding Loss to follow up Intention to Treat (IT) analysis Groups treated equally? Groups similar at the start? 2015: GRADE Risk of bias RCTs (clearly documented) Adequacy of randomization Allocation concealment Blinding Loss to follow-up, Intention to Treat (IT) analysis Any other risks? Early stopping for positive benefit others
Allocation of quality Vienna 2012 vs downgrading for RCTs: similar 12 2010: Quality for individual RCT Good studies most/all of the relevant quality items Fair studies have some of the relevant quality items Poor studies have few of the relevant quality items (but sufficient value to include for further review). Excluded insufficient relevant quality items to be included 2015: Overall risk of bias for individual RCT Low (?Good) Unclear (?Fair) High (?Poor)
13 Published limitations (extras at 5)
14 Remember our evidence table
15
Overall risk of bias for study: Low, Moderate or High Low risk of bias = most or all key criteria listed are met, and any violations are not crucial. Moderate risk of bias if have a crucial limitation in one criterion or some limitations in multiple criteria, sufficient to lower the confidence in the estimate of effect. High risk of bias if have a crucial limitation in one or more criteria, sufficient to substantially lower the confidence in the estimate of effect. 16
17 Differences between processes 2010 ILCOR Narrative description of key information from all studies 2015 ILCOR (GRADE) Narrative description of key information from all studies Plus specific information regarding evidence for each key outcome Evidence profile table(s) Summary of findings table(s)
18 Major difference in approach for GRADE Key outcome measures are decided for each PICO question, and allocated a numerical rating Critical 9, 8, 7 Important 6, 5, 4 Limited importance 3, 2, 1 Quality of evidence is assessed (across studies) for each key outcome
19 How is the evidence for each outcome collated? Evidence Profile table(s) Summary of Findings table(s) Narrative description Consensus on Science statement
20 Evidence Profile table(s) PICO question Outcome to which evidence applies No of studies that report data for that outcome Study Design (RCT, Observational etc) Risk of bias (limitations: no serious, serious, very serious) Inconsistency (limitations: no serious, serious, very serious) Indirectness (limitations: no serious, serious, very serious) Imprecision (limitations: no serious, serious, very serious) Other including publication bias (Undetected, strongly suspected) Overall Quality of evidence for outcome High, Moderate, Low, Very low
Evidence profile table 21
Classification across all studies 22 for each outcome Risk of Bias/Inconsistency/Indirectness/Imprecision No serious limitations: Most information is from studies at low risk of bias. Do not downgrade Serious limitations: Most information is from studies at moderate risk of bias. Rate down one level Very serious limitations: Most information is from studies at high risk of bias. Rate down two levels
Classification across all studies for 23 each outcome Publication Bias Undetected Strongly suspected Quality of Evidence across all included studies for outcome High Moderate Low Very Low
24
25
Quality of evidence for outcome (across all studies) Four final categories High Moderate Low Very low Start with high for RCTs Start with low for observational Can be modified according to a number of factors 26
27 RCTs can be downgraded all or most of the studies had sufficient problems to make estimate unreliable or uncertain design and execution (risk of bias) consistency of results directness of comparisons precision publication bias
Overall quality of evidence: Vienna 2012 28 for each key outcome across studies
29 What about even lower levels of evidence? Case series (LOE 4) Manikin/Models/Animals (LOE 5)
30 Case series Start at low/very low quality Can be upgraded
31
32
33 Case series Start at low quality Can be upgraded BUT not if would be downgraded first!
34
Manikin/Models/Animals (LOE 5) (NB. GRADE not created for these) 35 Starting point depends on methodology RCTs at high quality Observational at low quality BUT Even RCTs can be downgraded Very serious indirectness = -2 Can be explained in comments
Summary of Findings table: columns PICO Key Outcomes Illustrative comparative risks (categorical or continuous with 95% CI) Control group Intervention group Relative effect (95% CI) No of Participants (studies) Overall quality of the evidence (GRADE) Comments/Footnotes 36
Summary of findings table 37
38 Next = CoSTR Consensus on Science statement and Treatment Recommendations
Differences between processes 39 2010 Consensus on Science statement (based on listing of levels of evidence for specific outcomes) Treatment recommendation: using behavioral wording, based largely on quality of evidence 2015 Consensus on Science statement (based on overall quality of evidence for critical outcomes ) Treatment recommendation: strong or weak (with behavioral correlates), based on evidence quality (high, moderate, low, very low)
40 So how will our Consensus on Science statement and Treatment Recommendations differ? COS: Quality level of evidence (GRADE) TR: Strength of Recommendation (GRADE)
GRADE: Strength of Recommendation Vienna 2012 41 Strong: the desirable effects of an intervention clearly outweigh the undesirable effects, or clearly do not. Weak: the trade-offs are less certain either because of low quality evidence or because evidence suggests that desirable and undesirable effects are closely balanced.
42 Strong Recommendation For patients most people in your situation would want the recommended course of action and only a small proportion would not; request discussion if the intervention is not offered For clinicians most patients should receive the recommended course of action
43 Weak Recommendation For patients most people in your situation would want the recommended course of action, but many would not For clinicians you should recognize that different choices will be appropriate for different patients and that you must help each patient to arrive at a management decision consistent with her or his values and preferences
44