Introduzione al metodo GRADE Atto Billio MD, MSc EBHC Ematologia e TMO-Bolzano Gruppo linee guida SIE
Critique of EBM De-emphasizes patient values Doesn t account for individual variation Devalues clinical judgment Leads to therapeutic nihilism
Definition of EBM The integration of best research evidence with clinical expertise and patient values. Sackett DL et al; Churchill Livingstone, 2000
Evidence-based clinical decisions Clinical state and circumstances Patient values and preferences Expertise Research evidence Equal for all Haynes et al. 2002
Evidence-based clinical decisions Clinical state and circumstances Patient values and preferences Expertise costs Research evidence Equal for all Haynes et al. 2002
Definition of Evidence Based Health Practice (EBHP) EBPH is the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of communities and populations in the domain of health protection, disease prevention, health maintenance and improvement. Jenicek M. J Epidemiol 1997;7:187-97 Jenicek (1997)
Hierarchy of evidence STUDY DESIGN Randomized Controlled Trials Cohort Studies and Case Control Studies Case Reports and Case Series, Non-systematic observations Expert Opinion BIAS
Before GRADE Level of evidence I II III IV V Source of evidence SR, RCTs Cohort studies Case-control studies Case series Expert opinion Grades of recomend. A B C D Oxford Centre of Evidence Based Medicine 8
Limitations of older systems & approaches confuse quality of evidence with strength of recommendations criteria not comprehensive or transparent focus on single outcomes
80+ Organizations 2005 2006 2007 2008 2009 2010 2011 10
What are we grading? two components quality of body of evidence confidence in estimate of effect high, moderate, low, very low strength of recommendation strong and weak
STRUCTURED QUESTION PICO Pazienti e caratteristiche della malattia Intervento (i) Comparatore Outcomes
Hierarchy of outcomes according to their patient-importance Mortality 9 Importance of endpoints Myocardial infarction 8 Fractures 7 Pain due to soft tissue Calcification / function 6 5 4 Critical for decision making Important, but not critical for decision making Flatulence 3 2 1 Of low patientimportance
Quality of evidence across studies Outcome #1 Outcome #2 Outcome #3 Quality: High Quality: Moderate Quality: Low III V II I B
Determinants of quality RCTs start high observational studies start low What lowers quality of evidence? 5 factors: Methodological limitations Inconsistency of results Indirectness of evidence Imprecision of results Publication bias
Methodological limitations Inconsistency of results Indirectness of evidence Imprecision of results Publication bias Assessment of detailed design and execution (risk of bias) For RCTs: Lack of allocation concealment No true intention to treat principle Inadequate blinding Loss to follow-up Early stopping for benefit
Cochrane Risk of bias graph in RevMan 5 17
Methodologica l limitations Inconsistency of results Indirectness of evidence Imprecision of results Publication bias Look for explanation for inconsistency patients, intervention, comparator, outcome, methods Judgment variation in size of effect overlap in confidence intervals statistical significance of heterogeneity I 2
Methodologica l limitations Inconsistency of results Indirectness of evidence Imprecision of results Publication bias Indirect comparisons Interested in head-to-head comparison Drug A versus drug B (not A vs C and B vs C) Differences in patients (early cirrhosis vs end-stage cirrhosis) interventions (CRC screening: flex. sig. vs colonoscopy) comparator (e.g., differences in dose) outcomes (non-steroidal safety: ulcer on endoscopy vs symptomatic ulcer complications)
Methodological limitations Inconsistency of results Indirectness of evidence Imprecisio n of results Small sample size small number of events wide confidence intervals uncertainty about magnitude of effect Publication bias
Methodologica l limitations Inconsistency of results Indirectness of evidence Imprecision of results Publication bias Reporting of studies publication bias number of small studies industry sponsored
GRADE evidence profile
Summary of Findings Table Alendronate 10 mg compared to Control for the secondary prevention of osteoporotic fractures in postmenopausal women Patient or population: postmenopausal women with previous osteoporotic fractures Intervention: Alendronate 10 mg Comparison: Control Outcomes BENEFITS Vertebral Fractures radiographs (follow-up: mean 2 years) Hip Fractures clinical presentation + radiograph (follow-up: mean 2 years) Illustrative comparative risks* (95% CI) Assumed risk Corresponding risk Control Alendronate 10 mg Relative effect (95% CI) Low risk population 1 RR 0.55 (0.43 to 53 per 1000 29 per 1000 0.69) (23 to 37) High risk population 112 per 1000 62 per 1000 (48 to 77) Low risk population RR 0.47 (0.26 to 19 per 1000 9 per 1000 0.85) (5 to 16) High risk population 87 per 1000 41 per 1000 (23 to 74) No of Participants (studies) 2785 (4) 5376 (5) Quality of the evidence (GRADE) ++++ high 2 ++++ high 2 Comments NNTB Low risk : 42 (95% CI: 34-61) High risk: 20 (95% CI: 16-29) NNTB Low risk : 100 (95% CI: 72-351) High risk: 22 (95% CI: 16-77)
GRADE Quality of Evidence In the context of a systematic review The quality of evidence reflects the extent to which we are confident that an estimate of effect is correct. In the context of making recommendations The quality of evidence reflects the extent to which our confidence in an estimate of the effect is adequate to support a particular recommendation.
GRADE: Quality of evidence The extent to which our confidence in an estimate of the treatment effect is adequate to support a particular recommendation. GRADE defines 4 categories of quality: High Moderate Low Very low 25
Conceptualizing quality HIGH Further research is very unlikely to change our confidence in the estimate of effect ++++ MODERATE Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate +++- LOW Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate ++-- VERY LOW Any estimate of effect is very uncertain +---
GRADE - From evidence to decisions 27
Quality of evidence & strength of recommendation GRADE separates quality of evidence from strength of recommendation Linked but no automatism Other factors beyond the quality of evidence influence our confidence that adherence to a recommendation causes more benefit than harm
GRADE: Factors influencing decisions and recommendations Quality of Evidence Balance of desirable and undesirable consequences Values and preferences Cost 29
Strength of recommendations Desirable effects health benefits less burden savings Undesirable effects harms more burden costs
Strength of recommendation STRONG WEAK
Strength of recommendations degree of confidence that desirable effects of adhering to recommendation outweigh undesirable effects. strong recommendation benefits clearly outweigh risks/hassle/cost risk/hassle/cost clearly outweighs benefit what can downgrade strength? low quality evidence close balance between up and downsides
Implications of a strong recommendation Patients: Most people in your situation would want the recommended course of action and only a small proportion would not Clinicians: Most patients should receive the recommended course of action Policy makers: The recommendation can be adapted as a policy in most situations
Implications of a weak/conditional recommendation Patients: The majority of people in your situation would want the recommended course of action, but many would not Clinicians: Be prepared to help patients to make a decision that is consistent with their own values Policy makers: There is a need for substantial debate and involvement of stakeholders
Judgements about the strength of a recommendation No precise threshold for going from a strong to a weak recommendation The presence of important concerns about one or more of these factors make a weak recommendation more likely Panels should consider all of these factors and make the reasons for their judgements explicit. Recommendations should specify the perspective that is taken (e.g. individual patient, health system) and which outcomes were considered (including which, if any costs).
Risk/Benefit tradeoff aspirin after myocardial infarction 25% reduction in relative risk side effects minimal, cost minimal benefit obviously much greater than risk/cost warfarin in low risk atrial fibrillation warfarin reduces stroke vs ASA by 50% but if risk only 1% per year, ARR 0.5% increased bleeds by 1% per year
Are values important? Should resources be considered? 37
Value and preference statements underlying values and preferences always present sometimes crucial important to make explicit
Oseltamivir Avian Influenza Factors that can weaken the strength of a recommendation. Example: treatment of H5N1 patients with oseltamivir Lower quality evidence Uncertainty about the balance of benefits versus harms and burdens Uncertainty or differences in values Uncertainty about whether the net benefits are worth the costs Decision Yes No Yes No Yes No Yes No Explanation The quality of evidence is very low The benefits are uncertain because several important or critical outcomes where not measured. However, the potential benefit is very large despite potentially small relative risk reductions. All patients and care providers would accept treatment for H5N1 disease For treatment of sporadic patients the price is not high ($45). Frequent yes answers will increase the likelihood of a weak recommendation
Example: Oseltamivir for Avian Flu Recommendation: In patients with confirmed or strongly suspected infection with avian influenza A (H5N1) virus, clinicians should administer oseltamivir treatment as soon as possible (strong recommendation, very low quality evidence). Values and Preferences Remarks: This recommendation places a high value on the prevention of death in an illness with a high case fatality. It places relatively low values on adverse reactions, the development of resistance and costs of treatment. Schunemann et al. The Lancet ID, 2007
Other explanations Remarks: Despite the lack of controlled treatment data for H5N1, this is a strong recommendation, in part, because there is a lack of known effective alternative pharmacological interventions at this time. The panel voted on whether this recommendation should be strong or weak and there was one abstention and one dissenting vote.
GRADE GRADE Prioritize problems, establish panel Systematic review Searches, selection of studies, data collection and analysis Assess the relative importance of outcomes Prepare evidence profile: Quality of evidence for each outcome and summary of findings Assess overall quality of evidence Decide direction and strength of recommendation Draft guideline Consult with stakeholders and / or external peer reviewer Disseminate guideline Implement the guideline and evaluate
Grade up Grade down RCT start high, obs. data start low P I C O Outcome Critical Outcome Critical Outcome Important Outcome Not Systematic review Summary of findings & estimate of effect for each outcome High Moderate Low Very low 1. Risk of bias 2. Inconsistency 3. Indirectness 4. Imprecision 5. Publication bias 1. Large effect 2. Dose response 3. Confounders Guideline development Formulate recommendations: For or against (direction) Strong or weak (strength) By considering: Quality of evidence Balance benefits/harms Values and preferences Revise if necessary by considering: Resource use (cost) Rate overall quality of evidence across outcomes based on lowest quality of critical outcomes We recommend using We suggest using We recommend against using We suggest against using
Conclusion clinicians, policy makers need summaries quality of evidence strength of recommendations explicit rules transparent, informative GRADE simple, transparent, systematic increasing wide adoption
PFS in the combined daratumumab 16 mg/kg group. Saad Z. Usmani et al. Blood 2016;128:37-44 2016 by American Society of Hematology