GRADE Grading of Recommendations Assessment, Development and Evaluation British Association of Dermatologists April 2018
Previous grading system Level of evidence Strength of recommendation Level of evidence 1++ 1+ 1-2++ 2+ 2- Type of evidence High-quality meta-analyses, systematic reviews of RCTs, or RCTs with a very low risk of bias Well-conducted meta-analyses, systematic reviews of RCTs, or RCTs with a low risk of bias Meta-analyses, systematic reviews of RCTs, or RCTs with a high risk of bias* High-quality systematic reviews of case-control or cohort studies High-quality case-control or cohort studies with a very low risk of confounding, bias or chance and a high probability that the relationship is causal Well-conducted case-control or cohort studies with a low risk of confounding, bias or chance and a moderate probability that the relationship is causal Case-control or cohort studies with a high risk of confounding, bias or chance and a significant risk that the relationship is not causal* 3 Non-analytical studies (for example, case reports, case series) 4 Expert opinion, formal consensus Class A B C D D (GPP) Evidence At least one meta-analysis, systematic review, or RCT rated as 1++, and directly applicable to the target population, or A systematic review of RCTs or a body of evidence consisting principally of studies rated as 1+, directly applicable to the target population and demonstrating overall consistency of results Evidence drawn from a NICE technology appraisal A body of evidence including studies rated as 2++, directly applicable to the target population and demonstrating overall consistency of results, or Extrapolated evidence from studies rated as 1++ or 1+ A body of evidence including studies rated as 2+, directly applicable to the target population and demonstrating overall consistency of results, or Extrapolated evidence from studies rated as 2++ Evidence level 3 or 4, or Extrapolated evidence from studies rated as 2+, or Formal consensus A good practice point (GPP) is a recommendation for best practice based on the experience of the guideline development group *Studies with a level of evidence - should not be used as a basis for making a recommendation. RCT: randomised controlled trial; NICE: National Institute for Health and Care Excellence.
Advantages of GRADE over other systems Developed by a widely representative group of international guideline developers and systematic reviewers Clear separation between quality of evidence and strength of recommendations Explicit evaluation of the importance of outcomes of alternative management strategies Explicit & comprehensive criteria for downgrading and upgrading quality of evidence ratings Transparent process of moving from evidence to recommendations (LETR) Explicit acknowledgment of patient values and preferences Clear & pragmatic interpretation of strong vs. weak recommendations for clinicians, patients and policy makers GRADE has been adopted by the WHO, Cochrane Collaboration, NICE, SIGN and around 100 other organisations in 19 countries
PICO method a technique used in evidence-based practice to frame and answer a clinical question P: population/patient I: intervention C: comparator/control (if applicable) O: outcome
Evidence hierarchy
How does GRADE classify the quality of evidence? Evidence: high quality (further research is very unlikely to change our confidence in the estimate of effect) moderate quality (further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate) low quality (further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate) very low quality (any estimate of effect is very uncertain) The quality of evidence reflects the extent to which confidence in an estimate of the effect is adequate to support a particular recommendation
How does GRADE classify the quality of evidence? Evidence based on RCTs begins as high quality evidence, but confidence may be decreased due to five factors: 1. study limitations 2. inconsistency of results 3. indirectness of evidence 4. imprecision 5. reporting/publication bias
How does GRADE classify the quality of evidence? 1. Domains for study limitations: selection bias lack of allocation concealment or sequence generation performance bias lack of blinding attrition bias loss of participants (dropouts; non-responders; stopping early for benefit; protocol deviators and failure to adhere to an intention-to-treat analysis) measurement/detection bias inaccuracy in the measurement instrument; bias in the expectations of study participants, carers or researchers outcome reporting bias selective outcome-reporting or failure to report outcomes
How does GRADE classify the quality of evidence? 2. Inconsistency of results : widely differing estimates of a treatment effect across studies suggest true differences in the underlying treatment effect; i.e. heterogeneity (of pooled results) without plausible explanation 3. Indirectness of evidence: indirect comparisons of the magnitude of effect of drug A vs. drug B when trials = drug A vs. placebo and drug B vs. placebo; indirectness may arise in differences in the P, I, C, O 4. Imprecision: studies with relatively few patients and few events wide CI 5. Reporting/publication bias: failure to report studies or funding issue
How does GRADE classify the quality of evidence? For study limitations and indirectness domains, the quality of evidence is assessed: for each study for a particular outcome, then across all studies for that outcome (e.g. in a meta-analysis) For imprecision, inconsistency and publication bias domains, the quality of evidence is assessed for ALL the studies as a whole for a particular outcome (i.e. when appropriate outcome data have been pooled)
How does GRADE classify the quality of evidence? Using RCTs as an example, for each outcome, obtain an overall quality rating: 0 if no problems with any of the five factors (high) -1 if problem in one factor (high moderate) -2 if problem in two factors (high low) -3 if problem in three factors (high very low)
How does GRADE classify the quality of evidence? Observational studies start with low quality ratings; grading upwards may be warranted if: the magnitude of the treatment effect is large or very large (large effect: RR > 2 or RR < 0.5; very large effect: RR > 5 or RR < 0.2) evidence of a dose-response relationship all plausible confounders/biases would have ordinarily decreased the magnitude of an apparent treatment effect Grading upwards is only possible if evidence has not already been downgraded (e.g. due to study limitations, inconsistency, imprecision) Upgrade to: +1 when RR > 2 or RR < 0.5 = (low moderate) +2 when RR > 5 or RR < 0.2 (low high)
How does GRADE classify the quality of evidence? Study design Quality of evidence Lower if Higher if Randomised trials Observational studies High Moderate Low Very low Study limitations Inconsistency Indirectness Imprecision Treatment effect Dose-response relationship Confounders or biases Publication bias
Defining outcomes GRADE is outcome-centric Outcome #1 Outcome #2 Outcome #3 Outcome #4 Quality: High Quality: Moderate Quality: Low Quality: Very low Quality: 3 1++ 4 2- GRADE Previous system Critical outcomes determine the overall quality of evidence
Defining outcomes GRADE challenges guideline developers to: specify all outcomes of importance to patients differentiate outcomes that are critical for decision-making from those that are important but not critical, and those that are less important limit to a suggested maximum of 7 outcomes per clinical/review question
Defining outcomes Example from BAD guideline on hidradenitis suppurativa (medical interventions): Quality of Life (9) Adverse effects serious (9) Pain (8) Disease-specific physician score (6) Physician s Global Assessment (5) Patient s Global Assessment (5) Adverse effects nuisance (4)
Defining outcomes Example from BAD guideline on photodynamic therapy (NMSC): Clearance of treated NMSC at 3 months (9) Sustained clearance at 1, 2 or 5 years (9) Recurrence rate > 1 year (8) Severe pain leading to break in treatment (8) Cosmetic outcome (6) Treatment tolerability low or manageable pain (5) Other adverse effects pigmentation, etc. (4)
Decision-making For decision-making need to look at: the outcomes that GDG decides as being critical (i.e. the clinical importance of each outcome): mortality and QoL otherwise, outcomes most likely to affect mortality and QoL the magnitude of effect, as absolute differences/changes and direction, per outcome the overall evidence quality per outcome a GRADE evidence profile can help Judging clinical importance: GDG decides if the effect estimate represents a clinically important difference if there is a clinically important difference, need to indicate the direction (i.e. benefit or harm)
How does GRADE classify the strength of recommendations? Factors determining the strength of recommendations: balance between desirable and undesirable effects quality of evidence variability (or uncertainty) in patient values and preferences costs (resource allocation) Desirable effects include: reduction in morbidity & mortality improvement in QoL reduction in the burden of treatment reduction in resource expenditures Undesirable effects include: adverse effects having a deleterious impact on morbidity & mortality, QoL or increase in resource expenditures
How does GRADE classify the strength of recommendations? strong (unconditional): desirable effects of an intervention clearly outweigh the undesirable effects (or clearly do not) high-quality methods with large, precise effect low variability or uncertainty in patient values and preferences low resource allocation weak (conditional): desirable effects not clearly greater or smaller than undesirable effects low quality evidence with imprecise estimate patient values and preferences very important high resource allocation N.B. Recommendations to use interventions in a research context may be appropriate
Strength of recommendation NICE wording for recommendations: strong: (the benefits clearly outweigh the harms for most people and the intervention is likely to be cost-effective; the guideline panel believes that a vast majority of clinicians and patients would choose a particular intervention if they considered the evidence in the same way that the panel has) OFFER (or similar, e.g. Initiate, Asses, Investigate, etc.) weak: (there is a closer balance between benefits and harm; some patients are averse to some side effects whilst others are not) CONSIDER N.B. Management options associated with strong recommendations are candidates for quality criteria. When recommendations are weak, discussing relative merits of alternative management options with patients and families may become a quality criterion.
Grade up Grade down P I C O Outcome Outcome Outcome Outcome Systematic review Critical Critical Important Less Summary of findings & estimate of effect for each outcome High Moderate Low Very low RCT start high, observational data start low 1. Risk of bias 2. Inconsistency 3. Indirectness 4. Imprecision 5. Publication bias 1. Large effect 2. Dose response 3. Confounders Guideline development Formulate recommendations: For or against (direction) Strong or weak (strength) By considering: Quality of evidence Balance benefits/harms Values and preferences Resource use (cost) NICE recommendations: Offer Consider Do not offer GDG reviews the evidence profile for rating the overall quality of evidence
ETANERCEPT vs. PLACEBO RevMan meta-analyses and forest plots CRITICAL OUTCOME: Clear/nearly clear (minimal residual activity/pasi>90/pga 0 or 1) at 3-4 months Dose-independent Adults: Strata (children):
GRADE evidence profile
Next steps GRF and GDG reading background information to aid understanding of the clinical areas developing systematic review protocol(s) designing and testing literature search strategies in a guideline carrying out literature searches based on agreed strategies and key words sifting identified literature based on article titles GDG: working in pairs, sifting shortlisted literature based on abstracts and against inclusion and exclusion criteria set in systematic review protocols reading, data-extracting and critically appraising full texts of shortlisted articles preparing lists of excluded studies (with reasons for exclusion) carrying out data syntheses and meta-analyses using the Review Manager software package preparing summary tables of included systematic reviews, comparative, non-comparative and within-patient studies
Next steps GRF and GDG preparing GRADE evidence profiles using the GRADE profiler software package summarising both synthesised and non-synthesised evidence at GDG meetings GDG: attending meeting(s) to discuss the evidence (discussions will be recorded and form part of the LETR) and formulate recommendations GDG: writing introductory/background and other appropriate sections in the main guideline manuscript as required GDG: expanding the discussion points in the LETR into coherent narratives (with reference to evidence identified, synthesised and discussed) in the supplementary information document and finalising the recommendations putting the guideline together (main manuscript and supplementary information, including the PRISMA diagrams) with associated editing and citation work ensuring methodological rigour and compliance with NICE Accreditation criteria documenting all processes and maintaining clear audit trails