GRADE Grading of Recommendations Assessment, Development and Evaluation British Association of Dermatologists April 2014
Previous grading system Level of evidence Strength of recommendation Level of evidence 1++ 1+ 1-2++ 2+ 2- Type of evidence High-quality meta-analyses, systematic reviews of RCTs, or RCTs with a very low risk of bias Well-conducted meta-analyses, systematic reviews of RCTs, or RCTs with a low risk of bias Meta-analyses, systematic reviews of RCTs, or RCTs with a high risk of bias* High-quality systematic reviews of case-control or cohort studies High-quality case-control or cohort studies with a very low risk of confounding, bias or chance and a high probability that the relationship is causal Well-conducted case-control or cohort studies with a low risk of confounding, bias or chance and a moderate probability that the relationship is causal Case-control or cohort studies with a high risk of confounding, bias or chance and a significant risk that the relationship is not causal* 3 Non-analytical studies (for example, case reports, case series) 4 Expert opinion, formal consensus Class A B C D D (GPP) Evidence At least one meta-analysis, systematic review, or RCT rated as 1++, and directly applicable to the target population, or A systematic review of RCTs or a body of evidence consisting principally of studies rated as 1+, directly applicable to the target population and demonstrating overall consistency of results Evidence drawn from a NICE technology appraisal A body of evidence including studies rated as 2++, directly applicable to the target population and demonstrating overall consistency of results, or Extrapolated evidence from studies rated as 1++ or 1+ A body of evidence including studies rated as 2+, directly applicable to the target population and demonstrating overall consistency of results, or Extrapolated evidence from studies rated as 2++ Evidence level 3 or 4, or Extrapolated evidence from studies rated as 2+, or Formal consensus A good practice point (GPP) is a recommendation for best practice based on the experience of the guideline development group *Studies with a level of evidence - should not be used as a basis for making a recommendation. RCT: randomised controlled trial; NICE: National Institute for Health and Care Excellence.
Advantages of GRADE over other systems Developed by a widely representative group of international guideline developers Clear separation between quality of evidence and strength of recommendations Explicit evaluation of the importance of outcomes of alternative management strategies Explicit, comprehensive criteria for downgrading and upgrading quality of evidence ratings Transparent process of moving from evidence to recommendations Explicit acknowledgment of patient values and preferences Clear, pragmatic interpretation of strong versus weak recommendations for clinicians, patients and policy makers GRADE has been adopted by the WHO, Cochrane Collaboration, NICE, SIGN and around 100 other organisations in 19 countries
PICO method a technique used in evidence-based practice to frame and answer a clinical question P: population/patient I: intervention C: comparator/control (if applicable) O: outcome In an elderly man with acutely painful herpes zoster would treatment with antivirals or antivirals and corticosteroids lead to a more rapid resolution of pain?
Evidence hierarchy
Quality of evidence Evidence: high quality (further research is very unlikely to change our confidence in the estimate of effect) moderate quality (further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate) low quality (further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate) very low quality (any estimate of effect is very uncertain) The quality of evidence reflects the extent to which confidence in an estimate of the effect is adequate to support a particular recommendation
How does GRADE classify the quality of evidence? Evidence based on RCTs begins as high quality evidence, but confidence may be decreased due to five factors: 1. study limitations 2. inconsistency of results 3. indirectness of evidence 4. imprecision 5. reporting/publication bias
How does GRADE classify the quality of evidence? 1. Domains for study limitations: selection bias lack of allocation concealment or sequence generation performance bias lack of blinding attrition bias loss of participants (dropouts; non-responders; stopping early for benefit; protocol deviators and failure to adhere to an intention-to-treat analysis) measurement/detection bias inaccuracy in the measurement instrument; bias in the expectations of study participants, carers or researchers outcome reporting bias selective outcome-reporting or failure to report outcomes
How does GRADE classify the quality of evidence? 2. Inconsistency of results : widely differing estimates of the treatment effect across studies suggest true differences in the underlying treatment effect; heterogeneity of the results without plausible explanation 3. Indirectness of evidence: indirect comparisons of the magnitude of effect of drug A vs. drug B when trials = drug A vs. placebo and drug B vs. placebo; differences between the P, I, C, O 4. Imprecision: studies with relatively few patients and few events wide CI 5. Reporting/publication bias: failure to report studies or funding issue
How does GRADE classify the quality of evidence? For study limitations and indirectness (individually), the quality of evidence is assessed: for each study for the particular outcome then across all studies for the particular outcome (e.g. in metaanalyses) For imprecision, inconsistency and publication bias (individually), the quality of evidence is assessed for ALL the studies as a whole, for a particular outcome
How does GRADE classify the quality of evidence? Using RCTs as an example, for each outcome, obtain an overall quality rating: 0 if no problems with any of the five factors (high) -1 if problem in one factor (high moderate) -2 if problem in two factors (high low) -3 if problem in three factors (high very low)
How does GRADE classify the quality of evidence? Observational studies start with low quality ratings; grading upwards may be warranted if: the magnitude of the treatment effect is large or very large (large effect: RR > 2 or RR < 0.5; very large effect: RR > 5 or RR < 0.2) evidence of a dose-response relationship all plausible confounders/biases would have ordinarily decreased the magnitude of an apparent treatment effect Grading upwards is only possible if evidence has not already been downgraded (due to study limitations, inconsistency, imprecision) Upgrade to: +1 when RR > 2 or RR < 0.5 = (low moderate) +2 when RR > 5 or RR < 0.2 (low high)
How does GRADE classify the quality of evidence? Study design Quality of evidence Lower if Higher if Randomised trials Observational studies High Moderate Low Very low Study limitations Inconsistency Indirectness Imprecision Treatment effect Dose-response relationship Confounders or biases Publication bias
Defining outcomes GRADE is outcome-centric Outcome #1 Outcome #2 Outcome #3 Outcome #4 Quality: High Quality: Moderate Quality: Low Quality: Very low Quality: 3 1++ 4 2- GRADE Previous system Critical outcomes determine the overall quality of evidence
Defining outcomes GRADE challenges guideline developers to: specify all outcomes of importance to patients differentiate outcomes that are critical for decision-making from those that are important but not critical, and those that are less important limit to a suggested maximum of 7 outcomes per clinical/review question
Defining outcomes For decision-making need to know: the outcomes that GDG decides as being critical: mortality and QoL otherwise, outcomes most likely to affect mortality and QoL the magnitude of effect as absolute differences per outcome the overall evidence quality per outcome an evidence profile helps with decision-making For decision-making: look at the clinical importance for each outcome, and not statistical significance (need to know absolute changes, and direction) Judging clinical importance: GDG decides if the effect estimate represents a clinically important difference If there is a clinically important difference, need to indicate the direction (i.e. benefit or harm)
Defining outcomes Example from BAD guidelines on hidradenitis suppurativa: Quality of Life (9) Adverse effects serious (9) Pain (8) Disease-specific physician score (6) Physician s Global Assessment (5) Patient s Global Assessment (5) Adverse effects nuisance (4)
Defining outcomes Example from BAD guidelines on photodynamic therapy: Clearance of treated NMSC (9), or Improvement of inflammatory/infectious dermatoses (9) Recurrence rate (8) Cosmetic outcome (6) Treatment tolerability low or manageable pain (5) Other adverse effects pigmentation, etc. (4)
How does GRADE classify the strength of recommendations? Factors determining the strength of recommendations: balance between desirable and undesirable effects quality of evidence variability (or uncertainty) in patient values and preferences costs (resource allocation) Desirable effects include: reduction in morbidity and mortality improvement in QoL reduction in the burden of treatment reduction in resource expenditures Undesirable effects include: adverse effects having a deleterious impact on morbidity, mortality, QoL or increase in resource expenditures
How does GRADE classify the strength of recommendations? strong (unconditional): desirable effects of an intervention clearly outweigh the undesirable effects (or clearly do not) high-quality methods with large, precise effect low variability or uncertainty in patient values and preferences low resource allocation weak (conditional): desirable effects not clearly greater or smaller than undesirable effects low quality evidence with imprecise estimate patient values and preferences very important high resource allocation N.B. Recommendations to use interventions in a research context may be appropriate
Strength of recommendation NICE wording for recommendations: strong: (the benefits clearly outweigh the harms for most people and the intervention is likely to be cost-effective; the guideline panel believes that a vast majority of clinicians and patients would choose a particular intervention if they considered the evidence in the same way that the panel has) OFFER weak: (there is a closer balance between benefits and harm; some patients are averse to some side effects whilst others are not) CONSIDER Management options associated with strong recommendations are candidates for quality criteria When recommendations are weak, discussing relative merits of alternative management options with patients and families may become a quality criterion
P I C O Outcome Outcome Outcome Outcome Systematic review Critical Critical Important Less Summary of findings & estimate of effect for each outcome High Moderate Low Very low Grade down Grade up RCT start high, observational data start low 1. Risk of bias 2. Inconsistency 3. Indirectness 4. Imprecision 5. Publication bias 1. Large effect 2. Dose response 3. Confounders Guideline development Formulate recommendations: For or against (direction) Strong or weak (strength) By considering: Quality of evidence Balance benefits/harms Values and preferences Resource use (cost) NICE recommendations: Offer Consider Do not offer GDG reviews the evidence profile for rating the overall quality of evidence
RevMan meta-analyses and forest plots ETANERCEPT vs. PLACEBO CRITICAL OUTCOME: Clear/nearly clear (minimal residual activity/pasi>90/pga 0 or 1) at 3-4 months Dose-independent Adults: Strata (children):
GRADE evidence profile