Workshop The GRADE Approach to Grading Quality of Evidence and Strength of Recommendations

Transcription

1 Workshop The GRADE Approach to Grading Quality of Evidence and Strength of Recommendations Venue: AMC, Amsterdam, 22 november 2012 Facilitators: Ton Kuijpers, Miranda Langendam, Hans de Beer, Lotty Hooft, Rob Scholten Exercise introduction Work in small groups Select someone from your small group to report back to the whole group Watch the time Follow the subsequent instructions Outline (you will find detailed instructions on the respective pages): 1. Read the abstract of the systematic review (attachment) 2. Identify the clinical question asked in the review and its components (page 2 and worksheet 1 on page 4) 3. Identify the main comparison from the review that you want to work on Read the relevant sections of the review (if marked, please focus on the text marked up on the margin). 4. Consider all outcomes important to patients (page 3 and 4) Decide on the relative importance of each outcome Judge the relative importance of the outcomes Compare your judgements with your colleagues or other participants Discuss disagreements and try to agree 5. Complete the assessment of the quality of evidence in an evidence profile (instructions on page 5) Begin by making judgments about the risk of bias in one study skip this step if instructed by tutors Go to the marked up text in the systematic review or use the review to evaluate the quality of evidence Make judgement about the quality of the evidence for each outcome Make judgments about the overall quality of evidence 6. rovide information about the magnitude of desirable and undesirable outcomes (in GRADEpro) Fill out the table to provide information about the outcomes 7. Develop a recommendation for the use of this intervention (page 10) GRADE Introductory Workshop therapy GRADEpro version GRADE Working Group 1

2 1. Specify details of the review and the clinical question asked Title of the review: Brainstorm which question you are addressing (choose your key comparator, e.g. cleaner stoves vs no intervention) Identify the following components: opulation (atients ): Intervention (I): Comparator (C): Outcomes (O): [consider all outcomes that the population may experience (see and use worksheet 1 on the next page)] Final question: GRADE Introductory Workshop therapy GRADEpro version GRADE Working Group 2

3 2. Choose the outcomes for health decision making Suggestions a) Generate a list of relevant outcomes (see worklist 1 on page 4 of this handout) Use the following strategies if you are not familiar with the topic of interest List outcomes that the authors of the systematic review identified as primary outcomes (see CRITERIA FOR CONSIDERING STUDIES FOR THIS REVIEW) Add other outcomes for which data are reported (see RESULTS and ABSTRACT) Add any other outcomes that were not reported in the review, but you think might be important to someone making a decision to use or not to use the interventions being the subject of the review (make sure to include both benefits and adverse effects and to include costs, if relevant) Find consensus within your small group about which outcomes are important enough to be included in the Evidence rofile (worksheet 1) b) From this list choose up to 7 outcomes (after agreeing with the group) that you think are most important to a guideline panel or others making recommendations and should be included in the evidence profile; transfer them to a blank evidence profile or use GRADEpro (see worksheet 3 on page 6 of this handout or go to ). Work in your small group or in pairs. GRADE Introductory Workshop therapy GRADEpro version GRADE Working Group 3

4 3. Rate the relative importance of the outcomes Worklist 1: List of outcomes for decision making Choose the most important outcomes for decision making. Consider: outcomes that might be important to someone making a decision to use or not to use the interventions (make sure to include both benefits and adverse effects and to include costs, if relevant) outcomes that have been reported in systematic reviews and individual studies Rate the relative importance for each outcome on a 9 point scale ranging from 1 (not important) to 9 (critical). You can use the same rating several times (i.e. same number for more than one outcome) Rate the importance on a 9 point scale (please note that you can assign the same rating several times) 1 3 not important (not included in the evidence profile) 4 6 important, but not critical for making a decision (included in the evidence profile) 7 9 critical for making a decision (included in the evidence profile) Outcome Importance Include in Evidence rofile table? Yes No Yes No Yes No Yes No Yes No Yes No Yes No Yes No 10. Yes No 11. Yes No 12. Yes No 13. Yes No GRADE Introductory Workshop therapy GRADEpro version GRADE Working Group 4

5 4. Working with GRADEpro creating an evidence profile Suggestions Use the instructions on page USE THE HEL FILE/HANDBOOK in GRADEpro if you have questions by moving the cursor over terms about which you have questions a question mark appears and click to open the help file Depending on how many participants have laptops, data entry can be done in a group utilizing the facilitator s laptop or you could split into pairs or groups of three and assign one or several outcomes to each small group Use table the Quality assessment criteria as the basis for your judgment about the quality of the evidence for the outcome or the context specific helpfile in GRADEpro See also Quality of evidence across studies for the outcome (below) for quality definitions reparing Evidence rofiles with GRADEpro 1. Open GRADEpro. 2. Choose New rofile in the welcome screen and name the file. 3. Create profile group (usually overall topic of guidelines or main question) 4. Create a profile (specific health care question) 5. Choose the format of the question in the drop down box (e.g. Should intervention versus comparison be used for health situation?) 6. Add information about the comparison, intervention, setting, etc. 7. Add bibliographic information about the studies or reviews used to create the profile. 8. Create the outcomes by naming them. 9. Select an outcome. For each outcome there are 2 sections: Summary of Findings screen and the Quality Assessment screen. 10. Select the Quality Assessment screen. 11. Complete it by first confirming number and type of studies 12. Assess the quality of evidence for the outcome. Downgrade or upgrade evidence according to GRADE criteria and enter footnotes when necessary. 13. Select Summary of Findings screen. 14. Add data about participants, estimate of effect, baseline risks, etc. 15. Repeat (#7 and #8) for all outcomes in the profile. 16. review GRADE evidence profile, double check presentation and edit if necessary. Do not skip this step. 17. Export the table to a document in word, html, image, etc. GRADE Introductory Workshop therapy GRADEpro version GRADE Working Group 5

6 5. Assess quality of evidence across studies for each outcome in a review in GRADEpro and complete data entry (see example at the end of the workpackage) For detailed instruction regarding assessing the quality of evidence see Appendix 1. For detailed instructions regarding assessing risk of bias/ study limitations in a study see Appendix Grade the overall quality of evidence across all critical outcomes (if you assessed the quality for all outcomes above) Choose one Symbol Quality Interpretation High We are very confident that the true effect lies close to that of the estimate of the effect. Moderate We are moderately confident in the effect estimate: The true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different Low Our confidence in the effect estimate is limited: The true effect may be substantially different from the estimate of the effect. Very low We have very little confidence in the effect estimate: The true effect is likely to be substantially different from the estimate of effect. 7. Move from evidence to recommendation Values and preferences (assume a set of values for each outcome considered) Consider the importance ratings that you assigned to the outcomes above. Example: Values and references For this recommendation we placed a high value on the prevention of death in an illness with a high case fatality. It places relatively low values on adverse reactions, the development of resistance and costs of treatment. Values and preferences associated with this recommendation (assume a set of values for each outcome that you considered) GRADE Introductory Workshop therapy GRADEpro version GRADE Working Group 6

7 8. Decide about the strength of a recommendation (strong or conditional/weak) Use the table on the next page to make a judgment. The four factors in this table will determine whether the recommendation is likely to be strong or conditional ( aka weak ). Frequent positive answers increase the likelihood of a strong recommendation. Make sure to add an explanation for your judgment. GRADE Introductory Workshop therapy GRADEpro version GRADE Working Group 7

8 The four factors in this table will determine whether the recommendation is likely to be strong or conditional ( aka weak ). Frequent positive answers increase the likelihood of a strong recommendation. Make sure to add an explanation for your judgment. Decision table about the strength of a recommendation Question/Recommendation: Should parenteral anticoagulation be used in prolonging survival of patients with cancer? opulation: atients with cancer Intervention: parenteral anticoagulation Setting (if relevant): outpatient Decision domain Judgment Summary of reason for judgment Explanation Subdomains influencing judgment 1. Quality of evidence (QoE) e.g. O Consider Is there high or moderate quality evidence? The higher the quality of evidence, the more likely is a strong yes no (moderate) QoE for benefits QoE for harms QoE for resource use recommendation Key reasons for down or upgrading? All critical outcomes measured? 2. Balance of benefits versus harms and burdens Are you confident that the benefits outweigh the harms and burden or vice versa? The larger the difference between the benefits and harms and the certainty around that difference, the more likely is a strong recommendation. The smaller the net benefit or net harm and the lower the certainty for that net effect, the more likely is a conditional/weak recommendation. yes no. Baseline risk for benefits, harms and burden Is the baseline risk similar across subgroups? Should there be separate recommendations for subgroups? Relative risk for benefits and harms Are the relative benefits large? Are the relative harms large? Requirement for modeling Is there a lot of extrapolation and modeling required for these outcomes? Average values Are there differences in the relative value of the GRADE Introductory Workshop therapy GRADEpro version GRADE Working Group 8

9 Question/Recommendation: Should parenteral anticoagulation be used in prolonging survival of patients with cancer? opulation: atients with cancer Intervention: parenteral anticoagulation Setting (if relevant): outpatient Decision domain Judgment Summary of reason for judgment Explanation Subd omains influencing judgment critical outcomes? 3. Values and preferences Are you confident about the assumed or identified relative values and are they similar across the target population? The more certainty or similarity in values and preferences, the more likely a strong recommendation. 4. Resource implications Are the resources worth the expected net benefit from following the recommendation? The lower the cost of an intervention compared to the alternative, and other costs related to the decision that is, the fewer resources consumed the more likely is a strong recommendation in favour of that intervention. yes no yes no erspective taken: atients or general population/society? Source of values Source of variability if any Method for determining values satisfactory for this recommendation. What are the cost per resource unit? Feasibility Is this intervention generally available? Opportunity cost Is this intervention and its effects worth withdrawing or not allocating resources from other interventions Differences across settings Is there lots of variability in resource requirements across settings? GRADE Introductory Workshop therapy GRADEpro version GRADE Working Group 9

10 If consensus has not been reached by discussion, the panel can use the following table to record their views (votes) about the strength of the recommendation related to a specific management option, based on their analysis of the available evidence and its quality, the benefits and downsides, values and preferences, and resource use (cost). This assessment is then mapped to the strength of recommendation for the use, or nonuse, of each intervention. Insert the number of votes for the recommendation in each category Assessors view of the balance between desirable and undesirable consequences of the intervention Desirable consequences clearly outweigh undesirable consequences Desirable consequences probably outweigh undesirable consequences Undesirable consequences probably outweigh desirable consequences Undesirable consequences clearly outweigh desirable consequences Strength of Strong for an Conditional (weak) Conditional (weak) Strong against an recommendation intervention for an intervention against an intervention intervention Wording of a We recommend to We suggest We suggest We recommend not recommendation do something (conditionally (conditionally to do something recommend) to do recommend) not to something do something Number of votes Strength of the recommendation: Strong Conditional (weak) Final recommendation Strength: Quality of evidence: Rationale Done! GRADE Introductory Workshop therapy GRADEpro version GRADE Working Group 10

11 Appendix 1 Explanations of criteria for down and upgrading EXLANATIONS OF CRITERIA FOR DOWNGRADING Limitations of design: lack of allocation concealment lack of blinding (particularly if outcomes are subjective and their assessment highly susceptible to bias) large loss to follow up failure to adhere to an analysis according to intention to treat principle Selective reporting of events: investigators neglect to report outcomes that they have measured (typically those for which they observed no effect). Inconsistency: Widely differing estimates of the treatment effect (i.e. heterogeneity or variability in results) across studies suggest true differences in underlying treatment effect. When heterogeneity exists, but investigators fail to identify a plausible explanation, the quality of evidence should be downgraded by one or two levels, depending on the magnitude of the inconsistency in the results. Inconsistency may arise from differences in: populations (e.g. drugs may have larger relative effects in sicker populations) interventions (e.g. larger effects with higher drug doses) outcomes (e.g. diminishing treatment effect with time). Indirectness: There are two types of indirectness. 1. Indirect comparison occurs when a comparisons of intervention A versus B is not available, but A was compared with C and B was compared with C. Such trials allow indirect comparisons of the magnitude of effect of A versus B. Such evidence is of lower quality than head to head comparisons of A and B would provide. 2. Indirect population, intervention, comparator, or outcome the question being addressed by the authors of a systematic review is different from the available evidence regarding the population, intervention, comparator, or an outcome. Imprecision: Results are imprecise when studies include relatively few patients and few events and thus have wide confidence intervals around the estimate of the effect. 1. For dichotomous outcomes total (cumulative) sample size is lower than the calculated optimal information size (OIS, comparable to a sample size calculation in a single trial) total number of events is less than 300 a rule of thumb based on simulations and dependent on the baseline risk and effect sizes 95% confidence interval (or alternative estimate of precision) around the pooled or best estimate of effect includes both negligible effect and appreciable benefit or appreciable harm. GRADE suggests that GRADE Introductory Workshop therapy GRADEpro version GRADE Working Group 11

12 threshold for "appreciable benefit" or "appreciable harm" that warrants downgrading is a relative risk reduction (RRR) or relative risk increase (RRI) greater than 25%. Exception When event rates are very low, 95% confidence intervals around relative effects can be very wide, but 95% confidence intervals around absolute effects may be narrow. Under such circumstances one may not downgrade the quality of evidence for imprecision. 2. For continuous outcomes 95% confidence interval includes no effect and the upper or lower confidence limit crosses the minimal important difference (MID), either for benefit of harm if the MID is not known or use of different outcomes measures required calculation of an effect size (ES), we suggest downgrading if the upper or lower confidence limit crosses an effect size of 0.5 in either direction. ublication Bias: ublication bias is a systematic underestimate or an overestimate of the underlying beneficial or harmful effect due to the selective publication of studies (publication bias). That is, investigators fail to report studies they have undertaken (typically those that show no effect) or journals are less likely to accept studies that show no effect for publication. GRADE Introductory Workshop therapy GRADEpro version GRADE Working Group 12

13 EXLANATIONS OF CRITERIA FOR UGRADING Strong Association: When methodologically strong observational studies yield large or very large and consistent estimates of the magnitude of a treatment or exposure effect, we may be confident about the results. In those situations, the weak study design is unlikely to explain all of the apparent benefit or harm, even though observational studies are likely to provide an overestimate of the true effect. The larger the magnitude of effect, the stronger becomes the evidence. Magnitude of effect Effect measure large RR >2 or <0.5 (based on consistent evidence from at least 2 studies, with no plausible confounders): upgrade 1 level very large RR >5 or <0.2 (based on direct evidence with no major threats to validity): upgrade 2 levels Effects of all lausible Confounding: On occasion, all plausible confounding from observational studies or randomised trials may be working to reduce the demonstrated effect or increase the effect if no effect was observed. For example, if only sicker patients receive an experimental intervention or exposure, yet they still fare better, it is likely that the actual intervention or exposure effect is larger than the data suggest. Example 1 A rigorous systematic review of observational studies including a total of 38 million patients demonstrated higher death rates in private for profit versus private not for profit hospitals (Devereaux 2004). One possible bias relates to different disease severity in patients in the two hospital types. It is likely, however, that patients in the not for profit hospitals were sicker than those in the for profit hospitals. Thus, to the extent that residual confounding existed, it would bias results against the not for profit hospitals. The second likely bias was the possibility that higher numbers of patients with excellent private insurance coverage could lead to a hospital having more resources and a spill over effect that would benefit those without such coverage. Since for profit hospitals are likely to admit a larger proportion of such well insured patients than not for profit hospitals, the bias is once again against the not for profit hospitals. Because the plausible biases would all diminish the demonstrated intervention effect, one might consider the evidence from these observational studies as moderate rather than low quality. Example 2 A parallel situation exists when observational studies have failed to demonstrate an association but all plausible biases would have increased an intervention effect. This situation will usually arise in the exploration of apparent harmful effects. For example, because the hypoglycaemic drug phenformin causes lactic acidosis, the related agent metformin is under suspicion for the same toxicity. Nevertheless, very large observational studies have failed to demonstrate an association (Salpeter S, Greyber E, asternak G, Salpeter E. Risk of fatal and nonfatal lactic acidosis with metformin use in type 2 diabetes mellitus. Cochrane Database of Systematic Reviews 2007, Issue 4. Art No: CD ). Given the likelihood that clinicians would be more alert to lactic acidosis in the presence of the agent and overreport its occurence, one might consider this moderate, or even high quality evidence refuting a causal relationship between typical therapeutic doses of metformin and lactic acidosis. Only a body of evidence with no important threats to validity should be upgraded. GRADE Introductory Workshop therapy GRADEpro version GRADE Working Group 13

14 Dose response relation: The presence of a dose response gradient may increase our confidence in the findings of observational studies and thereby increase the quality of evidence. Only studies with no threats to validity (not downgraded for any reason) can be upgraded. Example The observation that, in patients receiving anticoagulation with warfarin, there is a dose response gradient between higher levels of the international normalized ratio (INR), an indicator of the degree of anticoagulation, and an increased risk of bleeding increases our confidence that supratherapeutic anticoagulation levels increase bleeding risk. GRADE Introductory Workshop therapy GRADEpro version GRADE Working Group 14

15 Appendix 2 Assessing risk of bias / study limitations Criteria for assessing risk of bias/ study limitations in RCTs Criteria for RCTs 1 Description Yes/unclear /No Quote from study Lack of allocation concealment Lack of blinding Incomplete accounting of patients & outcome events Selective outcome reporting Other limitations Those enrolling patients are aware of the group to which the next enrolled patient will be allocated (major problem in pseudo or quasi randomized trials with allocation by day of week, birth date, chart number etc.) atient, caregivers, those recording outcomes, those adjudicating outcomes, or data analysts are aware of the arm to which patients are allocated Loss to follow up and failure to adhere to the intention to treat principle when indicated Reporting of some outcomes and not others on the basis of the results For example: stopping early for benefit observed in randomized trials, in particular in the absence of adequate stopping rules use of unvalidated patientreported outcomes carry over effects in cross over trials recruitment bias in clusterrandomized trials 1 For criteria assessing risk of bias in cohort studies see Appendix 1on page 11 of this handout GRADE Introductory Workshop therapy GRADEpro version GRADE Working Group 15

16 Criteria for assessing risk of bias in cohort studies 1. Was selection of exposed and non exposed cohorts drawn from the same population? Definitely yes robably yes robably no Definitely no (low risk of bias) (high risk of bias) Examples of low risk of bias: Exposed and unexposed drawn for same administrative data base of patients presenting at same points of care over the same time frame Examples of high risk of bias: exposed and unexposed presenting to different points of care or over a different time frame 2. Can we be confident in the assessment of exposure? Definitely yes robably yes robably no Definitely no (low risk of bias) (high risk of bias) Examples of low risk of bias: Secure record [e.g. surgical records, pharmacy records]; Repeated interview or other ascertainment asking about current use/exposure Examples of higher risk of bias: Structured interview at a single point in time; Written self report; Individuals who are asked to retrospectively confirm their exposure status may be subject to recall bias less likely to recall an exposure if they have not developed an adverse outcome, and more likely to recall an exposure (whether an exposure occurred or not) if they have developed an adverse outcome. Examples of high risk of bias: uncertain how exposure information obtained 3. Can we be confident that the outcome of interest was not present at start of study Definitely yes robably yes robably no Definitely no (low risk of bias) (high risk of bias) 4. Did the study match exposed and unexposed for all variables that are associated with the outcome of interest or did the statistical analysis adjust for all prognostic variables? Definitely yes Mostly yes Mostly no Definitely no (low risk of bias) (high risk of bias) Examples of low risk of bias: comprehensive matching or adjustment for all plausible prognostic variables Examples of higher risk of bias: matching or adjustment for most plausible prognostic variables Examples of high risk of bias: matching or adjustment for a minority of plausible prognostic variables, or no matching or adjustment at all. Statements of no differences between groups or that differences were not statistically significant are not sufficient for establishing comparability. 5. Can we be confident in the assessment of the presence or absence of prognostic factors? Definitely yes robably yes robably no Definitely no (low risk of bias) (high risk of bias) GRADE Introductory Workshop therapy GRADEpro version GRADE Working Group 16

17 Examples of low risk of bias: Interview of all participants; self completed survey from all participants; review of charts with reproducibility demonstrated; from data base with documentation of accuracy of abstraction of prognostic data Examples of higher risk of bias: Chart review without demonstration of reproducibility; data base with uncertain quality of abstraction of prognostic information Examples of high risk of bias: rognostic information from data base with no available documentation of quality of abstraction of prognostic variables 6. Can we be confident in the assessment of outcome? Definitely yes robably yes robably no Definitely no (low risk of bias) (high risk of bias) Examples of low risk of bias: Independent blind assessment; Record linkage; For some outcomes (e.g. fractured hip), reference to the medical record is sufficient to satisfy the requirement for confirmation of the fracture. Examples of higher risk of bias: Independent assessment unblinded; self report; For some outcomes (e.g. vertebral fracture where reference to x rays would be required) reference to the medical record would not be adequate outcomes. Examples of high risk of bias: uncertain (no description) 7. Was the follow up of cohorts adequate? Definitely yes robably yes robably no Definitely no (low risk of bias) (high risk of bias) Examples of low risk of bias: No missing outcome data; Reasons for missing outcome data unlikely to be related to true outcome (for survival data, censoring is unlikely to introduce bias); Missing outcome data balanced in numbers across intervention groups, with similar reasons for missing data across groups; For dichotomous outcome data, the proportion of missing outcomes compared with observed event risk is not enough to have a important impact on the intervention effect estimate; For continuous outcome data, plausible effect size (difference in means or standardized difference in means) among missing outcomes is not large enough to have an important impact on the observed effect size; Missing data have been imputed using appropriate methods. Examples of high risk of bias: Reason for missing outcome data likely to be related to true outcome, with either imbalance in numbers or reasons for missing data across intervention groups; For dichotomous outcome data, the proportion of missing outcomes compared with observed event risk is enough to induce important bias in intervention effect estimate; For continuous outcome data, plausible effect size (difference in means or standardized difference in means) among missing outcomes is large enough to induce clinically relevant bias in the observed effect size; As treated analysis done with substantial departure of the intervention received from that assigned at randomization; otentially inappropriate application of simple imputation. 8. Were co Interventions Similar Between Groups? Definitely yes robably yes robably no Definitely no (low risk of bias) (high risk of bias) Examples of low risk of bias: Most or all relevant co interventions that might influence the outcome of interest are documented to be similar in the exposed and unexposed. Examples of high risk of bias: Few or no relevant co interventions that might influence the outcome of interest are documented to be similar in the exposed and unexposed. GRADE Introductory Workshop therapy GRADEpro version GRADE Working Group 17

18 Worksheet 3: Evidence rofile ICO: Author(s): Date: Question: Settings: Bibliography: Quality assessment No of patients No of studies Design Limitations Inconsistency Indirectness Imprecision Other considerations Intervention Summary of findings Control Relative (95% CI) Effect Absolute Quality Importance EXAMLE: Complete response of tumor to chemotherapy 5 randomised trial limitations serious 1 indirectness imprecision ublicationx bias 4 216/ /344 RR 1.0 (0.92 to 1.1) 0 fewer per 1000 (from 49 fewer to 61 more) LOW CRITICAL Mention footnotes here GRADE Introductory Workshop therapy GRADEpro version GRADE Working Group 18

19 EXAMLE SUMMARY OF FINDINGS TABLE Self management for patients with chronic obstructive pulmonary disease atient or population: patients with chronic obstructive pulmonary disease Settings: primary care, community, outpatient Intervention: self management 1 Comparison: usual care Outcomes Illustrative comparative risks* Relative No of Quality of the Comments (95% CI) effect articipants evidence Assumed risk Corresponding risk (95% CI) (studies) (GRADE) usual care self management Quality of Life St George's Respiratory life ranged across Questionnaire. Scale from: 0 to 100. (follow up: 3 to 12 The mean quality of control groups from 38 to 60 points The mean quality of Life in the intervention groups was 2.58 lower (5.14 to 0.02 lower) 698 (7) months) Dyspnoea The mean dyspnoea The mean dyspnoea in 144 Borg Scale. Scale from: ranged across control the intervention (2) 0 to 10. groups from groups was (follow up: 3 to to 4.1 points 0.53 lower months) (0.96 to 0.1 lower) Number and severity See comment See comment Not 591 of exacerbations 5 estimable 5 (3) moderate 2 low 3,4 Lower score indicates better quality of life. A change of less than 4 points is not shown to be important to patients. Lower score indicates improvement See comment Effect is uncertain Respiratory related hospital admissions (follow up: 3 to 12 months) Low risk population 6 10 per per 100 (5 to 9) High risk population 6 OR 0.64 (0.47 to 0.89) 966 (8) moderate 7 Emergency department visits for lung diseases (follow up: 6 to 12 months) 50 per per 100 (32 to 47) The mean emergency The mean emergency department visits for department visits for lung diseases ranged lung diseases in the across control groups intervention groups from was 0.2 to 0.7 visits per 0.1 higher person per year (0.2 lower to 0.3 higher) Doctor and nurse visits The mean doctor and The mean doctor and (follow up: 6 to 12 nurse visits ranged nurse visits in the months) across control groups intervention groups from 1 to 5 vists per person per year was 0.02 higher (1 lower to 1 higher) 328 (4) 629 (8) moderate 4 moderate 8 *The basis for the assumed risk (e.g. the median control group risk across studies) is provided in footnotes. The corresponding risk (and its 95% confidence interval) is based on the assumed risk in the comparison group and the relative effect of the intervention (and its 95% CI). CI: Confidence interval; OR: Odds ratio; GRADE Working Group grades of evidence GRADE Introductory Workshop therapy GRADEpro version GRADE Working Group 19

20 High quality: We are very confident that the true effect lies close to that of the estimate of the effect. Moderate quality: We are moderately confident in the effect estimate: The true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different Low quality: Our confidence in the effect estimate is limited: The true effect may be substantially different from the estimate of the effect. Very low quality: We have very little confidence in the effect estimate: The true effect is likely to be substantially different from the estimate of effect. 1 regimens specific to the disease, guide health behaviour change, and provide emotional support for patients to control their disease and live functional lives. Of the 14 studies, there were four in which the education delivery mode consisted of group education; nine which were individual education and one study which was written education material only. In six studies the use of an action plan for selfof exacerbations was treatment assessed and 51% to 50%). 7 applicability of effect to all risk groups. 8 Self management is a term applied to any formalized patient education programme aimed at teaching skills needed to carry out medical Seven other studies were not pooled and some showed non significant effects. No allocation concealment in 1 study. Incomplete follow up. Sparse data. Different definitions of exacerbations used and studies could not be pooled. The low and high risk values are the two extreme numbers of admissions in the control groups from two studies (8% was rounded to 10% Two studies with very severe COD patients weighted heavily in meta analysis. Therefore, there is some uncertainty with the Unexplained heterogeneity. We are very confident that the true effect lies close to that of the estimate of the effect. We are moderately confident in the effect estimate: The true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different Our confidence in the effect estimate is limited: The true effect may be substantially different from the estimate of the effect. We have very little confidence in the effect estimate: The true effect is likely to be substantially different from the estimate of effect. GRADE Introductory Workshop therapy GRADEpro version GRADE Working Group 20

21 EXAMLE GRADE EVIDENCE ROFILE Question: Should intranasal levocabastine vs placebo be used in adults with seasonal allergic rhinitis? Bibliography: 1. Dahl R., edersen B., Larsen B. Intranasal levocabastine for the treatment of seasonal allergic rhinitis: a multicentre, double blind, placebo controlled trial. Rhinology, 1995;33: Di Lorenzo G., Gervasi F., Drago A., Esposito ellitteri M., Di Salvo A., Cosentino D., otestio M., Colombo A., Candore G., Mansueto S., Caruso C. Comparison of the effects of fluticasone propionate, aqueous nasal spray and levocabastine on inflammatory cells in nasal lavage and clinical activity during the pollen season in seasonal rhinitics. Clin Exp Allergy, 1999;29: Hampel F., Jr., Martin B.G., Dolen J., Travers S., Karcher K., Holton D. Efficacy and safety of levocabastine nasal spray for seasonal allergic rhinitis. Am J Rhinol, 1999;13: Ortolani C., Foresi A., Di Lorenzo G., Bagnato G., Bonifazi F., Crimi N., Emmi L., randini M., Senna G.E., Tursi A., Mirone C., Leone C., Fina., Testi R. A double blind, placebo controlled comparison of treatment with fluticasone propionate and levocabastine in patients with seasonal allergic rhinitis. FLNCO2 Italian Study Group. Allergy, 1999;54: Schata M., Jorde W., Richarz Barthauer U. Levocabastine nasal spray better than sodium cromoglycate and placebo in the topical treatment of seasonal allergic rhinitis. The Journal of allergy and clinical immunology, 1991;87: Svensson C., Andersson M., Greiff L., Blychert L.O., ersson C.G. Effects of topical budesonide and levocabastine on nasal symptoms and plasma exudation responses in seasonal allergic rhinitis. Allergy, 1998;53: No of studies Quality assessment Design Limitations Inconsistency Indirectness Imprecision nasal symptoms (follow up 4 to 6 weeks; Better indicated by lower values) Other considerations No of patients intranasal levocabastine placebo Summary of findings Relative (95% CI) Effect Absolute Quality Importance 6 randomised trials limitations 1 inconsistency indirectness imprecision 2 none SMD 0.48 lower (0.15 to 0.81 lower) 3 HIGH CRITICAL nasal symptoms (global assessment of efficacy) (follow up 2 to 4 weeks; rating: excellent or good) 2 randomised trials limitations 4 inconsistency indirectness serious 5 none 89/150 (59.3%) 53/149 (35.6%) RR 1.83 (1.16 to 2.89) 30 more per 100 (from 6 more to 67 more) MODERATE CRITICAL ocular symptoms (follow up 2 to 6 weeks; Better indicated by lower values) 4 randomised trials limitations 6 inconsistency indirectness serious 7 none not pooled 8 MODERATE IMORTANT quality of life not measured 0 none 0 0 CRITICAL adverse effects (follow up 2 to 4 weeks; somnolence/fatigue 9 ) 2 randomised trials serious 9,10 inconsistency indirectness serious 11 none 4/151 (2.6%) 1/157 (0.6%) RR 3.17 (0.51 to 19.71) 14 more per 1000 (from 3 fewer to 119 more) LOW CRITICAL 1 One study blinded patients only and 2 did not report type of analysis (overall 10% of patients did not complete these studies) 2 Results do not exclude small or large effect. 3 Other 3 trials did not report variability in results so it was not possible to combine their results. However, all reported statistically significant difference favouring levocabastine (one showed a difference of ~12% and the other of ~28%; the third trial did not report the scale or variability in results). 4 In one study 25% patients did not complete treatment. 5 Results do not exclude a large effect or a negligible one. Only 142 events. 6 One study blinded patients only, 2 did not report type of analysis, and only one study reported variability in results. 7 It was not possible to estimate the precision of the findings since only one study provided variability in results. 8 One study reported 18% difference vs placebo, second reported 71.4% days free of eye symptoms compared to 56.1% with placebo, third reported a change in symptoms 1.92 vs 0.19 on a 4 point scale, and the last reported an effect size of 0.28 (95% CI: 1.10 to 0.54) favouring levocabastine. 9 Other adverse effects were poorly reported. In one study 4 vs 3 patients withdrew due to unspecified adverse effects, one reported that there were no adverse effects, one provided a detailed table but there were no differences and 2 studies dd not repot adverse effects at all % patients did not complete the studies 11 Results do not exclude a large harm or a benefit. Very few events. GRADE Introductory Workshop therapy GRADEpro version GRADE Working Group 21