ANNALS OF EMERGENCY MEDICINE JOURNAL CLUB. Wesley H. Self, MD, MPH, Tyler W. Barrett, MD, MSCI

Transcription

1 ANNALS OF EMERGENCY MEDICINE JOURNAL CLUB Is PERC Negative Adequate to Rule Out Pulmonary Embolism in the Emergency Department? Evaluating Meta-analysis for Studies of Clinical Prediction Models Answers to the July 2012 Journal Club Questions Wesley H. Self, MD, MPH, Tyler W. Barrett, MD, MSCI From the Vanderbilt University Medical Center, Nashville, TN /$-see front matter Copyright 2012 by the American College of Emergency Physicians. Editor s Note: You are reading the 28th installment of Annals of Emergency Medicine Journal Club. This Journal Club refers to the Singh et al article that was published in the June 2012 Annals of Emergency Medicine issue. Information about Journal Club can be found at content/journalclub. Readers should recognize that these are suggested answers. We hope they are accurate; we know that they are not comprehensive. There are many other points that could be made about these questions or about the article in general. Questions are rated novice, ( ) intermediate, ( ) and advanced ( ) so that individuals planning a journal club can assign the right question to the right student. The novice rating does not imply that a novice should be able to spontaneously answer the question. Novice means we expect that someone with little background should be able to do a bit of reading, formulate an answer, and teach the material to others. Intermediate and advanced questions also will likely require some reading and research, and that reading will be sufficiently difficult that some background in clinical epidemiology will be helpful in understanding the reading and concepts. We are interested in receiving feedback about this feature. Please journalclub@acep.org with your comments. DISCUSSION POINTS 1. Meta-analysis is a statistical technique used to pool data collected from multiple research studies addressing the same or similar questions. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement was developed to assist researchers in developing and reporting sound methods for metaanalyses. 1 A. Singh et al 2 reported a meta-analysis of published literature evaluating the diagnostic accuracy of the pulmonary embolism rule-out criteria (PERC) in the emergency department (ED) developed by Kline et al. 3 Review the PRISMA recommended elements (available at htm). Discuss whether Singh et al 2 reported each of these elements completely. The respondent may also wish to review the January 2010 Journal Club answers that examined PRISMA. 4 B. Singh et al 2 included 11 studies in their meta-analysis. To select studies for inclusion in the meta-analysis, the authors sought published articles and abstracts of original work reporting the diagnostic accuracy of PERC for pulmonary embolism in the ED. Two reviewers independently evaluated each potentially eligible study and judged whether each should be included. The authors observed that agreement between the 2 reviewers about whether an article should be included was excellent, with Cohen s What does measure? Do you agree that 0.80 represents excellent agreement in this setting? C. The authors sought to include only high-quality studies in the meta-analysis by excluding studies that scored less than 50% on their methodological quality checklist. Why is consideration of methodological quality of individual studies important in meta-analyses? Other than excluding low-quality studies, what other methods exist for accounting for differences in methodological quality in meta-analyses? D. Among the 11 studies included in this meta-analysis, 3 were abstracts presented at national meetings, was a letter to the editor, 8 and 7 were peer-reviewed published articles. 3,9-14 What are the differences in the review and publication process for abstracts and letters to the editor compared with full peer-reviewed articles? Should data presented in abstracts and letters be considered equal to that presented in peer-reviewed articles? E. Six of the 11 studies included in the meta-analysis were authored by at least 1 researcher involved in the original derivation of PERC. These 6 studies 3,5,6,10,12,14 included 79% of the total patients included in the metaanalysis, indicating that most of the research was conducted by a few investigators. Is this a limitation to the meta-analysis? Volume 60, NO. 6 : December 2012 Annals of Emergency Medicine 803

2 2. The authors reported that PERC had a pooled sensitivity and specificity for ruling out pulmonary embolism of 0.97 (95% confidence interval 0.96 to 0.98) and 0.23 (95% confidence interval 0.22 to 0.24), respectively. A. How do you interpret the meaning of this pooled sensitivity and specificity when deciding whether to use the PERC rule in the evaluation of patients within the ED? B. In the abstract, the authors state that the specificity of PERC was low but acceptable. Do you agree this low specificity is acceptable? C. In the Editor s Capsule Summary, the editors concluded: This pooled analysis strongly corroborates the safety of using PERC to defer D-dimer testing. Given that 2% to 4% of patients with a pulmonary embolism are PERC negative, is this conclusion justified? When discussing your answer, consider the risk-benefit ratio of D-dimer testing in patients who are PERC negative. D. Aujesky et al 15 published the results of a randomized, noninferiority trial in Lancet suggesting that outpatient treatment is safe and effective in selected, low-risk patients with pulmonary embolism. Additionally, a recent trial published in the New England Journal of Medicine suggested that the oral anticoagulant rivaroxaban resulted in fewer major bleeding complications and similar efficacy for pulmonary embolism treatment compared with traditional anticoagulation (enoxaparin bridged to warfarin). 16 Using these examples, discuss how safer management options for pulmonary embolism may change the riskbenefit ratio for D-dimer testing among patients who are PERC negative. 3. Results of individual studies included in the metaanalysis are shown in Figure 2 of the Singh et al 2 article. A. What is the name of this type of plot and why is it commonly used in meta-analyses? What is the significance of the different sizes of the dots in Figure 2? B. In the Limitations section, the authors observed that they did not evaluate for the possibility of publication bias because the meta-analysis included fewer than 20 studies. What is publication bias? Why does a small number of studies included in a meta-analysis preclude assessing for publication bias? C. Consider a hypothetical meta-analysis evaluating the efficacy of the herbal therapy nocturno-doc to maintain alertness of emergency physicians during night shifts. Eight years after release of nocturno-doc onto the market, 10 studies reporting efficacy were published; a meta-analysis of these studies demonstrated high efficacy for nocturno-doc. A funnel plot of these studies is displayed in Figure 1. Discuss how publication bias may have affected the results of this hypothetical meta-analysis of nocturno-doc efficacy. Figure 1. Funnel plot of hypothetical studies evaluating efficacy of nocturo-doc treatment. 4. In an Annals of Emergency Medicine editorial, Newman and Schriger 17 suggested that diagnosing a small pulmonary embolism on computed tomography scan in a patient without physiologic compromise may not be beneficial. They argued that small pulmonary emboli without physiologic strain appear to be common and nearly universally nonfatal. 18 Furthermore, they observed that anticoagulation is routinely prescribed to treat small pulmonary emboli, but the harm of anticoagulation may outweigh the benefit in this setting. 19 A. PERC was developed to maximize sensitivity for the diagnosis of any pulmonary embolism. Would a pulmonary embolism clinical decision tool based on a different, patient-centered outcome, such as fatal pulmonary embolism or pulmonary embolism resulting in long-term disability, be better? B. Discuss how a clinical prediction model based on an outcome of pulmonary embolism resulting in right-sided heart failure might differ from PERC. ANSWER 1 Q1. Meta-analysis is a statistical technique used to pool data collected from multiple research studies addressing the same or similar questions. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement was developed to assist researchers in developing and reporting sound methods for meta-analyses. 1 Q1.a Singh et al 2 reported a meta-analysis of published literature evaluating the diagnostic accuracy of the pulmonary embolism rule-out criteria (PERC) in the emergency department (ED) developed by Kline et al. 3 Review the PRISMA recommended elements (available at org/index.htm). Discuss whether Singh et al 2 reported each of these elements completely. The respondent may also wish to review the January 2010 Journal Club answers that examined PRISMA Annals of Emergency Medicine Volume 60, NO. 6 : December 2012

3 PRISMA s objective is to help authors report a wide array of systematic reviews to assess the benefits and harms of a health care intervention. PRISMA focuses on ways in which authors can ensure the transparent and complete reporting of systematic reviews and meta-analyses. 20 In Table 1, we compare the Singh et al 2 meta-analysis with the PRISMA guidelines. Also note the discussion of PRISMA in the answers to the January 2010 Journal Club. 4 Readers should understand that a study s adherence to the appropriate methodology statement (eg, PRISMA, CONSORT) alone does not guarantee that the metaanalysis or trial is a high-quality investigation. Skilled medical writers can create highly regarded papers from poor data and erroneous statistical analyses. A proposed yet still uncommon solution to validate a study s conclusions is to require the investigators to make their original data available for independent review and analysis. Q1.b Singh et al 2 included 11 studies in their meta-analysis. To select studies for inclusion in the meta-analysis, the authors sought published articles and abstracts of original work reporting the diagnostic accuracy of PERC for pulmonary embolism in the ED. Two reviewers independently evaluated each potentially eligible study and judged whether each should be included. The authors observed that agreement between the 2 reviewers about whether an article should be included was excellent, with Cohen s What does measure? Do you agree that 0.80 represents excellent agreement in this setting? Cohen introduced the statistic as a test to measure agreement for nominal scales beyond chance. 21 The statistic is defined as: % Agreement observed % Agreement expected due to chance 1 % Agreement expected due to chance The statistic is used to measure the agreement or interrater reliability when 2 observers independently repeat a measurement. In the meta-analysis by Singh et al, 2 is used to measure the level of agreement between 2 investigators during decisions about which studies should be included in the metaanalysis. We agree that, in this context, a 0.80 is good evidence that the inclusion and exclusion criteria for selecting studies across investigators were applied with adequate consistency. However, interpreting the significance of a particular level is highly dependent on the question under consideration. The 1977 Landis and Koch 22 article, which is frequently cited to describe qualitative descriptors of values, introduced the following interpretations: less than 0 as poor agreement; 0 to 0.20 as slight agreement; 0.21 to 0.40 as fair agreement; 0.41 to 0.60 as moderate agreement; 0.61 to 0.80 as substantial agreement; and 0.81 to 1.00 as almost perfect agreement. However, as previously discussed by Day and Schriger 23 in the answers to the July 2009 Journal Club, the Landis and Koch 22 interpretations of values cannot be blindly applied without consideration of context. We have included an excerpt from the Day and Schriger 23 commentary here: Many investigators contrast their values to arbitrary guidelines originally proposed by Landis and Koch 22 and further popularized by Fleiss. 24 [However,] the mechanical mapping of numeric values of to the [Landis and Koch] adjectives...is fraught with problems. A of.75 might be good enough if the cost of being wrong is low (such as categorizing subjects into personality types), but nothing less than nearperfect agreement is requisite if the decision has important consequences. We would not be pleased if our airplane s copilots attained a of 0.75 on is it safe to land? Some tests (eg, a set of historical questions that are used to identify patients at high risk for alcohol addiction) might be useful even if their results are only somewhat reliable. Other tests, however (eg, a set of history and physical examination data that are used to identify which patients with traumatic neck pain can safely forgo cervical spine radiography), will be useful only if they are highly reliable. This is because no poorly reliable test will ever be highly valid when used by multiple fallible observers. Conceptualizing any specific degree of agreement as poor, excellent, or anywhere in between regardless of the test s clinical context is, therefore, a dangerous oversimplification. 23 Q1.c The authors sought to include only high-quality studies in the meta-analysis by excluding studies that scored less than 50% on their methodological quality checklist. Why is consideration of methodological quality of individual studies important in metaanalyses? Other than excluding low-quality studies, what other methods exist for accounting for differences in methodological quality in meta-analyses? When multiple studies are conducted to evaluate the same clinical question, there is likely to be variation in the quality of those studies. Because of methodological flaws, lower-quality studies may systematically result in different and inaccurate results compared with higher-quality studies. If low- and highquality studies are equally considered in a meta-analysis, data from low-quality studies may dilute the data from high-quality studies and lead to inaccurate conclusions. For example, consider these 2 hypothetical studies evaluating the efficacy of a new antibiotic for the treatment of adults with sinusitis. In study A, patients were randomized to receive antibiotics plus decongestants (experimental group) or decongestants alone (control group) in an open-label, nonblinded fashion patients and clinicians knew who received antibiotics and who did not. In study B, patients were randomized to receive antibiotics plus decongestants (experimental group) or placebo plus decongestants (control group) in a blinded fashion with clinicians and patients not knowing who received the antibiotics. The primary outcome for both studies was patients report of sinusitis-related symptoms at 10 days. The lack of blinding in Study A is a serious threat to internal validity; this may lead to an overestimation of the antibiotic s efficacy due to patients expectations that the antibiotic will lead to a cure. With patients blinded to treatment allocation, Study B has a stronger methodological design and is likely to reveal a more accurate estimate of the antibiotic s Volume 60, NO. 6 : December 2012 Annals of Emergency Medicine 805

4 Table 1. Overview of recommended elements for published systematic reviews and meta-analyses by PRISMA 20 and evaluation of complaince with these guidelines in the Singh et al 2 meta-analysis. Section/Topic No. PRISMA Checklist Item Item Reported in Singh et al 2 Title Title 1 Identify the report as a systematic review, metaanalysis, or both. Abstract Structured summary 2 Provide a structured summary including, as applicable, background; objectives; data sources; study eligibility criteria, participants, and interventions; study appraisal and synthesis methods; results; limitations; conclusions and implications of key findings; systematic review registration number. Introduction Rationale 3 Describe the rationale for the review in the context of what is already known. Objectives 4 Provide an explicit statement of questions being addressed with reference to participants, interventions, comparisons, outcomes, and study design. Methods Protocol and registration 5 Indicate whether a review protocol exists, indicate whether and where it can be accessed (eg, Web address), and, if available, provide registration information, including registration number. Eligibility criteria 6 Specify study characteristics, length of follow-up, and criteria used to establish study eligibility. Information sources 7 Describe all information sources (eg, databases with dates of coverage, contact with study authors to identify additional studies) in the search and date last searched. Search 8 Present full electronic search strategy for at least 1 database, including any limits used, such that it could be repeated. Study selection 9 State the process for selecting studies (ie, screening, determining eligibility to include in the systematic review and meta-analysis). Data collection process 10 Describe method of data extraction from reports and any processes for obtaining and confirming data from investigators. Data items 11 List and define all variables for which data were sought and any assumptions and simplifications made. Yes Yes Yes. Singh et al 2 stated, Emergency physicians have been increasing their use of diagnostic testing in an attempt to avoid missing this potentially life-threatening diagnosis (pulmonary embolism), increasing both cost and use of medical resources... However, a recent systematic review of clinical decision rules for pulmonary embolism did not include PERC. 2 Yes, The objective is clearly described as we performed a systematic review and meta-analysis to summarize the diagnostic accuracy of PERC. The authors detail their search strategy with additional specifics provided in an online appendix. The authors did not report registration information. Yes, the authors detail their search strategy in the methods section and provide additional details in an online appendix. Yes, the first sentence of the methods section reports, We performed a comprehensive search of the following biomedical databases through August 14, 2011: EMBASE, MEDLINE, SCOPUS, Web of Knowledge, and all the EBM reviews that included the Cochrane Database of Systematic Reviews. Yes, this information is provided in the appendix. Yes, the authors detail their procedures in the second paragraph of the methods section. Yes, in the last paragraph of the methods, the authors state, Two reviewers then independently extracted data from the included articles, using a predesigned form, and assessed the reported quality of the methods. They also describe the specific data points abstracted and the primary outcome diagnosis of pulmonary embolism or venous thromboembolism or death caused by venous thromboembolism within 90 days of initial ED evaluation. Singh et al 2 clearly reported, Data points were study characteristics (author, country, publication year, number of patients, study settings, study design, description of study participants, and duration of follow-up), subject selection (inclusion and exclusion criteria), PERC classification, outcome definition and measurement, outcomes in PERC positives and negatives, and follow-up. 806 Annals of Emergency Medicine Volume 60, NO. 6 : December 2012

5 Table 1. Continued Section/Topic No. PRISMA Checklist Item Item Reported in Singh et al 2 Risk of bias in individual studies 12 Describe methods used for assessing risk of bias of individual studies (including specification of whether this was done at the study or outcome level) and how this information is to be used in any data synthesis. Summary measures 13 State the principal summary measures (eg, risk ratio, difference in means). Synthesis of results 14 Describe the methods of handling data and combining results of studies, if done, including measures of consistency (eg, I 2 ) for each meta-analysis. Risk of bias across studies 15 Specify any assessment of risk of bias that may affect the cumulative evidence (eg, publication bias, selective reporting within studies). Additional analyses 16 Describe methods of additional analyses (eg, sensitivity or subgroup analyses, meta-regression), if done, indicating which were prespecified. Results Study selection 17 Give numbers of studies screened, assessed for eligibility, and included in the review, with reasons for exclusions at each stage, ideally with a flow diagram. Study 18 For each study, present characteristics for which data characteristics were extracted and provide the citations. Risk of bias within studies Results of individual studies 19 Present data on risk of bias of each study and, if available, any outcome level assessment (see item 12). 20 For all outcomes considered (benefits or harms), present, for each study, (a) simple summary data for each intervention group; (b) effect estimates and CIs, ideally with a forest plot. Synthesis of results 21 Present results of each meta-analysis done, including CIs and measures of consistency. Risk of bias across 22 Present results of any assessment of risk of bias studies across studies (see item 15). Additional analysis 23 Give results of additional analyses, if done (eg, sensitivity or subgroup analyses, meta-regression [see item 16]). Discussion Summary of 24 Summarize the main findings, including the strength of evidence evidence for each main outcome; consider their relevance to key groups (eg, health care providers, users, policymakers). Limitations 25 Discuss limitations at study and outcome level (eg, risk of bias) and at review level (eg, incomplete retrieval of identified research, reporting bias). Conclusions 26 Provide a general interpretation of the results in the context of other evidence, and implications for future research. Funding Funding 27 Describe sources of funding for the systematic review and other support (eg, supply of data) and the role of funders for the systematic review. No, this is not specifically addressed. Singh et al 2 used contingency tables to report pooled sensitivity and specificity and a random-effects model to calculate pooled likelihood ratios and diagnostic odds ratios. I 2 statistic was used to measure the statistical heterogeneity between studies. As discussed in answer 3b, the authors acknowledge that the small number of studies on this topic limits their ability to accurately measure the potential of publication bias. The authors did perform a subgroup analysis according to pulmonary embolism prevalence. This information is detailed a flow diagram (Figure 1 in the Singh et al 2 meta-analysis). This information is detailed in the appendix Table E1. No, this is not specifically addressed. These data are presented in a forest plot (Figure 2 in the Singh et al 2 meta-analysis). These data are presented in a forest plot (Figure 2 in the Singh et al 2 meta-analysis). No, this is not specifically addressed. The authors report the results of 1 subgroup analysis: PERC pooled specificity in studies with pulmonary embolism prevalence above and below 10%. The authors report the following summary: We found that when the pretest probability is low, PERC are highly sensitive in predicting pulmonary embolism, and D-dimer testing is thus unnecessary....ourmeta-analysis reports consistent high sensitivity and negative predictive value of PERC, with missed pulmonary embolism in just 0.5% of patients. The authors acknowledge that the meta-analysis is limited by the small number of published studies available on this topic. The authors provide a concise conclusion that is supported by their results: our meta-analysis has demonstrated high sensitivity for the PERC rule and evidence that the rule can be used in settings of low pretest probability with confidence. The major limitation of PERC is its low but acceptable specificity. The authors do not identify a specific funding source. By Annals policy, all authors are required to disclose all commercial, financial, and other relationships in any way related to the subject of the article per ICMJE conflict of interest guidelines (see The authors stated that no such potential conflict of interest exists. Volume 60, NO. 6 : December 2012 Annals of Emergency Medicine 807

6 efficacy. When conducting a meta-analysis of studies evaluating the efficacy of this antibiotic, authors must decide how to address the differences in methodological quality between these 2 studies. One strategy is to exclude studies that do not satisfy a minimum quality threshold; this was the strategy used in the PERC meta-analysis by Singh et al. 2 Using this minimum threshold strategy, Study A would be excluded from the metaanalysis. Alternatively, data from all studies may be included in the meta-analysis but greater weight applied to data from studies with higher methodological quality. In this case, both Study A and Study B would be included the meta-analysis, but data from Study B would be considered more heavily. Using this weighting method, if the methodological quality of Study B were scored twice as highly as that of Study A, we would weigh the data produced by Study B twice as heavily as Study A when pooling data in the meta-analysis. Q1.d Among the 11 studies included in this meta-analysis, 3 were abstracts presented at national meetings, was a letter to the editor, 8 and 7 were peer-reviewed published articles. 3,9-14 What are the differences in the review and publication process for abstracts and letters to the editor compared with full peer-reviewed articles? Should data presented in abstracts and letters be considered equal to that presented in peer-reviewed articles? The peer review process varies among scientific journals; however, most original research articles first undergo a review by a journal editor, who decides whether to send the article for peer review or to reject it according to an editor s review. Peer review may include one or multiple additional reviewers with specific expertise in the topic covered by the article. Some journals, including Annals of Emergency Medicine, also have designated experts in study methodology and statistics who specifically evaluate study design and analytical strategies. Original articles accepted for publication therefore have been favorably reviewed by peer reviewers, a decision editor, and the journal s editor in chief. Although there is variability in the review process for letters at different journals, a typical letter to the editor in many journals is reviewed only by a section editor, who may or may not have specific expertise in the topics covered by the letter. (Note that some journals, such as Nature, publish letters that are rigorously peer reviewed and would be categorized as brief reports in many other journals.) Furthermore, letters to the editor typically have stringent word and reference count constraints that limit the authors ability to thoroughly describe and the readers ability to evaluate the study. Published abstracts are generally reviewed by a panel of experts in a particular focus area who are tasked with reviewing numerous submissions and scoring the abstracts according to specific criteria. Similar to letters to the editor, abstracts are limited by specific word or character counts that may lead to omission of important data and limitations. One should cautiously interpret research that is published only as an abstract without a subsequent article. Investigators may include preliminary data, overstate their results or inflate their conclusions in an abstract to increase their likelihood for acceptance. 25 Because of the less rigorous review process and greater restriction on word counts, letters to the editor and abstracts should typically not be considered equal to published full-length articles. Authors and readers of meta-analyses should carefully consider the implications of including data from these publication types. Q1.e Six of the 11 studies included in the meta-analysis were authored by at least 1 researcher involved in the original derivation of PERC. These 6 studies 3,5,6,10,12,14 included 79% of the total patients included in the meta-analysis, indicating that most of the research was conducted by a few investigators. Is this a limitation to the meta-analysis? Ideally, a meta-analysis pools data collected from highquality studies conducted by different investigators in different patient populations. Replication of results across investigator groups helps protect against research findings being biased by the enthusiasm of a small group of investigators promoting their own work. Since the original publication of PERC in 2004 by Kline et al, 3 members of the same research group have evaluated PERC in multiple settings. 5,6,10,12,14 Because these investigators and the EDs in which they work have extensive experience with PERC, findings from these research settings may not be generalizable to the general ED setting, in which providers have less expertise with PERC. Studies included in the meta-analysis conducted by other research groups found lower sensitivity for PERC. 8,11 As noted in the discussion by Singh et al, 2 lower sensitivity for PERC reported in these studies may be related to the higher prevalence of pulmonary embolism in European settings in which these studies were conducted compared with US EDs. However, additional data about PERC s accuracy in clinical settings not directly influenced by the PERC investigators would be useful. ANSWER 2 Q2. The authors reported that PERC had a pooled sensitivity and specificity for ruling out pulmonary embolism of 0.97 (95% confidence interval 0.96 to 0.98) and 0.23 (95% confidence interval 0.22 to 0.24), respectively. Q2.a How do you interpret the meaning of this pooled sensitivity and specificity when deciding whether to use the PERC rule in the evaluation of patients within the ED? PERC-positive subjects are those who do not fulfill all 8 lowrisk PERC criteria and therefore cannot be ruled out. PERCnegative subjects do fulfill all 8 low risk criteria and are ruled out for pulmonary embolism according to PERC. PERC sensitivity is the proportion of subjects with pulmonary embolism who are PERC positive (Table 2). A pooled sensitivity of 0.97 indicates that 97% of subjects with a pulmonary embolism were PERC positive and 3% were PERC negative; 3% of subjects with a pulmonary embolism were misclassified by PERC as ruled out. Specificity indicates the proportion of patients without a pulmonary embolism who were PERC negative. A pooled specificity of 0.23 indicates that 808 Annals of Emergency Medicine Volume 60, NO. 6 : December 2012

7 Table 2. Two-by-two contingency table for calculating PERC sensitivity and specificity.* Pulmonary Embolism Present Pulmonary Embolism Absent PERC Positive A B PERC Negative C D *PERC sensitivity A/(A C). PERC specificity D/(B D). only 23% of subjects without a pulmonary embolism were PERC negative; 23% of subjects without a pulmonary embolism could have avoided further pulmonary embolism testing according to PERC (Table 2). Negative predictive value can also be informative when evaluating clinical decision tools designed to define patients with a very low risk of disease. For PERC, the negative predictive value is the proportion of subjects who are PERC negative and do not have a pulmonary embolism. Negative predictive value can be calculated from the sensitivity, specificity, and prevalence (Figure 2). The overall prevalence of pulmonary embolism in studies included in the meta-analysis was The calculated negative predictive value is Therefore, this meta-analysis suggests that in an ED patient population with a 0.10 prevalence of pulmonary embolism, a negative PERC result corresponds to a 0.99 probability of the patient s not having a pulmonary embolism and 0.01 probability of having a pulmonary embolism. With a Bayesian strategy, the negative likelihood ratio can also be used to calculate the probability of pulmonary embolism when PERC is negative. Singh et al 2 reported a pooled PERC negative likelihood ratio of This approach requires consideration of a clinician s gestalt pretest probability of pulmonary embolism before applying PERC. For example, after performing a history and physical examination, an emergency physician considers a patient to have a 0.25 probability of pulmonary embolism. Then, the clinician applies PERC and finds it to be negative (all 8 lowrisk PERC criteria are fulfilled). The probability of pulmonary embolism given the negative PERC (post-perc probability) can then be calculated according to the equations in Figure 2. With an estimated pre-perc probability of 0.25, the probability of pulmonary embolism after a negative PERC result is (Figure 3). Q2.b In the abstract, the authors state that the specificity of PERC was low but acceptable. Do you agree this low specificity is acceptable? Pooled PERC specificity was 0.23 (95% confidence interval [CI] 0.22 to 0.24), indicating that only 23% of the subjects who did not have a pulmonary embolism were PERC negative. Figure 2. Calculation of PERC negative predictive value from data provided in Singh et al. Figure 3. Three step method of calculating the probability of pulmonary embolism after a negative PERC result, given a 0.25 pre-perc probability of pulmonary embolism and a 0.18 PERC negative predictive value. Because of this low specificity, PERC would be a poor test to diagnose ( rule-in ) pulmonary embolism. However, PERC was designed to identify patients at very low risk, for whom clinicians could potentially safely forgo further diagnostic testing for pulmonary embolism. PERC was designed to maximize sensitivity; that is, to reliably rule out pulmonary embolism. When clinical decision tools are built, there is usually a tradeoff between sensitivity and specificity. Altering the tool to increase specificity usually results in decreased sensitivity, and vice versa. For example, if PERC were modified by eliminating the criterion age younger than 50 years, substantially more patients would be PERC negative; this would increase the number of both true negatives and false negatives, with a corresponding increase in specificity and decrease in sensitivity. Because PERC is used to rule out pulmonary embolism, low specificity is acceptable. However, clinicians using PERC must understand that low specificity and a positive likelihood ratio near 1.0 indicate that a positive PERC result cannot be used to support a diagnosis of pulmonary embolism. Q2.c In the Editor s Capsule Summary, the editors concluded: This pooled analysis strongly corroborates the safety of using PERC to defer D-dimer testing. Given that 2% to 4% of patients with a pulmonary embolism are PERC negative, is this conclusion justified? When discussing your answer, consider the risk-benefit ratio of D-dimer testing in patients who are PERC negative. Pooled PERC sensitivity was 0.97 (95% CI 0.96 to 0.98), indicating that 3% of ED patients with a pulmonary embolism were PERC negative. Then how could a strategy for pulmonary embolism evaluation based on PERC be safe? PERC clearly does not identify all patients with a pulmonary embolism. However, finding the 3% of PERC-negative patients who do have a pulmonary embolism would likely require diagnostic testing in large numbers of very low-risk patients, and this extensive testing is likely to lead to greater net harm than failing to diagnose a pulmonary embolism in these 3% of PERC-negative patients. Understanding the relative safety of using PERC requires a comparison to the safety of alternative strategies to evaluate for pulmonary embolism. We will consider the alternative strategy of D-dimer testing, with a positive D-dimer result followed by a CT pulmonary angiogram (CT-PA) and positive CT-PAs leading to hospitalization and treatment with warfarin. According to a recent systematic review on the diagnostic accuracy of D-dimer for pulmonary embolism, the estimated Volume 60, NO. 6 : December 2012 Annals of Emergency Medicine 809

8 sensitivity and specificity of a rapid quantitative D-dimer assay are 0.95 (95% CI 0.83 to 1.0) and 0.39 (0.28 to 0.51), respectively. 26 Meanwhile, the sensitivity and specificity of CT- PA was 0.83 (95% CI 0.76 to 0.92) and 0.96 (95% CI 0.93 to 0.97), respectively, in the Prospective Investigation of Pulmonary Embolism Diagnosis II (PIOPED II) study, the largest study evaluating the accuracy of CT-PA for pulmonary embolism. 27 Using these point estimates of sensitivity and specificity for PERC, D-dimer, and CT-PA, we can compare the expected aggregate outcomes of using PERC to rule out pulmonary embolism versus testing PERC-negative patients with a D- dimer. We will use a hypothetic sample of 1,000 PERCnegative ED patients to illustrate; the key values for each calculation below are listed in brackets at the end of each sentence. Of 1,000 PERC-negative patients, 30 are expected to have a pulmonary embolism [1 PERC sensitivity 0.03]. Among these 30 PERC-negative patients with a pulmonary embolism, D-dimer is expected to be positive in 29 patients [D-dimer sensitivity 0.95]. A subsequent CT-PA in these 29 patients is expected to be positive in 24 patients [CT-PA sensitivity 0.83]. Therefore, the strategy of D-dimer testing 1,000 PERC-negative patients would lead to 24 patients with a pulmonary embolism receiving a correct diagnosis and being treated with warfarin. The diagnosis of pulmonary embolism would have been missed in these 24 patients if the PERC strategy were used. Of 1,000 PERC-negative patients, 970 are expect not to have a pulmonary embolism [PERC sensitivity 0.97]. Among these 970 patients who do not have a pulmonary embolism, D-dimer result is expected to be positive in 592 patients [1 Ddimer specificity 0.61]. If all 592 of these patients underwent CT-PA, 24 of them would be expected to have a false-positive CT-PA result [1 CT-PA specificity 0.04]. Therefore, the strategy of D-dimer testing 1,000 PERC-negative patients is expected to lead to 24 patients without pulmonary embolism receiving a false diagnosis of pulmonary embolism, being hospitalized, and beginning to receive warfarin. These 24 patients will be subjected to the risks of hospitalization and anticoagulation therapy 28 without any benefit. Anticoagulated patients will also experience morbidity related to lifestyle changes and routine international normalized ratio (INR) monitoring demanded by chronic warfarin use. Using the D- dimer strategy is expected to result in 621 patients undergoing CT-PA who would not have had a CT scan in the PERC strategy. In addition to diagnosing 24 pulmonary embolisms, the CT scans may be helpful in discovering alternative diagnoses responsible for the patient s symptoms, such as aortic dissection, cancer, or pneumonia. However, CT scanning has potential risks as well, including renal injury and allergic reactions from contrast dye exposure, extravasation of contrast into soft tissue structures, and radiation exposure. A high volume of CT scanning also inhibits patient flow through EDs, which are increasingly burdened with crowding. CT scans also greatly increase the cost of ED visits at a time when society is struggling to curtail increasing health care costs. In summary, in a population of 1,000 PERC-negative patients, ruling out pulmonary embolism according to PERC is expected to miss 30 pulmonary embolisms. If a D-dimer test were obtained for all 1,000 pulmonary embolism patients, 24 pulmonary embolisms would be correctly diagnosed, 6 pulmonary embolisms would still be missed, and 24 patients who did not have a pulmonary embolism would receive a false diagnosis of a pulmonary embolism. The D-dimer strategy is associated with enormous CT-PA use, with the associated risks of exposure to radiation and intravenous contrast dye. Furthermore, as discussed in question 4 below, the benefits of anticoagulation treatment for patients with small pulmonary embolisms that do not result in physiologic compromise have been questioned. Many of the PERC-negative patients with pulmonary embolism are likely to fall into this category of no physiologic compromise. Using a similar risk-benefit analysis, Newman and Schriger 17 concluded that a strategy of liberal D-dimer testing for pulmonary embolism results in 6 times as many deaths from anticoagulation-related hemorrhage, contrastassociated renal failure, and radiation-induced cancer than lives saved through the prevention of fatal pulmonary embolisms through anticoagulation therapy. Compared with the most feasible alternative strategy of D-dimer testing, we agree that using PERC to rule out pulmonary embolism in the ED is safe. Q2.d Aujesky et al 15 published the results of a randomized, noninferiority trial in Lancet suggesting that outpatient treatment is safe and effective in selected, low-risk patients with pulmonary embolism. Additionally, a recent trial published in the New England Journal of Medicine suggested that the oral anticoagulant rivaroxaban resulted in fewer major bleeding complications and similar efficacy for pulmonary embolism treatment compared with traditional anticoagulation (enoxaparin bridged to warfarin). 16 Using these examples, discuss how safer management options for pulmonary embolism may change the risk-benefit ratio for D-dimer testing among patients who are PERC negative. Hospitalization is associated with numerous risks for patients, including nosocomial infections, medical error, and delirium. Aujesky et al 15 suggested that initial outpatient management for low-risk patients with pulmonary embolism is as effective as inpatient management. To avoid the costs and patient risks of hospitalization, outpatient management for lowrisk pulmonary embolism may become standard. Most PERCnegative patients with a pulmonary embolism are likely to fall into this low-risk category for outpatient management. Using the hypothetical example introduced above in answer 2c, 48 of 1,000 PERC-negative patients would receive a diagnosis of a pulmonary embolism with the D-dimer strategy; 24 of these patients would have a pulmonary embolism (true positive), and 24 would not have pulmonary embolism (false positive). If these 48 patients could be managed without hospitalization, the D- dimer strategy would become safer because the risks of hospitalization would be avoided. However, this change is 810 Annals of Emergency Medicine Volume 60, NO. 6 : December 2012

9 unlikely to significantly alter the relative safety of the PERC and D-dimer strategies because hospitalization risks are relatively minor compared with the risks of warfarin use and CT-PA use associated with the D-dimer strategy. The Food and Drug Administration recently approved multiple novel anticoagulants that are advertised as safer alternatives to warfarin. The Einstein-pulmonary embolism investigators 16 reported that rivaroxaban was noninferior to standard enoxaparin and warfarin therapy in preventing symptomatic recurrent venous thromboembolism after an initial pulmonary embolism. Major bleeding was observed in 1.1% of subjects treated with rivaroxaban compared with 2.2% of those treated with standard therapy. If new anticoagulants do indeed have the same efficacy as warfarin, with a lower incidence of major hemorrhage, the risk:benefit ratio for testing PERCnegative patients with a D-dimer may become more favorable. However, at this point, it is unclear whether benefits from reduced major hemorrhage risk with the new anticoagulants will be offset by the lack of a true reversal agent, or other, currently unrecognized toxicities. Furthermore, the questionable benefit of anticoagulation in hemodynamically normal pulmonary embolism patients and risks of CT-PA will likely maintain the relative safety of the PERC strategy over the D-dimer strategy despite improvements in the safety of anticoagulation therapy. ANSWER 3 Q3. Results of individual studies included in the meta-analysis are shown in Figure 2 of the Singh et al 2 article. Q3.a What is the name of this type of plot and why is it commonly used in meta-analyses? What is the significance of the different sizes of the dots in Figure 2? Figure 2 in the Singh et al 2 meta-analysis contains forest plots, which are frequently used to display data from individual studies compiled in a meta-analysis. 29 By reviewing a forest plot, readers can quickly understand the how data from different studies contributed to the overall conclusions and the level of variation among studies. Therefore, inclusion of forest plots in published meta-analyses has become nearly universal in the past decade. We will discuss the PERC sensitivity forest plot (top left plot in Figure 2 of Singh et al 2 ) as an example. PERC sensitivity is plotted along the x axis. The y axis lists each study included in the sensitivity meta-analysis. Dots represent the point estimate for PERC sensitivity in each study. The size of each dot is proportional to the study s sample size. The study by Kline et al 12 listed sixth from the top (denoted with **) had the largest sample size, with 8,138 subjects; thus, this study has the largest dot on the forest plot. The horizontal lines running through each dot represent 95% CIs for sensitivity in each study. The pooled sensitivity from all studies is denoted by the diamond at the bottom of the y axis. The center of the diamond represents the pooled point estimate for sensitivity (0.97), whereas the width of the diamond represents the 95% CI for pooled sensitivity (0.96 to 0.98). Horizontal lines are extended upward from the pooled CI (from the edges of the diamond) to illustrate how data from individual studies compare with the pooled data. These vertical lines can resemble the appearance of multiple trees in a forest, which led to the name forest plot. Q3.b In the Limitations section, the authors observed that they did not evaluate for the possibility of publication bias because the meta-analysis included fewer than 20 studies. What is publication bias? Why does a small number of studies included in a metaanalysis preclude assessing for publication bias? The goal of meta-analysis is to pool data from all highquality studies addressing the same research question. Similar to the meta-analysis by Singh et al, 2 many medical meta-analyses consider only published studies. However, some high-quality studies on a particular research question may not have been published. This is problematic for a meta-analysis of published literature if the unpublished studies are systematically different from those that are published. Writers of meta-analyses have found that studies demonstrating a statistically significant effect ( positive studies ) are frequently easier to find and more likely to be published than studies with null results ( negative studies ). 30 This positive outcome bias is the most common form of publication bias. The causes of such bias likely stem from investigators and funders more aggressively seeking publication and editors being more receptive to studies that report large positive findings. Therefore, a meta-analysis of published literature may overrepresent studies with statistically significant findings and overestimate treatment effect. In the context of clinical decision rules for pulmonary embolism, a study describing a clinical decision rule that accurately rules out pulmonary embolism may be more likely to be published than a study describing a rule that does not rule out pulmonary embolism with high accuracy. As discussed in answer 3c below, a common method to assess for potential publication bias is the funnel plot. The association between estimates for treatment effect and the standard error of those estimates for treatment effect are graphed for each study included in a meta-analysis on a funnel plot. The shape of a funnel plot can suggest the presence or absence of publication bias (see answer 3c and Figures 4 and 5). To supplement the subjective, visual evaluation of a funnel plot, 2 statistical tests a rank correlation test (Begg s method) 31 and a regression test (Egger s method) 32 can be used to analyze funnel plot shape. The power of these statistical methods to detect evidence of publication bias declines with decreasing number of studies included in a meta-analysis. Statistical techniques have little power to detect publication bias in meta-analyses with fewer than 10 to 20 studies. 33 Therefore, although publication bias may be present, these small meta-analyses often do not include statistical techniques to evaluate for publication bias. Q3.c Consider a hypothetical meta-analysis evaluating the efficacy of the herbal therapy nocturno-doc to maintain alertness of emergency physicians during night shifts. Eight years after release of nocturno-doc onto the market, 10 studies reporting efficacy were published; a meta-analysis of these studies demonstrated high efficacy for nocturno-doc. A funnel plot of Volume 60, NO. 6 : December 2012 Annals of Emergency Medicine 811

10 Figure 4. Funnel plot of hypothetical published and unpublished studies evaluating efficacy of nocturno-doc treatment. The symmetrical shape of this plot around the pooled log RR of 0 (dashed line) is suggestive of the absence of publication bias. Figure 5. Funnel plot of hypothetical published studies evaluating efficacy of noturno-doc treatment. This plot is suggestive of possible publication bias because of its asymmetrical shape from the lack of studies to the left of the pooled log RR of 0.75 (dashed line). these studies is displayed in the Figure 1. Discuss how publication bias may have affected the results of this hypothetical meta-analysis of nocturno-doc efficacy. The funnel plot is a method to assess for the possibility of publication bias. Funnel plots, such as those displayed in Figures 4 and 5, graph individual studies included in a metaanalysis, with a measure of treatment effect on the x axis and a measure of the precision of that treatment effect (ie, standard error, sample size) on the y axis. In the hypothetical example here, studies evaluating the efficacy of nocturno-doc reported the relative risk (RR) for alertness after nocturnodoc is received compared with placebo. The log RR of each study is plotted on the x axis, with log RR greater than 0 indicating nocturno-doc superiority over placebo and log RR less than 0 indicating inferiority to placebo. The standard errors for log RR estimates are plotted on the y axis, with greater precision (lower standard error) positioned higher on the axis. From a statistical perspective, we expect studies with less precision to vary more widely from the pooled RR than studies with greater precision. Figure 4 includes the 10 unpublished and 10 published studies of nocturno-doc efficacy. As displayed here, in the absence of publication bias, we expect the data points to form a symmetrical funnel, or pyramid, around the pooled log RR, with less precise studies at the bottom splayed widely and more precise studies at the top tightly clustering around the pooled estimate (Figure 4). The pooled log RR using all 20 published and unpublished studies is 0 (no treatment effect). The funnel plot in Figure 5 contains only the 10 published studies. The pooled log RR limited to these published studies is 0.75 (favors nocturno-doc). The funnel plot of published studies (Figure 5) loses the left half of the funnel plot in Figure 4 because the 10 studies showing no advantage for nocturno-doc were never published and therefore were not included in the meta-analysis. Although there are other possible explanations for an asymmetrical funnel plot, 34,35 publication bias must be considered when a plot is found to be asymmetrical. ANSWER 4 Q4. In an Annals of Emergency Medicine editorial, Newman and Schriger 17 suggested that diagnosing a small pulmonary embolism on computed tomography scan in a patient without physiologic compromise may not be beneficial. They argued that small pulmonary emboli without physiologic strain appear to be common and nearly universally nonfatal. 18 Furthermore, they observed that anticoagulation is routinely prescribed to treat small pulmonary emboli, but the harm of anticoagulation may outweigh the benefit in this setting. 19 Q4.a PERC was developed to maximize sensitivity for the diagnosis of any pulmonary embolism. Would a pulmonary embolism clinical decision tool based on a different, patientcentered outcome, such as fatal pulmonary embolism or pulmonary embolism resulting in long-term disability, be better? Defining the specific outcome of interest is one of the first steps in developing a clinical prediction model and is critical not only for proper derivation and validation of the decision tool but also for its potential effect on patient care. Because most disease processes have a spectrum of severity (eg, stable angina to unstable angina to acute myocardial infarction), investigators must decide what level of disease severity will define their outcome. Traditionally, pulmonary embolism clinical decision rules, including PERC, have been designed to identify all patients pulmonary embolism according to the rationale that any pulmonary embolism is potentially life threatening and should not be missed during an ED evaluation. However, 812 Annals of Emergency Medicine Volume 60, NO. 6 : December 2012