What Is Evidence-Based Medicine? 1 Critical Thinking Skills Symposium

Special Report Radiology Alliance for Health Services Research What Is Evidence-Based Medicine? 1 Critical Thinking Skills Symposium Kelly H. Zou, PhD, Julia R. Fielding, MD, Silvia Ondategui-Parra, MD, MPH, MSc Rationale and Objectives. In this review article, we present the definition and useful concepts of evidence-based medicine (EBM). The principles of EBM are provided and major steps of practicing EBM are described. Materials and Methods. We emphasize the importance of the Cochrane Collaboration (see http://www.cochrane.org), which initiated the research and practice in this area. Because it can be difficult to systematically access and review individual research studies, it is often useful to focus on a critical overview of clinical trials by conducting a meta-analysis. Results. Useful literature and resources related to meta-analysis are provided. Conclusion. Statistical methods for evaluating radiologic diagnostic performances derived from meta-analysis are summarized, with a special focus on summary outcomes measures. Key Words. Evidence-based medicine; evidence-based radiology; meta-analysis; sensitivity; specificity; receiver operating characteristic curve. AUR, 2004 Acad Radiol 2004; 11:127 133 1 From the Radiology Management Group, Department of Radiology, Brigham and Women s Hospital, Harvard Medical School, Boston, MA; Department of Health Care Policy, Harvard Medical School, Boston, MA; Department of Radiology, University of North Carolina at Chapel Hill, Chapel Hill, NC; Department of Health Policy and Management, Harvard School of Publish Health, Boston, MA. Received July 14, 2003; revision requested August 28; received in revised form September 8; accepted September 9. Partially supported by a research grant R03HS13234-01 from the National Institutes of Health. Address correspondence to Kelly H. Zou e-mail: zou@bwh.harvard.edu AUR, 2004 doi:10.1016/s1076-6332(03)00650-0 Evidence-based medicine (EBM), also called evidencebased health care, was developed because of the awareness of the limitations of traditional determinants. The principles of EBM offer a useful solution to clinical problems to acquire valid and current information for clinical and policy decisions (1). EBM is defined as the process of systematically finding, critically appraising, and using contemporary research published in the medical literature as a basis for making decisions regarding individual patient care and health care policy. It is also a habit that is recommended to all practicing radiologists so that they use the best logical thinking techniques to derive treatment plans (2 4). A comprehensive review of EBM applied to radiology was recently published (1). However, so far these developments have received limited applications in radiology. The link between EBM and evidence-based radiology is aimed at integration of evaluative sciences and technology assessment into clinical practice (2). There are several approaches for EBM. It is recognized that one of the best methods for reviewing a field of information and to systematically access individual research studies is to conduct a meta-analysis. It is important for radiologists to realize that EBM offers solutions that can be applied at many levels of professional involvement. A unique feature of EBM is that it can be used readily by practicing radiologists working at the effectiveness level: performance in their own departments under ordinary clinical practice (1). 127

ZOU ET AL Academic Radiology, Vol 11, No 2, February 2004 Table 1 Trends in Evidence-based Medicine From 1995 to 2002 Using MEDLINE Search with Subject Heading Words Evidencebased Medicine Year Number of Publications Relative Frequency (%) 1995 20 0.2 1996 215 2.0 1997 678 6.2 1998 1075 9.8 1999 1611 14.7 2000 1889 17.2 2001 2573 23.4 2002 2919 26.6 Total 10980 100.0 First, in this article, we will focus on how to integrate EBM techniques into daily clinical practice. We will then review the basic steps required to perform a meta-analysis. Finally, a summary and concluding remarks are provided. EVIDENCE-BASED MEDICINE IN RADIOLOGY RADIOLOGY The number of articles published on EBM has seen a dramatic increase. The trends of conducting EBM between 1995 and 2002, based on a MEDLINE search using the subject heading evidence-based medicine, are provided (Table 1). The major work of evidence-based medicine is to draw conclusions from results collected from literature, and alternatively, from narrative reviews and data pooled from independent studies often clinical trials. The main methodologic approach is to perform a systematic statistical analysis to extract, compare, and combine the reported results from these studies to derive quantifiable outcomes (2). Explicit criteria are necessary to render an objective assessment of the study s value. Such criteria include inclusion and exclusion subject criteria with assessment of bias, reasonableness of the clinical intervention, appropriateness of the statistical evaluation, and overall credibility when compared with a practitioner s medical knowledge. There are four major steps to the practice of evidencebased medicine (4), as described in the following sections. Step 1. Formulate the Clinical Question This first step is the single most important one, and requires careful thought. It is best to formulate the clinical question in the form of an exposure and patient outcome. For example, Does the use of contrast-enhanced computed tomography (CT) scan of the chest (exposure/ possible harm) as a primary test for dyspnea in a 68-yearold woman with a normal chest radiograph reduce her risk of dying of pulmonary embolus? The more detail incorporated into the clinical question, the more specific is the relevant literature review. In this example, the patient s age, symptoms, and chest radiograph findings are included. Thus, in the literature review, one could disregard articles describing young patients, asymptomatic patients, and those patients with abnormal chest radiographs. Step 2. Find the Evidence Conventionally, radiologists may search for useful evidence for clinical practice without applying appropriate or scientific methods. For example, one may discuss a case with a colleague or mentor, retrieve individual research articles, and review quality and relevance-filtered publications. The limitation in such practice is that the expert opinion can be out of date and even incorrect, particularly when confronted with a very rare disease or unusual manifestation of a more common ailment. Alternatively, many radiologists keep paper or electronic files of individual research articles, usually divided by organ system. These files, when critically reviewed and kept up to date, may help understand the current status of imaging a particular disease. Unfortunately, the clinical information from these personal files typically does not represent that using a population-based approach, which is relevant to the individual case to be diagnosed or treated. Therefore, sophisticated and robust methods are needed for systematically combining evidence. Computer databases such as MEDLINE and PUBMED have shown to be invaluable in these cases. The Cochrane Collaboration, based in the United Kingdom, recommends performing meta-analyses rather than the ad hoc combination of information present in the literature. Founded by Dr. Archie Cochrane in 1972, the goal of the group is to determine, in well-designed studies, which therapies are effective and then to use health care resources to provide these therapies to the population in an equitable and presumably cost-effective manner. In this group, reviewers, consumers, translators, and hand searchers submit work to an editorial team assigned to a particular health care project. Using predetermined eligibility criteria, studies are selected to provide the largest dataset possible. A meta-analysis is then performed on these studies. The results of the analysis form the basis of a report that 128

Academic Radiology, Vol 11, No 2, February 2004 WHAT IS EVIDENCE-BASED MEDICINE? makes recommendations on the usefulness of a particular test or therapy. In the United States, for example, technology assessment is reviewed by the Quality Interagency Coordination Task Force based in Washington, DC. Step 3. Critical Appraisal When reviewing an individual research study or conducting a meta-analysis, one must assess the work for relevance and methodologic rigor. This is a critical step for developing epidemiologic tools to assess the validity and the quality of the evidence found in the literature. There are different levels of evidence and grades for recommendation. The medical literature may be classified according to its quality level ranging from type (1), the highest quality, to type (5) the lowest. Type (1) evidence is from a systematic review, which includes at least one randomized controlled trial and a summary of all included studies. Examples of these studies include those published by the Cochrane Collaboration, the National Health Service Centre for Reviews and Dissemination, and the Agency for Healthcare Research and Quality. The evidence from such a review requires careful appraisal; if well done, such evidence is powerful. Type (2) evidence is from a properly designed randomized controlled trial of appropriate size. Type (3) evidence is from a well-designed intervention study without randomization. Evidence in this category will only be included if no type (1) or (2) evidence is available. A common research design is the before-and-after study. Type (4) evidence is supplied from a well-designed nonexperimental study (eg, cohort, case-control or cross-sectional study and any study using purely qualitative methods). Studies in this category will only be included if no type (1), (2), or (3) evidence is available. Economic analyses (cost-effectiveness studies) are also classified as type (4) evidence. Type (5) evidence consists of opinions of respected authorities based on clinical evidence, descriptive studies, or reports of expert consensus committees. Step 4. Develop Solutions Research results, even those that are carefully reviewed, should not be used to determine patient treatment. Instead, the practitioner should combine the information gathered from literature reviews with his or her clinical expertise and available external evidence. In most cases, this evidence will consist of the patient s history and physical examination and laboratory test results. In this way, the best diagnostic and therapeutic options available can be matched to a specific patient s condition. Table 2 Trends in Meta-analysis Medicine From 1995 to 2002 Using MEDLINE Search with Subject Heading Words Metaanalysis Year Number of Publications Relative Frequency (%) 1995 274 8.6 1996 261 8.2 1997 331 10.4 1998 350 11.0 1999 397 12.5 2000 491 15.4 2001 544 17.1 2002 534 16.8 Total 3182 100.0 META-ANALYSIS Definition and Trends Meta-analysis is a quantitative method for combining the results of independent studies, usually drawn from published literature, and for synthesizing summaries and conclusions, which may be used to evaluate therapeutic effectiveness and plan new studies (5,6). It has increasingly been used for evaluating and comparing diagnostic performances, for example, of imaging modalities (7 9), biopsy techniques (10), and vascular interventions (11). The number of papers published on meta-analyses has increased steadily in the past 10 years. See the trends of conducting meta-analysis between 1995 and 2002 in the results of a MEDLINE search using the subject heading of meta-analysis (Table 2). Advantages For the purpose of critically evaluating a clinical hypothesis based on published clinical trials, meta-analysis is an efficient tool for summarizing the results in the literature numerically. Meta-analysis allows for an objective appraisal of the evidence, which may lead to resolution of uncertainty and disagreement. It can reduce the probability of false-negative results and thus prevent undue delays in the introduction of effective treatments into clinical practice. A priori hypotheses regarding treatment effects in subgroups of patients may be tested. It may also explore and sometimes explain the heterogeneity between study results. Finally, the analysis may help guide the design of future research. Specifically, the sample size needed in future studies may be calculated more accurately. Classic textbooks on meta-analysis have been writ- 129

ZOU ET AL Academic Radiology, Vol 11, No 2, February 2004 ten by several authors (12 15). In the late 1990s, the British Journal of Medicine published a series of articles on this topic (17 23). Tools There are several epidemiologic and statistical tools required to scientifically synthesize and assemble literature data in a meta-analysis: (1) a carefully considered and detailed protocol should be written before beginning the project; (2) an a priori definition of eligibility criteria should be included, with a comprehensive search for such studies as a central part of the work; (3) the results should be graphed on a common scale to allow a visual examination of the heterogeneity between studies; (4) an appropriate statistical method should be chosen for combining data; and (5) a thorough sensitivity analysis should be performed to assess the robustness of combined estimates using different assumptions and inclusion criteria. In Table 3, we illustrate these five tools on a wellconducted meta-analysis by Oei et al. (9). The authors evaluated the diagnostic performance of magnetic resonance imaging of the menisci and cruciate ligaments of the knee and assessed the effect of study design characteristics and magnetic field strength on diagnostic performance. Based on 29 of the 120 retrieved articles, the authors found that the performance of magnetic resonance imaging differed according to lesion types and was influenced by study designs. In addition, higher magnetic field strength moderately improved diagnostic accuracy, with a significant effect on the identification of anterior cruciate ligament tears. Sources of Biases We have already defined the classification system of quality of the literature under EBM. However, there are several biases that may lead to an erroneous conclusion. The most common types of bias include: (1) publication bias: significant results are more likely to get published; (2) language and citation bias: among published studies, those with significant results are more likely to get published in English, to be cited, and to be published repeatedly; (3) database bias: in less developed countries, studies with significant results may be more likely to get published in a journal indexed in a literature database; and (4) inclusion bias: criteria for including studies in a metaanalysis may be influenced by knowledge of the results of the set of potential studies. To minimize these possible biases, one must search the world literature thoroughly Table 3 A Step-by-Step Illustration of a Toolbox for Meta-analysis on a Published Study on MRI of the Menisci and Cruciate Ligaments Step Tools Methods Applied 1. Detailed protocol 2. Eligibility criteria 3. Graphical display 4. Statistical methods 5. Sensitivity analysis Search engine: MEDLINE Date: January 1991 December 2000 Purpose: Diagnostic performance of MRI of knee lesions Terms: Magnetic resonance imaging, knee, meniscus, cruciate ligament, arthroscopy Extractors: Two independent readers, with third reader assessing discrepancies Study characteristics: Publication year, country, setting, patient characteristics, aspects of study design, verification bias, characteristics of MRI Likelihood for inclusion: Yes, possible, no (1) Inclusion criteria: Language: English Diagnosis: MRI of lesions of medial or lateral meniscus, ACL, or PCL Sample size: At least 30 subjects Gold standard: Arthroscopy Measurement: Magnetic field strength Threshold: Positive criteria for MRI Outcome data: Absolute numbers of TP, FN, TN, and FP (2) Exclusion: Patient population: Consists of infants or adolescents Study objective: MRI for postoperative evaluation Study design: Case-control Ligaments: Only the medial and lateral meniscus combined Measurement: Various magnetic field strengths Outcome data: Only the diagnostic value of specific features and indirect signs of knee lesions at MRI Funnel plot of log odds ratio Summary ROC plot Pooled weighted analysis of sensitivity and specificity Random effects model Summary ROC analysis per lesion and for all lesions Delete-one Jackknife method Note. MRI magnetic resonance imaging; ACL anterior cruciate ligament; PCL posterior cruciate ligament; TP true positive; FN false negative; TN true negative; FP false positive; ROC receiver operating characteristic. 130

Academic Radiology, Vol 11, No 2, February 2004 WHAT IS EVIDENCE-BASED MEDICINE? Table 4 Summary Measures Commonly Used for Conducting Meta-analysis in Radiology Summary Measure Sensitivity (true positive rate) Specificity (true negative rate) Positive predictive value Negative predictive value Likelihood ratio Odds ratio Relative risk Receiver operating curve Definition The proportion of subjects with disease who have a positive test. The proportion of subjects without disease who have a negative test The proportion of test positive subjects who truly have disease. The proportion of test negative subjects who truly do not have disease. The probability that a subject with disease would have a particular test result divided by the probability that a subject without the disease would have that result. The probability of the disease occurring divided by the probability that it doesn t occur. The probability of the disease in the risk group divided by the probability of the disease in the control group. A plot of (1-specificity, sensitivity) at all possible decision threshold. Table 5 Absolute Diagnostic Rates in the Computed Tomography of Metastasis in Lung Cancer(35) Study Sample Size (n) True Positive (TP) False Negative (FN) False Positive (FP) True Negative (TN) Sensitivity Specificity 1 50 11 7 6 26 0.611 0.813 2* 35 2 5 15 13 0.287 0.464 3 51 15 2 2 32 0.833 0.941 4 22 4 1 4 13 0.800 0.765 5 50 22 21 1 6 0.512 0.857 6 42 17 1 9 15 0.944 0.625 7 94 29 10 1 54 0.744 0.982 8 41 8 6 4 23 0.571 0.852 9 50 7 6 12 25 0.538 0.676 10 49 20 1 10 18 0.952 0.643 11 48 19 1 9 19 0.950 0.679 12 97 18 6 8 65 0.750 0.890 13 41 18 1 9 13 0.947 0.591 14 75 17 3 6 49 0.850 0.891 Note *Data from Study 2 were not included in the analysis because of inhomogeneity. Sensitivity TP/(TP FN) and Specificity TN/(FP TN) and use eligibility criteria stringently (24 26). A tutorial article addressing the statistical methods for meta-analysis including reduction of bias was published in 1999 by Normand (27). Diagnostic Imaging: A Receiver Operating Characteristic Curve There are several summary outcomes derived from meta-analyses useful for imaging research (28 30). These include sensitivity, specificity, predictive value, likelihood ratio, odds ratio, relative risk, and summary receiver operating characteristic (SROC) curve (Table 4). SROC methods are designed to show the accuracy of a specific test at predetermined sensitivity and specificity levels. There are several methods to derive such a curve. Kardaun and Kardaun (31) used a bivariate normal maximum-likelihood method. Littenberg et al. (32) employed a logit difference sum regression model, in which the logit-transformed true-positive fraction and false-positive fraction have a linear relationship; logit(p) ln{p/(1 p)}. Recently, the logit difference sum method was validated via statistical simulations (33). A more complicated approach using a latent-scale logistic regression analysis model was developed by Rutter and Gatsonis (34). As a simple example for derivation of the summary ROC curve, Table 5 provides the subset data from 14 131

ZOU ET AL Academic Radiology, Vol 11, No 2, February 2004 Figure 1. Summary Receiver Operating Curves of Diagnostic Rates in the Computed Tomography of Metastasis in Lung Cancer by Two Estimation Methods. studies found by Inouye and Sox (35) and analyzed by both Kardaun and Kardaun and by Litternberg et al. The intention of this meta-analysis was to evaluate the diagnostic accuracy of CT scans for detecting metastases in patients with non small-cell lung cancer. Summary ROC curves were constructed based on 13 studies. Data from study 2 were omitted because the sensitivity and specificity were less than 0.5, which was not homogeneous with remaining data. The two analysis models yielded the summary ROC curves, both displayed in Figure 1. The logit difference sum model gave the estimated regression equation logit(tp) 3.26 1.12 logit(fp), whereas the bivariate normal maximum-likelihood model gave the estimated equation logit(tp) 3.71 1.64 logit(fp). The large area under the ROC curves showed that CT was accurate in diagnosis of metastatic lung nodes. SUMMARY In this brief review article, we present the concepts and steps for EBM and meta-analysis, with a focus on evidence-based radiology. Careful scientific methodology is called for to minimize biases and to systematically synthesize medical literature. The application of EBM principles to diagnostic imaging can facilitate the interpretation of imaging studies and create a well-conducted radiologic evaluation. REFERENCES 1. The Evidence-Based Radiology Working Group. Evidence-based radiology: a new approach to the practice of radiology. Radiology 2001; 220:566 575. 2. Eisenberg JM. Then lesions for evidence-based technology assessment. JAMA 1999; 282:1865 1869. 3. Hoffrage U, Lindsey S, Hertwig R, et al. Medicine communicating statistical information. Science 2000; 290:2261 2262. 4. Sackett DL, Straus SE, Richardson S, et al: Evidence-based medicinehow to practice and teach EBM. 2nd ed. Edinburgh, Scotland: Churchill Livingstone, 2000. 5. Jaeschke R, Guyatt G, Sackett DLfor the Evidence-Based Medicine Working Group. Users guides to the medical literature, III. How to use an article about a diagnostic test. B. What are the results and will they help me in caring for my patients? JAMA 1994; 271:703 707. 6. Langlotz CP, Sonnad SS. Meta-analysis of diagnostic procedures: A brief overview. Academic Radiology 1998; 5(Suppl. 2):S269 S273. 132

Academic Radiology, Vol 11, No 2, February 2004 WHAT IS EVIDENCE-BASED MEDICINE? 7. Visser K, Hunink MG. Peripheral arterial disease: A gadolinium-enhanced MR angiography versus color-guided duplex US meta-analysis. Radiology 2000; 216:67 77. 8. Kinkel K, Lu Y, Both M, et al. Detection of hepatic metastases from cancers of the gastrointestinal tract by using noninvasive imaging methods (US, CT, MR imaging, PET): A meta-analysis. Radiology 2002; 224:748 756. 9. Oei EHG, Nikken JJ, Verstignen ACM, et al. MR imaging of the menisci and cruciate ligaments: A systematic review. Radiology 2003; 226:837 848. 10. Samson DJ, Flamm CR, Pisano ED, et al. Should FDG PET be used to decide whether a patient with an abnormal mammogram or breast finding at physical examination should undergo biopsy? Acad Radiol 2002; 9:773 783. 11. MacEneaney PM, Malone DE. Applying evidence-based medicine theory to interventional radiology. II. A spreadsheet for swift assessment of procedural benefit and harm. Clin Radiol 2000; 55:938 945. 12. Hedges LV, Olkin I. Statistical methods for meta-analysis. New York: Academic Press, 1985. 13. Eddy DM, Hasselblad V, Shachter R. Meta-analysis by the confidence profile method. The statistical synthesis of evidence. Boston: Academic Press, 1992. 14. Petitti DB. Meta-analysis, decision analysis, and cost-effectiveness analysis. Methods for quantitative synthesis in medicine. New York: Oxford University Press, 1994. 15. Cooper H, Hedges LV. The handbook of research synthesis. New York: Russell Sage Foundation, 1994. 16. Rosenberg W, Donald A. Evidence based medicine: An approach to clinical problem-solving. BMJ 1995; 310:1122 1126. 17. Sackett DL, Rosenberg WM, Gray JA, et al. Evidence based medicine: What it is and what it isn t. BMJ 1996; 312:71 72. 18. Egger M, Smith DG. Meta-analysis: Potential and promise. BMJ 1997; 315:1371 1374. 19. Egger M, Davey Smith G. Meta-analysis: Principles and procedure. BMJ 1997; 315:1533 1537. 20. Smith DG, Egger M, Phillips AN. Meta-analysis: Beyond the grand mean. BMJ 1997; 315:1610 1614. 21. Egger M, Smith DG. Meta-analysis: Bias in location and selection studies. BMJ 1998; 316:61 66. 22. Egger M, Schneider M, Smith DG. Spurious precision? Meta-analysis of observational studies. BMJ 1998; 316:140 144. 23. Smith DG, Egger M. Meta-analysis: Unresolved issues and future developments. BMJ 1998; 316:221 225. 24. Lyengar S, Greenhouse J. Selection models and the file-drawer problem. Stat Sci 1988; 3:109 135. 25. Dear KB, Begg CB. An approach for assessing publication bias prior to performing a meta-analysis. Stat Sci 1992; 7:237 245. 26. Hedges LV. Modeling publication selection effects in meta-analysis. Stat Sci 1992; 7:246 255. 27. Normand S-LT. Meta-analysis: Formulating, evaluating, combining, and reporting. Stat Med 1999; 18:321 359. 28. Rockette HE, Gur D, Campbell WL, et al. Use of meta-analysis in the evaluation of imaging systems. Acad Radiol 1994; 1:63 69. 29. Walter SD, Jadad AR. Meta-analysis of screening data: A survey of the literature. Stat Med 1999; 18:3409 3424. 30. Irwig L, Macaskill P, Glasziou P, et al. Meta-analytic methods for diagnostic test accuracy. J Clin Epidemiol 1995; 48:119 130. 31. Kardaun JWPF, Kardaun OJWF. Comparative diagnostic performance of three radiological procedures for detection of lumbar disk herniation. Meth Info Med 1990; 29:12 22. 32. Moses LE, Shapiro D, Littenberg B. Combining independent studies of a diagnostic test into a summary ROC curve: Data-analytic approaches and some additional considerations. Stat Med 1993; 12:1293 1316. 33. Mitchell MD. Validation of the summary ROC for diagnostic test metaanalysis: A monte-carlo simulation. Acad Radiol 2003; 10:25 31. 34. Rutter CM, Gatsonis CA. Regression methods for meta-analysis of diagnostic test data. Acad Radiol 1995; 2(Suppl. 1):S48 S56. 35. Inouye SK, Sox HC. Standard and computed tomography in the evaluation of neoplasms of the chest. Ann Int Med 1986; 105:906 924. 133