Studies reporting ROC curves of diagnostic and prediction data can be incorporated into meta-analyses using corresponding odds ratios
|
|
- Jeffrey Kelley
- 6 years ago
- Views:
Transcription
1 Journal of Clinical Epidemiology 60 (2007) 530e534 BRIEF REPORT Studies reporting ROC curves of diagnostic and prediction data can be incorporated into meta-analyses using corresponding odds ratios S.D. Walter a, *, T. Sinuff b a Department of Clinical Epidemiology and Biostatistics, McMaster University, 1200 Main Street West, HSC-2C16, Hamilton, Ontario, Canada L8N 3Z5 b Department of Critical Care Medicine, Sunnybrook Health Sciences Centre and Interdepartmental Division of Critical Care, University of Toronto, Toronto, Ontario, Canada Accepted 16 September 2006 Abstract Objective: To develop an approach by which studies describing the accuracy of diagnostic tests or clinical predictions can be combined in a meta-analysis, even though studies may report their results using different summary measures. Study design: A method is proposed to allow algebraic and numerical conversion of values of the Receiver Operating Characteristic Area Under the Curve (AUC) summary statistic into corresponding odds ratios (OR). A similar conversion is demonstrated for the standard errors (SEs) of these summary statistics. Results: The conversion of the AUC values into OR values was achieved using a logit-threshold model. The delta method was used to convert the associated SEs. An example concerning predictions of mortality in the intensive care unit illustrates the calculations. Conclusion: This paper provides an accessible method that permits the meta-analyst to overcome some of the difficulties implied by incomplete and inconsistent reporting of research studies in this area. It allows all studies to be included on the same metric, which in turn more easily permits exploration of issues such as heterogeneity. The method can readily be used for meta-analyses of diagnostic or screening tests, or for prediction data. Ó 2007 Elsevier Inc. All rights reserved. Keywords: Meta analysis; Diagnostic test; Prediction; Area Under the Curve; Odds ratio; Biostatistics 1. Introduction A difficulty encountered in carrying out a systematic review of a diagnostic test (or a prediction method) is that the component studies may report their results in different ways. Specifically, while some report diagnostic odds ratios (OR), others provide the Area Under the Curve (AUC) from a Receiver Operating Characteristic (ROC) curve. While both the OR and AUC are valid summary measures of diagnostic accuracy, they are on different metrics, which makes the combination of studies into the meta-analysis problematic. This difficulty is analogous to attempting to combine studies of therapeutic effectiveness that report different types of effect measure, for example, a mean reduction in blood pressure vs. the percentage of patients whose blood pressure has been reduced below a target threshold value [1]. In this paper we will illustrate a method to convert the studies reporting AUC values into corresponding ORs, which then facilitates their inclusion in a meta-analysis using standard methods for combining ORs [2]. The conversion of the AUC values into OR point estimates will be achieved using a logit-threshold model. We also use the delta method to obtain a conversion of their associated standard errors (SEs) and using a result proposed by Hanley and McNeil [3] for the SE of AUC. Our conversion method can be used equally well for data originating from a predictive rule (as in our illustrative example) or from a diagnostic test. * Corresponding author. Tel.: ext ; fax: address: walter@mcmaster.ca (S.D. Walter). 2. Example We were motivated to investigate this topic during a systematic overview of the accuracy of physicians predicting /07/$ e see front matter Ó 2007 Elsevier Inc. All rights reserved. doi: /j.jclinepi
2 S.D. Walter, T. Sinuff / Journal of Clinical Epidemiology 60 (2007) 530e future mortality of critically ill patients in intensive care units. Because of the subjective judgment required on the part of the physicians, we wished to compare their predictive accuracy with that of certain objective scoring systems. Full details of our systematic overview are given elsewhere [4], so we provide here only those details required for illustration of our method. From a total of 1,626 citations, 12 studies were identified that met our inclusion criteria. A funnel plot revealed no evidence of publication bias. There were seven studies where we could construct separate 2 2 tables of the mortality predicted by physicians and scoring systems vs. actual mortality. A prediction threshold of 50% was used to dichotomize data from studies where predictions were made using interval or continuous data. From each 2 2 table, a diagnostic OR was computed as the summary measure of predictive accuracy. The OR here is the ratio of the odds of prediction of death in a patient who actually dies, compared to the odds for a patient who actually survives. Thus, OR is obtained from (TP TN)/(FP FN), where TP, TN, FP, and FN are the true positive, true negative, false positive, and false negative predictions, respectively. Five studies did not give ORs, but instead used ROC methodology to report their findings. An ROC curve plots the TP prediction rate vs. the FP prediction rate as the prediction threshold varies between 0 and 1. These studies summarized their ROC curves using the AUC measure. A perfect prediction would yield AUC 5 1, whereas AUC 5 ½ would suggest predictive accuracy equal to that of chance alone. Two of the five studies reported a SE for the AUC, one gave confidence intervals (CI) for AUC, and the remaining two studies gave no indication of the precision of their AUC estimates. All five studies reported sample sizes, including the total number of patients and the number of deaths occurring. The five ROC studies had a total sample size of 3,215, while the seven studies reporting ORs had a total sample size of 2,706. Therefore, the ROC studies contained over half (54%) of the available data, and it seemed essential to be able to include them in the meta-analysis. In order to incorporate them in a standard analysis to estimate the overall OR, we needed to convert the AUC results into the OR metric. 3. Method We adopted the logit-threshold meta-analysis model proposed by Moses et al. [5,6] which relates the diagnostic (or predictive) OR to the test threshold. The model is defined as: D 5 a þ bs where D 5 ln(tpr/[1 TPR]) ln(fpr/[1 FPR]) is the diagnostic log odds ratio, ln(or), and S 5 ln(tpr/ [1 TPR]) þ ln(fpr/[1 FPR]) is a function of the test threshold. TPR and FPR are the true positive and false positive rates of the test, respectively. The regression slope b indicates the dependence of the test accuracy on the threshold. The intercept a can be taken as an overall estimate of ln OR. If the component studies in a meta-analysis are assumed to be approximately homogeneous with respect to their ORs, then we have shown elsewhere [7] that the AUC under the summary ROC curve can be expressed in terms of the corresponding OR through the following equation: AUC 5 OR ð1þ 2½ðOR 1Þ lnðorþš: ðor 1Þ This expression also gives a good approximation to the AUC even if the component studies are heterogeneous [7]. In the context of ROC curves for single studies, the homogeneous logit-threshold model is equivalent to assuming two logistic distributions for an underlying continuum of the test results, and where the distributions for true cases and noncases of disease differ only by a location shift. In the heterogeneous situation, the two distributions also differ by a scale factor (implying different variances). The logistic distribution also gives a reasonably close approximation to normally distributed test results. When one of our studies reported an AUC summary measure, we used equation (1) iteratively to solve for the corresponding value of OR, thus achieving the required conversion Standard errors We also converted SEs for the AUC estimates into the OR metric. First, we applied the delta method to equation (1) to obtain an approximate SE for AUC in terms of the SE of OR, as follows: 1 SEðAUCÞ 5 3½ðOR þ 1ÞlnðORÞ ðor 1Þ 2ðOR 1ÞŠSEðORÞ [cf. Ref. [7], equation 13]. This expression may be rearranged as SEðAUCÞðOR 1Þ 3 SEðORÞ 5 ðor þ 1ÞlnðORÞ 2ðOR 1Þ in order to give SE(OR) in terms of SE(AUC), the direction of conversion that we require here. We then used a method by Hanley and McNeil [3] through an approximation described by Zhou et al. [8] to obtain a SE for AUC from VarðAUCÞ5Q 1 =nþq 2 =m AUC 2 ðmþnþ=mn ð2þ where Q 1 5 AUC/(2 AUC), Q 2 5 2AUC/(1 þ AUC), m 5 number of deaths, n 5 number of survivors, and SE(AUC) 5 [Var(AUC)] 1/2. Finally, having obtained estimates of OR and their SEs, the five studies using ROC
3 532 S.D. Walter, T. Sinuff / Journal of Clinical Epidemiology 60 (2007) 530e534 methods could then be combined with the other seven studies that had reported in terms of OR originally. 4. Results Table 1 shows the AUC results for the five studies [9e13] that reported findings using ROC methods. Also shown are the corresponding ORs and their SEs, obtained using the approach outlined above. For the meta-analysis itself, all studies were expressed in terms of ln(or) and its corresponding SE, because of the greater stability and symmetry of the estimates after the logarithmic transformation [14]. Each study shows results for mortality predictions by physicians and a scoring algorithm, or in the case of one study [13], two alternative scoring methods. Three studies reported the AUC values to two decimal places (d.p.), while two studies gave them to three d.p. The limited accuracy of the former group may create numerical problems in the meta-analysis, because one then has only limited implied precision on the corresponding estimates of OR. For example, the study by Poses et al. [10] gives the AUC for physicians predictions as 0.82 (i.e., to two d.p.). From equation (1) we find that this value is compatible with (to an accuracy of one d.p.) OR values from 9.0 to 9.8, which is a relatively wide range. The associated values of SE(OR) ranged from 3.77 to In contrast, the study by Christensen et al. [9] gives the physicians AUC , that is, the summary measure is reported to three d.p. The value of is compatible with OR to 8.01 and the associated values of SE(OR) range from 2.36 to Thus, the narrower ranges for OR and its SE from studies that give their AUC value to more decimals is clearly evident. In fact, in meta-analyses that involve only a small number of studies, limited accuracy in reported AUC values might necessitate a sensitivity analysis to evaluate the impact of assuming various OR values within the range that is compatible with a study s AUC. Two of the ROC studies [10,12] in Table 1 reported empirical SEs for their AUC values, for each of the clinician and scoring system predictions. Numerical comparisons revealed that these were close to the SEs obtained from our delta method calculation. For the study that reported CI for AUC [12], we inferred an approximate empirical value of SE(AUC) from the width of the interval. For example, the width of the 95% CI for the AUC from physicians predictions was 0.10, implying an approximate SE(AUC) of 0.10/(2 1.96) ; the value calculated from the delta method was slightly higher, at The corresponding empirical and delta method SEs from the two scoring algorithms in this study were vs (scoring method 1) and vs (scoring method 2). The formulation by Zhou et al. [8] of SE(AUC) approximates the Hanley and McNeil method by ignoring second order and higher terms. We checked the accuracy of the Zhou et al. [8] approximation in two of the example papers. For the physicians prediction in the Christensen et al. paper [9], the discrepancy in SE(AUC) was less than 0.2%. As seen in Table 1, the Christensen et al. paper [9] has a fairly typical sample size. We also checked the accuracy of the Zhou et al. approximation using the physicians predictions from the Poses et al. study [10] that has the smallest sample size. Here the discrepancy of the approximation was approximately 4.5%. We regard this accuracy as quite satisfactory in practice. We constructed separate 2 2 tables of predicted mortality by physicians and scoring systems vs. actual mortality for the seven other studies that reported the original data [15e21]. The summary AUC for these studies was 0.85 (SE ) for physician predictions compared to 0.63 (SE ) for scoring system predictions, P The physicians summary OR derived from the AUC was significantly higher (12.43; 95% CI 5.47, 27.11) than the scoring systems summary OR (2.25; 95% CI 0.78, 6.52), P The combined results of all 12 studies indicated that physicians predict mortality more accurately than Table 1 Summary data from five studies using ROC methods to report accuracy of mortality prediction Study Sample size Number of deaths Prediction AUC OR SE (OR) Christensen et al MD Scoring Garrouste-Orgeas et al MD Scoring Knaus et al MD Scoring Poses et al MD Scoring Scholz et al MD Scoring Scoring AUC, Area Under the Curve (number of decimal places as reported in original papers); OR, Odds ratio; SE, Standard error.
4 S.D. Walter, T. Sinuff / Journal of Clinical Epidemiology 60 (2007) 530e scoring systems: the ratio of ORs (with 95% CI) was 1.92 (1.19, 3.08), P Discussion This paper has provided an accessible method to convert diagnostic or predictive studies that report their results in terms of AUC under the ROC curves into corresponding values of the OR. This approach allows all studies to be incorporated in a meta-analysis using the same metric, which in turn allows investigators to more easily explore issues such as heterogeneity. Approaches to analysis using the OR will be familiar to researchers who have conducted meta-analyses of therapeutic interventions. Of note, the Hanley and McNeil [3] formula is exact for nonparametric data, and it is very robust for various parametric distributions of data, including the normal, the gamma, and the negative exponential distributions. Hanley and McNeil [3] also indicate that the distribution of the AUC estimate will not display significant skewness as long as the number of misclassified pairs (conceptually, the pairs are made up of individuals with and without events) is less than five. The worst-case scenario for our example is exemplified by the study of Poses et al. [10], which has the smallest total sample size and the smallest number of deaths. The expected number of misclassified pairs for the Poses et al. study is approximately 12, well above the Hanley and McNeil threshold. We therefore conclude that it is reasonable to suppose the sampling distributions of the AUC estimates in our component studies will be reasonably symmetric. The demonstrated first order accuracy of the Hanley and McNeil [3] formula and the apparent symmetry of the AUC sampling distributions in our example imply that the delta method we have used will also be highly accurate to the first order with sample sizes such as we encountered in our example. For particularly small sample sizes, the accuracy of the delta method might need to be examined by Monte Carlo simulation. Inconsistent reporting of results is a very common problem in the medical literature regarding meta-analyses of diagnostic tests. To date, the medical literature includes 167 meta-analyses of evaluations of diagnostic tests (ranging from Doppler echocardiography and PSA tests to various diagnostic tests in the cancer literature), using ROCs to pool the data. More than half of these meta-analyses describe significant limitations due to (1) lack of reporting of raw sensitivity and specificity data to construct ROC curves, and (2) the use of different metrics which limit the authors abilities to pool all of the data. Our paper provides authors with a method to use AUC data (rather than sensitivity and specificity values), which are commonly presented in studies of diagnostic tests. The meta-analyst should be concerned about the possibility of asymmetry in the ROC curve from his/her component studies. However, as we noted earlier the AUC is very stable and robust to asymmetry. In fact, the AUC changes by less than 2% for practical cases of interest and the impact of asymmetry on the SE of the AUC is also rather small (see Ref. [7] for details). Furthermore, SE(AUC) declines with increasing asymmetry, so assuming the symmetric value of the SE will be conservative. With these points in mind, we feel that it is reasonable to use the AUC approach and its SE from the symmetric case. A practical consideration is that many authors do not report the value of the regression coefficients, their SEs, and their covariance, all of which are required to estimate SE(AUC) in the general case. Hence, there is then no alternative except to use the implied symmetric values of AUC and its SE. Strong asymmetry in an ROC curve would, however, be a concern, and indeed in such cases it may be undesirable to even attempt a meta-analysis at all with such studies. Nevertheless, to the extent that one trusts the AUC as a summary measure of the data (which, as noted above, seems reasonable in both symmetric and moderately asymmetric situations) one can proceed. It is only when the asymmetry is extreme that the method would break down. Prior to conducting the meta-analysis in our primary study [4], we assessed the symmetry of the ROC curves, whenever this information was available. There was at most only modest asymmetry of the ROC curves in the individual studies. Hence, we determined that it would be methodologically appropriate to conduct a meta-analysis. The conversion demonstrated in this paper permits the meta-analyst to overcome some of the difficulties implied by incomplete and inconsistent reporting of diagnostic or predictive studies. However, it should be recognized that this is still only a partial substitute for a more detailed analysis using individual patient data from all component studies. Acknowledgments The authors thank Drs. Neill Adhikari, Deborah Cook, Holger Schünemann, Lauren Griffith, and Graeme Rocker for their advice and contributions to the statistical analysis of the original systematic review. The work was partly supported by funding from the Natural Sciences and Engineering and Research Council. Dr. Sinuff is supported by a CIHR Clinician Scientist award. References [1] Morris SB, DeShon RP. Combining effect size estimates in metaanalysis with repeated measures and independent-groups designs. Psychol Methods 2002;7:105e25. [2] Deeks JJ, Altman DG, Bradburn MJ. Statistical methods for examining heterogeneity and combining results from several studies in meta-analysis. In: Egger M, Smith GD, Altman DG, editors. Systematic reviews in health care: meta-analysis in context. 2nd ed. London: BMJ Books; p. 285e312.
5 534 S.D. Walter, T. Sinuff / Journal of Clinical Epidemiology 60 (2007) 530e534 [3] Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982;143: 29e36. [4] Sinuff T, Adhikari NKJ, Cook DJ, Schünemann HJ, Griffith LE, Rocker G, et al. Mortality predictions in the intensive care unit: comparing physicians to scoring systems. Crit Care Med 2006;34: 878e85. [5] Moses LE, Shapiro D, Littenberg B. Combining independent studies of a diagnostic test into a summary ROC curve: data-analytic approaches and come additional considerations. Stat Med 1993;12: 1293e316. [6] Littenberg B, Moses LE. Estimating accuracy from multiple conflicting reports: a new analytic method. Med Decis Making 1993;13: 313e21. [7] Walter SD. Properties of the summary receiver operating characteristic (SROC) curve for diagnostic test data. Stat Med 2002;21:1237e56. [8] Zhou XA, Obuchowski NA, McClish DK. Statistical methods in diagnostic medicine. New York, NY: John Wiley & Sons, Inc.; [9] Christensen C, Cottrell JJ, Murakami J, Mackesy ME, Fetzer B, Elstein AS, et al. Forecasting survival in the medical intensive care unit: a comparison of clinical prognoses with formal estimates. Methods Inf Med 1993;32:302e8. [10] Poses RM, McClish DK, Bekes C, Scott WE, Morely JN. Ego bias, reverse ego bias, and physicians prognostic. Crit Care Med 1991; 19:1533e9. [11] Knaus WA, Harrell FE Jr, Lynn J, Goldman L, Phillips RS, Connore AF, et al. The SUPPORT prognostic model. Objective estimates of survival for seriously ill hospitalized adults. Ann Intern Med 1995;122:191e203. [12] Garrouste-Orgeas M, Montuchad L, Timsit J-F, Misset B, Christias M, Carlet J. Triaging patients to the ICU: a pilot study of factors influencing admission decisions and patient outcomes. Intensive Care Med 2003;29:774e81. [13] Scholz N, Basler K, Saur P, Burchardi H, Felder S. Outcome prediction in critical care: physicians prognoses vs scoring systems. Eur J Anaesthesiol 2004;21:606e11. [14] Armitage P, Berry G, Matthews JNS, editors. Statistical methods in medical research. 4th ed. Oxford: Blackwell Science; [15] Brannen AL II, Godfrey LJ, Goetter WE. Prediction of outcome from critical illness. A comparison of clinical judgment with a prediction rule. Arch Intern Med 1999;149:1083e6. [16] Copeland-Fields L, Griffin T, Jenkins T, Buckley M, Wise LC. Comparison of outcome predictions made by physicians, by nurses, and by using the mortality prediction model. Am J Crit Care 2001;10:313e9. [17] Meyer AA, Messick WJ, Young P, Baker CC, Fakhry S, Muakkassa F, et al. Prospective comparison of clinical judgment and APACHE II score in predicting the outcome in critically ill surgical patients. J Trauma 1992;32:747e54. [18] Marks RJ, Simons RS, Blizzard RA, Browne DRG. Predicting outcome in intensive therapy unitsda comparison of APACHE II with subjective assessment. Intensive Care Med 1991;17:159e63. [19] Chang RWS, Lee B, Jacobs S, Lee B. Accuracy of decisions to withdraw therapy in critically ill patients: clinical judgment versus a computer model. Crit Care Med 1989;12:1091e7. [20] Kruse JA, Thill-Baharozian MC, Carlson RW. Comparison of clinical assessment with APACHE II for predicting mortality risk in patients admitted to a medical intensive care unit. JAMA 1988;260: 1739e42. [21] Katzman-McClish D, Powell SH. How well can physicians estimate mortality in a medical intensive care unit? Med Decis Making 1989;9:125e32.
Introduction to diagnostic accuracy meta-analysis. Yemisi Takwoingi October 2015
Introduction to diagnostic accuracy meta-analysis Yemisi Takwoingi October 2015 Learning objectives To appreciate the concept underlying DTA meta-analytic approaches To know the Moses-Littenberg SROC method
More informationIntroduction to Meta-analysis of Accuracy Data
Introduction to Meta-analysis of Accuracy Data Hans Reitsma MD, PhD Dept. of Clinical Epidemiology, Biostatistics & Bioinformatics Academic Medical Center - Amsterdam Continental European Support Unit
More informationMeta-analyses evaluating diagnostic test accuracy
THE STATISTICIAN S PAGE Summary Receiver Operating Characteristic Curve Analysis Techniques in the Evaluation of Diagnostic Tests Catherine M. Jones, MBBS, BSc(Stat), and Thanos Athanasiou, MD, PhD, FETCS
More informationBayesian meta-analysis of Papanicolaou smear accuracy
Gynecologic Oncology 107 (2007) S133 S137 www.elsevier.com/locate/ygyno Bayesian meta-analysis of Papanicolaou smear accuracy Xiuyu Cong a, Dennis D. Cox b, Scott B. Cantor c, a Biometrics and Data Management,
More informationAn Empirical Assessment of Bivariate Methods for Meta-analysis of Test Accuracy
Number XX An Empirical Assessment of Bivariate Methods for Meta-analysis of Test Accuracy Prepared for: Agency for Healthcare Research and Quality U.S. Department of Health and Human Services 54 Gaither
More informationMETHODS FOR DETECTING CERVICAL CANCER
Chapter III METHODS FOR DETECTING CERVICAL CANCER 3.1 INTRODUCTION The successful detection of cervical cancer in a variety of tissues has been reported by many researchers and baseline figures for the
More informationReview. Imagine the following table being obtained as a random. Decision Test Diseased Not Diseased Positive TP FP Negative FN TN
Outline 1. Review sensitivity and specificity 2. Define an ROC curve 3. Define AUC 4. Non-parametric tests for whether or not the test is informative 5. Introduce the binormal ROC model 6. Discuss non-parametric
More informationWhat is indirect comparison?
...? series New title Statistics Supported by sanofi-aventis What is indirect comparison? Fujian Song BMed MMed PhD Reader in Research Synthesis, Faculty of Health, University of East Anglia Indirect comparison
More informationMethods Research Report. An Empirical Assessment of Bivariate Methods for Meta-Analysis of Test Accuracy
Methods Research Report An Empirical Assessment of Bivariate Methods for Meta-Analysis of Test Accuracy Methods Research Report An Empirical Assessment of Bivariate Methods for Meta-Analysis of Test Accuracy
More informationPerformance of the Trim and Fill Method in Adjusting for the Publication Bias in Meta-Analysis of Continuous Data
American Journal of Applied Sciences 9 (9): 1512-1517, 2012 ISSN 1546-9239 2012 Science Publication Performance of the Trim and Fill Method in Adjusting for the Publication Bias in Meta-Analysis of Continuous
More informationMeta-analysis of diagnostic accuracy studies. Mariska Leeflang (with thanks to Yemisi Takwoingi, Jon Deeks and Hans Reitsma)
Meta-analysis of diagnostic accuracy studies Mariska Leeflang (with thanks to Yemisi Takwoingi, Jon Deeks and Hans Reitsma) 1 Diagnostic Test Accuracy Reviews 1. Framing the question 2. Identification
More informationMeta-analysis of Diagnostic Test Accuracy Studies
GUIDELINE Meta-analysis of Diagnostic Test Accuracy Studies November 2014 Copyright EUnetHTA 2013. All Rights Reserved. No part of this document may be reproduced without an explicit acknowledgement of
More informationIntroduction to ROC analysis
Introduction to ROC analysis Andriy I. Bandos Department of Biostatistics University of Pittsburgh Acknowledgements Many thanks to Sam Wieand, Nancy Obuchowski, Brenda Kurland, and Todd Alonzo for previous
More informationMeta-Analysis. Zifei Liu. Biological and Agricultural Engineering
Meta-Analysis Zifei Liu What is a meta-analysis; why perform a metaanalysis? How a meta-analysis work some basic concepts and principles Steps of Meta-analysis Cautions on meta-analysis 2 What is Meta-analysis
More informationSystematic Reviews and meta-analyses of Diagnostic Test Accuracy. Mariska Leeflang
Systematic Reviews and meta-analyses of Diagnostic Test Accuracy Mariska Leeflang m.m.leeflang@amc.uva.nl This presentation 1. Introduction: accuracy? 2. QUADAS-2 exercise 3. Meta-analysis of diagnostic
More informationIntroduction to Meta-Analysis
Introduction to Meta-Analysis Nazım Ço galtay and Engin Karada g Abstract As a means to synthesize the results of multiple studies, the chronological development of the meta-analysis method was in parallel
More informationDepartment of Epidemiology, Rollins School of Public Health, Emory University, Atlanta GA, USA.
A More Intuitive Interpretation of the Area Under the ROC Curve A. Cecile J.W. Janssens, PhD Department of Epidemiology, Rollins School of Public Health, Emory University, Atlanta GA, USA. Corresponding
More informationWeek 2 Video 3. Diagnostic Metrics
Week 2 Video 3 Diagnostic Metrics Different Methods, Different Measures Today we ll continue our focus on classifiers Later this week we ll discuss regressors And other methods will get worked in later
More informationREPRODUCTIVE ENDOCRINOLOGY
FERTILITY AND STERILITY VOL. 74, NO. 2, AUGUST 2000 Copyright 2000 American Society for Reproductive Medicine Published by Elsevier Science Inc. Printed on acid-free paper in U.S.A. REPRODUCTIVE ENDOCRINOLOGY
More informationStatistics, Probability and Diagnostic Medicine
Statistics, Probability and Diagnostic Medicine Jennifer Le-Rademacher, PhD Sponsored by the Clinical and Translational Science Institute (CTSI) and the Department of Population Health / Division of Biostatistics
More informationMeta-analysis of diagnostic test accuracy studies with multiple & missing thresholds
Meta-analysis of diagnostic test accuracy studies with multiple & missing thresholds Richard D. Riley School of Health and Population Sciences, & School of Mathematics, University of Birmingham Collaborators:
More informationCochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy
Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Chapter 10 Analysing and Presenting Results Petra Macaskill, Constantine Gatsonis, Jonathan Deeks, Roger Harbord, Yemisi Takwoingi.
More informationMultivariate Mixed-Effects Meta-Analysis of Paired-Comparison Studies of Diagnostic Test Accuracy
Multivariate Mixed-Effects Meta-Analysis of Paired-Comparison Studies of Diagnostic Test Accuracy Ben A. Dwamena, MD The University of Michigan & VA Medical Centers, Ann Arbor SNASUG - July 24, 2008 Diagnostic
More informationMeta-analysis of diagnostic research. Karen R Steingart, MD, MPH Chennai, 15 December Overview
Meta-analysis of diagnostic research Karen R Steingart, MD, MPH karenst@uw.edu Chennai, 15 December 2010 Overview Describe key steps in a systematic review/ meta-analysis of diagnostic test accuracy studies
More informationResearch and Evaluation Methodology Program, School of Human Development and Organizational Studies in Education, University of Florida
Vol. 2 (1), pp. 22-39, Jan, 2015 http://www.ijate.net e-issn: 2148-7456 IJATE A Comparison of Logistic Regression Models for Dif Detection in Polytomous Items: The Effect of Small Sample Sizes and Non-Normality
More informationThe conditional relative odds ratio provided less biased results for comparing diagnostic test accuracy in meta-analyses
Journal of Clinical Epidemiology 57 (2004) 461 469 The conditional relative odds ratio provided less biased results for comparing diagnostic test accuracy in meta-analyses Sadao Suzuki a,b, *, Takeo Moro-oka
More informationChoice of axis, tests for funnel plot asymmetry, and methods to adjust for publication bias
Technical appendix Choice of axis, tests for funnel plot asymmetry, and methods to adjust for publication bias Choice of axis in funnel plots Funnel plots were first used in educational research and psychology,
More informationNet Reclassification Risk: a graph to clarify the potential prognostic utility of new markers
Net Reclassification Risk: a graph to clarify the potential prognostic utility of new markers Ewout Steyerberg Professor of Medical Decision Making Dept of Public Health, Erasmus MC Birmingham July, 2013
More informationProblem solving therapy
Introduction People with severe mental illnesses such as schizophrenia may show impairments in problem-solving ability. Remediation interventions such as problem solving skills training can help people
More informationSYSTEMATIC REVIEWS OF TEST ACCURACY STUDIES
Biomarker & Test Evaluation Program SYSTEMATIC REVIEWS OF TEST ACCURACY STUDIES Patrick MM Bossuyt Structure 1. Clinical Scenarios 2. Test Accuracy Studies 3. Systematic Reviews 4. Meta-Analysis 5.
More informationMeasuring association in contingency tables
Measuring association in contingency tables Patrick Breheny April 3 Patrick Breheny University of Iowa Introduction to Biostatistics (BIOS 4120) 1 / 28 Hypothesis tests and confidence intervals Fisher
More informationReceiver operating characteristic
Receiver operating characteristic From Wikipedia, the free encyclopedia In signal detection theory, a receiver operating characteristic (ROC), or simply ROC curve, is a graphical plot of the sensitivity,
More informationThe moderating impact of temporal separation on the association between intention and physical activity: a meta-analysis
PSYCHOLOGY, HEALTH & MEDICINE, 2016 VOL. 21, NO. 5, 625 631 http://dx.doi.org/10.1080/13548506.2015.1080371 The moderating impact of temporal separation on the association between intention and physical
More informationSystematic reviews of prognostic studies 3 meta-analytical approaches in systematic reviews of prognostic studies
Systematic reviews of prognostic studies 3 meta-analytical approaches in systematic reviews of prognostic studies Thomas PA Debray, Karel GM Moons for the Cochrane Prognosis Review Methods Group Conflict
More informationTreatment effect estimates adjusted for small-study effects via a limit meta-analysis
Treatment effect estimates adjusted for small-study effects via a limit meta-analysis Gerta Rücker 1, James Carpenter 12, Guido Schwarzer 1 1 Institute of Medical Biometry and Medical Informatics, University
More informationCochrane Pregnancy and Childbirth Group Methodological Guidelines
Cochrane Pregnancy and Childbirth Group Methodological Guidelines [Prepared by Simon Gates: July 2009, updated July 2012] These guidelines are intended to aid quality and consistency across the reviews
More informationMultivariate Mixed-Effects Meta-Analysis of Paired-Comparison Studies of Diagnostic Test Accuracy
Multivariate Mixed-Effects Meta-Analysis of Paired-Comparison Studies of Diagnostic Test Accuracy Ben A. Dwamena, MD The University of Michigan & VA Medical Centers, Ann Arbor SNASUG - July 24, 2008 B.A.
More informationMeasuring association in contingency tables
Measuring association in contingency tables Patrick Breheny April 8 Patrick Breheny Introduction to Biostatistics (171:161) 1/25 Hypothesis tests and confidence intervals Fisher s exact test and the χ
More informationGood Old clinical markers have similar power in breast cancer prognosis as microarray gene expression profilers q
European Journal of Cancer 40 (2004) 1837 1841 European Journal of Cancer www.ejconline.com Good Old clinical markers have similar power in breast cancer prognosis as microarray gene expression profilers
More informationEVIDENCE-BASED GUIDELINE DEVELOPMENT FOR DIAGNOSTIC QUESTIONS
EVIDENCE-BASED GUIDELINE DEVELOPMENT FOR DIAGNOSTIC QUESTIONS Emily Vella, Xiaomei Yao Cancer Care Ontario's Program in Evidence-Based Care, Department of Oncology, McMaster University, Ontario, Canada
More informationResults. NeuRA Mindfulness and acceptance therapies August 2018
Introduction involve intentional and non-judgmental focus of one's attention on emotions, thoughts and sensations that are occurring in the present moment. The aim is to open awareness to present experiences,
More informationEvaluating the results of a Systematic Review/Meta- Analysis
Open Access Publication Evaluating the results of a Systematic Review/Meta- Analysis by Michael Turlik, DPM 1 The Foot and Ankle Online Journal 2 (7): 5 This is the second of two articles discussing the
More informationMeta Analysis. David R Urbach MD MSc Outcomes Research Course December 4, 2014
Meta Analysis David R Urbach MD MSc Outcomes Research Course December 4, 2014 Overview Definitions Identifying studies Appraising studies Quantitative synthesis Presentation of results Examining heterogeneity
More informationVU Biostatistics and Experimental Design PLA.216
VU Biostatistics and Experimental Design PLA.216 Julia Feichtinger Postdoctoral Researcher Institute of Computational Biotechnology Graz University of Technology Outline for Today About this course Background
More informationGenetic risk prediction for CHD: will we ever get there or are we already there?
Genetic risk prediction for CHD: will we ever get there or are we already there? Themistocles (Tim) Assimes, MD PhD Assistant Professor of Medicine Stanford University School of Medicine WHI Investigators
More informationCorrecting AUC for Measurement Error
Correcting AUC for Measurement Error The Harvard community has made this article openly available. Please share how this access benefits you. Your story matters. Citation Published Version Accessed Citable
More informationTraumatic brain injury
Introduction It is well established that traumatic brain injury increases the risk for a wide range of neuropsychiatric disturbances, however there is little consensus on whether it is a risk factor for
More informationSystematic Reviews. Simon Gates 8 March 2007
Systematic Reviews Simon Gates 8 March 2007 Contents Reviewing of research Why we need reviews Traditional narrative reviews Systematic reviews Components of systematic reviews Conclusions Key reference
More informationResults. NeuRA Hypnosis June 2016
Introduction may be experienced as an altered state of consciousness or as a state of relaxation. There is no agreed framework for administering hypnosis, but the procedure often involves induction (such
More informationGraphical assessment of internal and external calibration of logistic regression models by using loess smoothers
Tutorial in Biostatistics Received 21 November 2012, Accepted 17 July 2013 Published online 23 August 2013 in Wiley Online Library (wileyonlinelibrary.com) DOI: 10.1002/sim.5941 Graphical assessment of
More informationHow to Conduct a Meta-Analysis
How to Conduct a Meta-Analysis Faculty Development and Diversity Seminar ludovic@bu.edu Dec 11th, 2017 Periodontal disease treatment and preterm birth We conducted a metaanalysis of randomized controlled
More information1 Introduction. st0020. The Stata Journal (2002) 2, Number 3, pp
The Stata Journal (22) 2, Number 3, pp. 28 289 Comparative assessment of three common algorithms for estimating the variance of the area under the nonparametric receiver operating characteristic curve
More informationState of the Art in Clinical & Anatomic Pathology. Meta-analysis of Clinical Studies of Diagnostic Tests
State of the Art in Clinical & Anatomic Pathology Meta-analysis of Clinical Studies of Diagnostic Tests Developments in How the Receiver Operating Characteristic Works N Meta-analytic summaries are needed
More informationThe recommended method for diagnosing sleep
reviews Measuring Agreement Between Diagnostic Devices* W. Ward Flemons, MD; and Michael R. Littner, MD, FCCP There is growing interest in using portable monitoring for investigating patients with suspected
More informationEvidence-Based Medicine and Publication Bias Desmond Thompson Merck & Co.
Evidence-Based Medicine and Publication Bias Desmond Thompson Merck & Co. Meta-Analysis Defined A meta-analysis is: the statistical combination of two or more separate studies In other words: overview,
More informationBiostatistics II
Biostatistics II 514-5509 Course Description: Modern multivariable statistical analysis based on the concept of generalized linear models. Includes linear, logistic, and Poisson regression, survival analysis,
More informationExplaining heterogeneity
Explaining heterogeneity Julian Higgins School of Social and Community Medicine, University of Bristol, UK Outline 1. Revision and remarks on fixed-effect and random-effects metaanalysis methods (and interpretation
More informationA Brief Introduction to Bayesian Statistics
A Brief Introduction to Statistics David Kaplan Department of Educational Psychology Methods for Social Policy Research and, Washington, DC 2017 1 / 37 The Reverend Thomas Bayes, 1701 1761 2 / 37 Pierre-Simon
More informationVarious performance measures in Binary classification An Overview of ROC study
Various performance measures in Binary classification An Overview of ROC study Suresh Babu. Nellore Department of Statistics, S.V. University, Tirupati, India E-mail: sureshbabu.nellore@gmail.com Abstract
More informationEstimation of Area under the ROC Curve Using Exponential and Weibull Distributions
XI Biennial Conference of the International Biometric Society (Indian Region) on Computational Statistics and Bio-Sciences, March 8-9, 22 43 Estimation of Area under the ROC Curve Using Exponential and
More informationFixed-Effect Versus Random-Effects Models
PART 3 Fixed-Effect Versus Random-Effects Models Introduction to Meta-Analysis. Michael Borenstein, L. V. Hedges, J. P. T. Higgins and H. R. Rothstein 2009 John Wiley & Sons, Ltd. ISBN: 978-0-470-05724-7
More informationIndirect Treatment Comparison Without Network Meta-Analysis: Overview of Novel Techniques
Indirect Treatment Comparison Without Network Meta-Analysis: Overview of Novel Techniques K. Jack Ishak, PhD, Executive Director, Statistics and Senior Research Scientist BACKGROUND There is generally
More informationStatistical considerations in indirect comparisons and network meta-analysis
Statistical considerations in indirect comparisons and network meta-analysis Said Business School, Oxford, UK March 18-19, 2013 Cochrane Comparing Multiple Interventions Methods Group Oxford Training event,
More informationResults. NeuRA Worldwide incidence April 2016
Introduction The incidence of schizophrenia refers to how many new cases there are per population in a specified time period. It is different from prevalence, which refers to how many existing cases there
More informationA scored AUC Metric for Classifier Evaluation and Selection
A scored AUC Metric for Classifier Evaluation and Selection Shaomin Wu SHAOMIN.WU@READING.AC.UK School of Construction Management and Engineering, The University of Reading, Reading RG6 6AW, UK Peter Flach
More informationResults. NeuRA Treatments for internalised stigma December 2017
Introduction Internalised stigma occurs within an individual, such that a person s attitude may reinforce a negative self-perception of mental disorders, resulting in reduced sense of selfworth, anticipation
More informationStatistical considerations in indirect comparisons and network meta-analysis
Statistical considerations in indirect comparisons and network meta-analysis Said Business School, Oxford, UK March 18-19, 2013 Cochrane Comparing Multiple Interventions Methods Group Oxford Training event,
More informationMeta-Analysis of Correlation Coefficients: A Monte Carlo Comparison of Fixed- and Random-Effects Methods
Psychological Methods 01, Vol. 6, No. 2, 161-1 Copyright 01 by the American Psychological Association, Inc. 82-989X/01/S.00 DOI:.37//82-989X.6.2.161 Meta-Analysis of Correlation Coefficients: A Monte Carlo
More informationMethods for Computing Missing Item Response in Psychometric Scale Construction
American Journal of Biostatistics Original Research Paper Methods for Computing Missing Item Response in Psychometric Scale Construction Ohidul Islam Siddiqui Institute of Statistical Research and Training
More informationMantel-Haenszel Procedures for Detecting Differential Item Functioning
A Comparison of Logistic Regression and Mantel-Haenszel Procedures for Detecting Differential Item Functioning H. Jane Rogers, Teachers College, Columbia University Hariharan Swaminathan, University of
More informationCapture-recapture method for assessing publication bias
Received: 30.9.2009 Accepted: 19.1.2010 Original Article Capture-recapture method for assessing publication bias Jalal Poorolajal* a, Ali Akbar Haghdoost b, Mahmood Mahmoodi a, Reza Majdzadeh a, Siavosh
More informationcloglog link function to transform the (population) hazard probability into a continuous
Supplementary material. Discrete time event history analysis Hazard model details. In our discrete time event history analysis, we used the asymmetric cloglog link function to transform the (population)
More informationMachine learning II. Juhan Ernits ITI8600
Machine learning II Juhan Ernits ITI8600 Hand written digit recognition 64 Example 2: Face recogition Classification, regression or unsupervised? How many classes? Example 2: Face recognition Classification,
More informationOptimal full matching for survival outcomes: a method that merits more widespread use
Research Article Received 3 November 2014, Accepted 6 July 2015 Published online 6 August 2015 in Wiley Online Library (wileyonlinelibrary.com) DOI: 10.1002/sim.6602 Optimal full matching for survival
More informationSanjay P. Zodpey Clinical Epidemiology Unit, Department of Preventive and Social Medicine, Government Medical College, Nagpur, Maharashtra, India.
Research Methodology Sample size and power analysis in medical research Sanjay P. Zodpey Clinical Epidemiology Unit, Department of Preventive and Social Medicine, Government Medical College, Nagpur, Maharashtra,
More informationEvaluating the Quality of Evidence from a Network Meta- Analysis
Evaluating the Quality of Evidence from a Network Meta- Analysis Georgia Salanti 1, Cinzia Del Giovane 2, Anna Chaimani 1, Deborah M. Caldwell 3, Julian P. T. Higgins 3,4 * 1 Department of Hygiene and
More informationZheng Yao Sr. Statistical Programmer
ROC CURVE ANALYSIS USING SAS Zheng Yao Sr. Statistical Programmer Outline Background Examples: Accuracy assessment Compare ROC curves Cut-off point selection Summary 2 Outline Background Examples: Accuracy
More informationUsing Statistical Principles to Implement FDA Guidance on Cardiovascular Risk Assessment for Diabetes Drugs
Using Statistical Principles to Implement FDA Guidance on Cardiovascular Risk Assessment for Diabetes Drugs David Manner, Brenda Crowe and Linda Shurzinske BASS XVI November 9-13, 2009 Acknowledgements
More informationYour submission states that the adjusted indirect comparison methodology was based largely on the publication by Glenny et al
Clarification of the indirect comparison Your submission states that the adjusted indirect comparison methodology was based largely on the publication by Glenny et al Your methodology is as follows: An
More informationClinical Epidemiology for the uninitiated
Clinical epidemiologist have one foot in clinical care and the other in clinical practice research. As clinical epidemiologists we apply a wide array of scientific principles, strategies and tactics to
More informationCOMPARING SEVERAL DIAGNOSTIC PROCEDURES USING THE INTRINSIC MEASURES OF ROC CURVE
DOI: 105281/zenodo47521 Impact Factor (PIF): 2672 COMPARING SEVERAL DIAGNOSTIC PROCEDURES USING THE INTRINSIC MEASURES OF ROC CURVE Vishnu Vardhan R* and Balaswamy S * Department of Statistics, Pondicherry
More informationEmpirical assessment of univariate and bivariate meta-analyses for comparing the accuracy of diagnostic tests
Empirical assessment of univariate and bivariate meta-analyses for comparing the accuracy of diagnostic tests Yemisi Takwoingi, Richard Riley and Jon Deeks Outline Rationale Methods Findings Summary Motivating
More informationGUIDELINE COMPARATORS & COMPARISONS:
GUIDELINE COMPARATORS & COMPARISONS: Direct and indirect comparisons Adapted version (2015) based on COMPARATORS & COMPARISONS: Direct and indirect comparisons - February 2013 The primary objective of
More informationIssues in Meta-Analysis: An Overview
REVIEW Issues in Meta-Analysis: An Overview Alfred A. Bartolucci, Charles R. Katholi, Karan P. Singh, and Graciela S. Alarc6n A summary is provided for the relevant issues that investigators must address
More informationFebruary 14, 2001 ! OUTLINE
February 14, 2001! OUTLINE ASSESSING THE GENERALIZABILITY OF THE RESULTS OF - Define the problem MULTINATIONAL TRIALS FOR INDIVIDUAL COUNTRIES PARTICIPATING IN THE TRIAL - Common Approaches to the problem
More informationFalse statistically significant findings in cumulative metaanalyses and the ability of Trial Sequential Analysis (TSA) to identify them.
False statistically significant findings in cumulative metaanalyses and the ability of Trial Sequential Analysis (TSA) to identify them. Protocol Georgina Imberger 1 Kristian Thorlund 2,1 Christian Gluud
More informationComparison of Different Methods of Detecting Publication Bias
Shaping the Future of Drug Development Comparison of Different Methods of Detecting Publication Bias PhUSE 2017, Edinburgh Janhavi Kale, Cytel Anwaya Nirpharake, Cytel Outline Systematic review and Meta-analysis
More informationMeasurement Error in Nonlinear Models
Measurement Error in Nonlinear Models R.J. CARROLL Professor of Statistics Texas A&M University, USA D. RUPPERT Professor of Operations Research and Industrial Engineering Cornell University, USA and L.A.
More informationKnowledge Discovery and Data Mining. Testing. Performance Measures. Notes. Lecture 15 - ROC, AUC & Lift. Tom Kelsey. Notes
Knowledge Discovery and Data Mining Lecture 15 - ROC, AUC & Lift Tom Kelsey School of Computer Science University of St Andrews http://tom.home.cs.st-andrews.ac.uk twk@st-andrews.ac.uk Tom Kelsey ID5059-17-AUC
More informationIntroduction. We can make a prediction about Y i based on X i by setting a threshold value T, and predicting Y i = 1 when X i > T.
Diagnostic Tests 1 Introduction Suppose we have a quantitative measurement X i on experimental or observed units i = 1,..., n, and a characteristic Y i = 0 or Y i = 1 (e.g. case/control status). The measurement
More informationResearch Synthesis and meta-analysis: themes. Graham A. Colditz, MD, DrPH Method Tuuli, MD, MPH
Research Synthesis and meta-analysis: themes Graham A. Colditz, MD, DrPH Method Tuuli, MD, MPH Today Course format Goals, competencies Overview of themes for the class SRMA M19-551 Course Format Lectures»
More informationDistraction techniques
Introduction are a form of coping skills enhancement, taught during cognitive behavioural therapy. These techniques are used to distract and draw attention away from the auditory symptoms of schizophrenia,
More informationChapter 13 Estimating the Modified Odds Ratio
Chapter 13 Estimating the Modified Odds Ratio Modified odds ratio vis-à-vis modified mean difference To a large extent, this chapter replicates the content of Chapter 10 (Estimating the modified mean difference),
More informationAtherosclerosis 220 (2012) Contents lists available at ScienceDirect. Atherosclerosis
Atherosclerosis 220 (2012) 128 133 Contents lists available at ScienceDirect Atherosclerosis journal homepage: www.elsevier.com/locate/atherosclerosis Carotid plaque, compared with carotid intima-media
More informationTeaching A Way of Implementing Statistical Methods for Ordinal Data to Researchers
Journal of Mathematics and System Science (01) 8-1 D DAVID PUBLISHING Teaching A Way of Implementing Statistical Methods for Ordinal Data to Researchers Elisabeth Svensson Department of Statistics, Örebro
More informationMethod Comparison for Interrater Reliability of an Image Processing Technique in Epilepsy Subjects
22nd International Congress on Modelling and Simulation, Hobart, Tasmania, Australia, 3 to 8 December 2017 mssanz.org.au/modsim2017 Method Comparison for Interrater Reliability of an Image Processing Technique
More informationOUTCOMES OF DICHOTOMIZING A CONTINUOUS VARIABLE IN THE PSYCHIATRIC EARLY READMISSION PREDICTION MODEL. Ng CG
ORIGINAL PAPER OUTCOMES OF DICHOTOMIZING A CONTINUOUS VARIABLE IN THE PSYCHIATRIC EARLY READMISSION PREDICTION MODEL Ng CG Department of Psychological Medicine, Faculty of Medicine, University Malaya,
More informationStudy protocol v. 1.0 Systematic review of the Sequential Organ Failure Assessment score as a surrogate endpoint in randomized controlled trials
Study protocol v. 1.0 Systematic review of the Sequential Organ Failure Assessment score as a surrogate endpoint in randomized controlled trials Harm Jan de Grooth, Jean Jacques Parienti, [to be determined],
More information