An empirical comparison of methods to meta-analyze individual patient data of diagnostic accuracy

Transcription

1 An empirical comparison of methods to meta-analyze individual patient data of diagnostic accuracy Gabrielle Simoneau Master of Science Department of Epidemiology, Biostatistics and Occupational Health McGill University Montreal, Quebec August 2015 A thesis submitted to McGill University in partial fulfillment of the requirements of the degree of Master in Biostatistics Gabrielle Simoneau 2015

2 ACKNOWLEDGEMENTS I acknowledge the CIHR Knowledge Synthesis Grant Improving Depression Screening by Reducing Bias in Accuracy Estimates: An Independent Patient Data Meta-Analysis of the PHQ-9 for funding my year of research. I thank my supervisor Dr. Andrea Benedetti for guiding me through this research and for offering me interesting academic and career opportunities. I thank Dr. Brett Thombs for his thoughtful advice and Brooke Levis for the large amount of work she has previously done to make my research feasible. I thank my parents, my boyfriend and my roommates for their patience, support, encouragement. ii

3 ABSTRACT Individual patient data (IPD) meta-analyses are increasingly common in the literature. In the context of diagnostic accuracy of ordinal or semi-continuous scale tests, sensitivity and specificity are traditionally reported for a given threshold and a meta-analysis is conducted via a bivariate approach to account for the correlation between sensitivity and specificity. With IPD, sensitivity and specificity can be pooled for every possible threshold. One way to analyze these data is to use the bivariate approach separately at every threshold. Another idea is to analyze sensitivity and specificity for all thresholds simultaneously, and thus account for their within-study correlation across thresholds. Among other approaches, the ordinal multivariate random-effects model and the normal multivariate random-effects model have been proposed to meta-analyze diagnostic test data with multiple thresholds. Our aim is to compare these two models to the bivariate approach when IPD of diagnostic accuracy are available, empirically and via simulations. The CAGE IPD dataset and the 9-item Patient Health Questionnaire (PHQ-9) IPD dataset are used to conduct the empirical comparisons. The empirical comparisons showed that the two multivariate methods were difficult to apply and prone to sporadic convergence issues while the bivariate approach was easy to apply but overly simplistic. Notwithstanding, simulations showed that ignoring the within-study correlation of sensitivity and specificity across thresholds does not dramatically affect the inference of the bivariate model. The two multivariate models were not suitable for simulations, which emphasized the complexity of the models. We recommend IPD of diagnostic accuracy to be iii

4 meta-analyzed via the bivariate approach or the more complete ordinal multivariate approach. iv

5 ABRÉGÉ Les méta-analyses sur les données individuelles de patient (IPD) sont de plus en plus communes dans la littérature scientifique. L utilisation de données individuelles de patient a plusieurs avantages. Dans le contexte d études sur la précision du diagnostique de test à échelle ordinale ou semi-continue, la sensibilité et la spécificité du test sont habituellement reportées pour un certain seuil et une méta-analyse est conduite via une approche bivariée qui prend en compte la corrélation entre la sensibilité et la spécificité. Avec des IPD, il est possible d obtenir une paire de sensibilité et spécificité pour tous les seuils possibles. Une façon d analyser ce genre de données est d appliquer le modèle bivarié seuil par seuil. Une autre façon d analyser ces données est de modéliser les résultats pour tous les seuils simultanément. La corrélation entre les différents seuils pour chaque étude est alors prise en compte. Parmi d autres méthodes, les modèles ordinal multivarié avec effets aléatoires et normal multivarié avec effets aléatoires ont été proposés pour méta-analyser des données sur la précision du diagnostic avec plusieurs seuils. Notre objectif est de comparer ces deux méthodes au modèle bivarié, empiriquement et via une étude de simulation, lorsque des données IPD sont disponibles. Les données IPD CAGE et les données IPD sur le Questionnaire sur la Santé du Patient-9 (PHQ-9) sont utilisées pour mener les comparaisons empiriques. Les comparaisons empiriques ont montré que les deux approches multivariées sont difficiles à appliquer et sujettes à d occasionnel problèmes de convergence tandis que l approche bivariée est simple à appliquer, mais exagérément simpliste. Néanmoins, les études de simulation ont montré qu ignorer la corrélation intra-étude v

6 n a pas une effet dramatique sur l inférence du modèle bivarié. Les deux approches multivariées ne convenaient pas aux études de simulation, ce qui met l accent sur la complexité des modèles. Nous recommandons de méta-analyser les données IPD sur la précision du diagnostique via l approche bivariée ou le plus complexe modèle ordinal multivarié. vi

7 TABLE OF CONTENTS ACKNOWLEDGEMENTS ABSTRACT ABRÉGÉ LIST OF TABLES LIST OF FIGURES ii iii v ix x 1 Introduction Background Diagnostic Accuracy Measures Traditional and Individual Patient Data Meta-Analysis Traditional meta-analysis Individual Patient Data meta-analysis Meta-analysis of diagnostic accuracy studies Models Bivariate random-effects model Ordinal multivariate random-effects model Normal multivariate random-effects model Comparison of proposed methods Assumptions Practical interest Methodology Objective Motivating examples Motivating example 1: CAGE questionnaire Motivating example 2: PHQ-9 questionnaire Methods vii

8 4 Results CAGE questionnaire: results PHQ-9 questionnaire: results Implementation in statistical software Simulation Discussion Diagnostic accuracy of the CAGE and the PHQ-9 questionnaires Related work Strengths and limitations Conclusion Appendix A Appendix B Appendix C References viii

9 Table LIST OF TABLES page 2 1 Reporting results of a diagnostic accuracy study for a given threshold, 2 2 table The IPD CAGE dataset Estimated parameters by the three methods: CAGE Estimated sensitivities and specificities by the three methods: CAGE The IPD PHQ-9 dataset Estimated parameters by the two methods: PHQ Estimated sensitivities and specificities by the three methods: PHQ Simulation results for scenarios 1 and Simulation results statistical analysis: bias of the pooled estimates Simulation results for scenarios 3 and Simulation results for scenarios 5 and Simulation results for scenarios 7 and Simulation results for scenarios 9 and Simulation results for scenarios 11 and Simulation results for scenarios 13 and Simulation results for scenarios 15 and ix

10 Figure LIST OF FIGURES page 4 1 Individual ROC curves: CAGE Pooled sroc curves from the three methods: CAGE Pooled sroc curves with their 95% confidence band: CAGE Individual ROC curves: PHQ Pooled sroc curves from the three methods: CAGE Pooled sroc curves with their 95% confidence band: CAGE x

11 CHAPTER 1 Introduction The use of diagnostic 1 tests in patients is of interest both for clinicians and health-policy makers [34]. On one hand, clinicians need to decide whether to use a diagnostic test and how to interpret it. On the other hand, health-policy makers need to assess the value of a diagnostic test compared to other available tests, and decide whether to make it available. The evaluation of the diagnostic accuracy of a test is an important step in this decision-making. All available published information on the accuracy of a test need to be properly accounted for to base decision-making on the overall value of diagnostic accuracy [15]. Considering this setting, systematic reviews and meta-analyses are well-established methods to rigorously combine information from multiple studies. Guidelines on how to conduct a systematic review in general [9, 38, 51] or specifically for diagnostic accuracy are discussed elsewhere [34, 39, 42]. Individual patient data (IPD) meta-analyses are increasingly common in the literature [7, 13, 39, 63, 70]. The use of IPD has many benefits over traditional meta-analyses of published data. In the context of diagnostic accuracy studies, a pair of sensitivity and specificity is usually reported for a given threshold and a 1 The tests we will consider in this work are actually screening tests. 1

12 traditional meta-analysis is conducted to obtain a pooled result for that threshold [33, 49, 52, 58, 67]. However, primary studies might report pairs of sensitivity and specificity for the different sets of thresholds and selective reporting of thresholds might lead to inaccurate conclusions [43]. When IPD are available for diagnostic studies, sensitivity and specificity can be estimated for every possible threshold of a test [37, 62]. One might analyze the data threshold by threshold. However, such approach ignores the correlation between sensitivities and specificities across thresholds within a study [66]. Alternatively, one might analyze data for all thresholds simultaneously and thus take into account the complex structure of the data. Such methods are referred to IPD meta-analysis methods [37, 63]. Few papers have considered analyzing multiple thresholds simultaneously in the context of IPD meta-analyses in the last 15 years [24, 37, 56, 66]. The objective of this work is 1) to understand current statistical challenges in IPD meta-analyses of diagnostic accuracy studies, 2) to explore the potential statistical benefits of IPD meta-analyses of diagnostic accuracy studies, 3) to empirically compare available meta-analytic methods of diagnostic studies when IPD are available and 4) to compare the validity of the inference across meta-analytic methods via simulation studies. In order to achieve these goals, a literature review will highlight the statistical challenges in this area. Two empirical comparisons and a simulation study will be perform to assess the performance of each method. In Chapter 2, important background concepts of diagnostic accuracy studies will be defined. The concepts of meta-analysis and IPD meta-analysis will be explained. Three existing IPD meta-analytic models will be presented and compared 2

13 in terms of assumptions and applicability. In Chapter 3, two motivating examples will be introduced, and our methods will be explained. In Chapter 4, results from the two empirical comparisons will be shown. Simulation results will be presented in Chapter 5 and Chapter 6 will be dedicated to a thoughtful discussion around these comparisons. 3

14 CHAPTER 2 Background 2.1 Diagnostic Accuracy Measures Diagnostic accuracy refers to the ability of a test to distinguish between diseased and non-diseased patients, where the disease status has to be clearly defined [34]. The objective of a diagnostic accuracy study is to evaluate the performance of a new test compared to the gold standard test. The gold standard gives the true disease status of a patient. Here, we make the assumption that it has a perfect diagnostic accuracy. However, the gold standard test can be expensive, complex and invasive, which motivate the development of a screening test. Therefore, to evaluate the diagnostic accuracy of this test, the screening test result is compared to the true disease status as obtained by the gold standard. In clinical practice, health care providers are interested in the predictive values of a test. The positive predictive value (PPV) refers to the probability that a patient with a positive test result will be diseased whereas the negative predictive value (NPV) refers to the probability that a patient with a negative test will not have the disease [17]. However, PPV and NPV depend on disease prevalence in the studied population [17]. Sensitivity and specificity are closely related to PPV and NPV. Sensitivity is the probability that a test result will be positive given that a patient is truly diseased whereas specificity is the probability that a test result will be negative 4

15 given that a patient is truly not diseased. Even if they are less clinically relevant, sensitivity and specificity are usually reported to assess the diagnostic accuracy of a test since both measures are independent of disease prevalence [17]. From estimates of sensitivity and specificity, one can derive PPV and NPV to obtain clinically relevant measures. Of interest here are binary classifier tests such that the test result is positive (T+) or negative (T-), whereas the true disease status of a patient is diseased (D+) or non-diseased (D-). For tests that are not binary by definition (ordinal or continuous tests), a threshold has to be defined to dichotomize it. Define x k as the test result for patient k and j as a threshold. Then, for tests where higher scores are associated with the presence of the disease, x k < j = T- x k j = T+ For continuous tests, there exist an infinite number of thresholds ranging through all possible values taken by the test. For ordinal tests with J ordered categories, there exist J-1 thresholds [24]. In this work, we will focus on tests with ordered categories. For every threshold investigated, a diagnostic accuracy study reports a 2 2 contingency table showing the distribution of truly diseased and truly non-diseased patients with respect to their test results as in Table

16 Table 2 1: Reporting results of a diagnostic accuracy study for a given threshold, 2 2 table Disease status Test Result D+ D- Total T+ TP FP POS T- FN TN NEG Total n 1 n 0 n Here, n 1 and n 0 are the total number of truly diseased and truly non-diseased patients in the study, respectively, where the sample size n is n = n 1 +n 0. POS and NEG are the number of patients with a positive and negative test result, respectively. TP, FP, FN and TN respectively refer to the number of true positive, false positive, false negative and true negative patients in a study. From this table, estimates of sensitivity and specificity when considering threshold j are derived as Sensitivity j = TP n 1 and Specificity j = TN n 0 Sensitivity and specificity of a test are closely related to the chosen threshold: for a test with an increasing scale, as the threshold value increases, specificity increases while sensitivity decreases [58]. This relation between sensitivity and specificity is clearly depicted in the receiver operating characteristic (ROC) curve. The ROC curve plots pairs of sensitivity and (1-specificity) over the range of all possible thresholds of a test. The ROC curve visually describes the overall diagnostic accuracy of a test. From the ROC curve, one can compute the area under the curve (AUC) to quantify diagnostic accuracy [28]. The AUC is equivalent to the probability that a randomly chosen individual from the truly diseased sample has a greater test value 6

17 than a randomly chosen individual from the truly non-diseased sample [28]. AUC scores range from 0.5 to 1. A non-informative test has an AUC of 0.5 with its corresponding ROC curve lying on the diagonal line of the graph. A perfect test has an AUC of 1.0 with a ROC curve following the left and upper boundary of the graph. The AUC summarizes the whole ROC curve by giving the same weighting to the full range of threshold values [23, 79]. In practice, one might want to focus on a range of sensitivity (specificity) of clinical importance by giving larger weights to these values. Interpretation and comparison of AUCs must consider this weighting scheme [23]. The disadvantage of summarizing the diagnostic accuracy of a test in a single measure is that it does not distinguish between the ability of correctly detecting diseased patients (sensitivity) and non-diseased patients (specificity) [58]. Other diagnostic accuracy measures exist (positive and negative likelihood ratios, diagnostic odds ratio) [17], but will not be considered here. 2.2 Traditional and Individual Patient Data Meta-Analysis Traditional meta-analysis A meta-analysis is a statistical tool to combine results from independent studies addressing a specific clinical question [51]. Traditional meta-analyses pool published data to produce summary estimates of the effect of interest along with its associated uncertainty [51, 63]. Here, effect can refer to any measure of association between an exposure and an outcome (e.g. risk difference, odds ratio, risk ratio, proportions difference, means difference, etc). By pooling results across studies, a meta-analysis essentially computes a weighted average [51]. 7

18 A meta-analysis is usually the last step of a systematic review, but all systematic reviews do not necessarily include a meta-analysis [38, 51]. A systematic review is a method motivated by the need for healthcare decisions and policy making to rely on strong evidence from high-quality studies. It includes a comprehensive and exhaustive search of primary studies reporting results on a specific research question. It further assesses the quality of primary studies based on pre-established criteria of reproducibility and clearness. Results from selected studies are then synthesized using explicit methods, such as a meta-analysis, to produce strong evidence on the question of interest. Key requirements to conduct a meta-analysis include that selected studies report quantitative summary estimates of the effect of interest [51]. However, differences between studies are common [22, 51] and occur because of clinical heterogeneity (differences in population, outcome definition, intervention definition) and methodological heterogeneity (differences in study design or quality) [58]. These variations in study characteristics are likely to influence study-specific effect estimates. Heterogeneity should be investigated in order to increase the clinical relevance and the quality of conclusions drawn from a systematic review [9, 22, 51]. Visual assessment of heterogeneity (e.g. forrest plot, l Abbé plot) or formal statistical tests for heterogeneity (e.g. I 2 ) [38, 51] can be used for such investigation. Meta-analytic statistical methods are in fact based on the interpretation of the observed differences between results from primary studies i.e. heterogeneity. Three schools of thought may characterize this interpretation. First, one could believe that 8

19 the same effect is present in all included studies, and that observed differences between study estimates are only due to random variation. Such a belief motivates the fixed-effect model [51]. This first interpretation of observed differences across studies is highly unrealistic. Second, one could believe that primary studies estimate slightly different effects: each study estimates an underlying true study-specific effect and all true study-specific effects vary around an overall average effect. The observed differences between study-specific estimates are then a mix of two sources of variations: between-study variability given that each study is not thought to estimate the same effect and random variability. This interpretation of observed variations yields to the random-effects model [51]. This model is more plausible since it makes no assumption on the true underlying study-specific effects. Still, both the fixed- and random-effects models make several distributional assumptions [30]. Finally, one could use predefined guidelines imposed by formal statistical test for heterogeneity, and rely on these test results to decide whether to use a fixed- or random-effects model. Notwithstanding, the random-effects model remains the most realistic and plausible and should be preferred most of the time [6, 51, 68]. Traditional meta-analyses have several limitations. Publication bias occurs when statistically significant positive results are more likely to be published than studies finding no statistically significant results [13, 51]. A meta-analysis prone to publication bias might find exaggerated overall estimates. Graphical methods (e.g. funnel plot) and statistical tests exist to check the presence of publication bias [51]. Also, traditional meta-analyses highly depend on the quality of included studies or the quality of reporting [7]. Finally, subgroup analyses are very limited in conventional 9

20 meta-analyses. Most primary studies simply report overall estimates of the effect of interest and might not even be powerful enough to estimate subgroup-specific effects [7] Individual Patient Data meta-analysis An alternative way to conduct a meta-analysis is to obtain and use individual patient data (IPD) instead of published estimates of effect [63]. This alternative meta-analytic method is referred to as individual patient or participant data metaanalysis. IPD are the raw individual-level data recorded for each participant in a study whereas published data or aggregated data consists of study-level summary measures derived from these raw data [13, 63]. From IPD, one can obtain published results but published results usually do not allow recovering raw individual patient data. The objective of an IPD meta-analysis is still to summarize results from independent studies. The statistical method used to meta-analyze IPD must account for clustering of patients within studies, and never analyze the data as if all patients were coming from one single study [63]. As with traditional meta-analyses, differences between studies can occur due to clinical or methodological heterogeneity and these differences should be accounted by the statistical analysis. IPD meta-analyses are performed using two statistical analyses, both retaining study clustering and allowing fixed-effects or random-effects modelling: two-stage and one-stage approaches [13, 63]. In the two-stage approach, IPD are first summarized in each study separately and independently using an adequate statistical 10

21 method (e.g. logistic regression). Aggregated data, i.e. study-level estimates of the effect of interest along with a measure of their variability, are then available for each study. These aggregated data are combined using an appropriate traditional meta-analysis method to produce a pooled summary effect. This first approach is laborious in the sense that it implies re-analyzing data for each study and conducting a traditional meta-analysis [63, 69]. In the one-stage approach, IPD from all studies are simultaneously analyzed in a single step while accounting for clustering of patients by study. In contrast with the two-stage approach, study-level effects are not explicitly estimated for each study [63]. However, model specification and clear distinction between the sources of variability make the one-stage approach more complex for non-statisticians [63, 69]. The two-stage approach is the most popular method to conduct IPD meta-analyses [69, 72]. However, the one-stage approach is a more flexible method and allows accounting for both within- and between-study effects [63]. IPD meta-analyses have many advantages over traditional aggregated data metaanalyses [13, 63]. Aggregated data are often poorly reported and statistical methods more or less clearly detailed [69]. Aggregated data can be presented differently from one study to the other, say relative risk versus odds ratio. The availability of IPD counterbalances some of these disadvantages. IPD meta-analyses ease the standardization of the statistical analysis performed within each study [63, 69] and allow directly deriving the study-level information of interest [63]. Standardization includes consistency in dealing with missing data, consistency in modelling techniques used across studies, ability to control for inclusion and exclusion criteria, consistency 11

22 in outcome and exposure definitions and ability to adjust for the same set of covariates [63, 69], among others. In other words, the use of IPD controls for a part of the factors affecting between-study heterogeneity. Furthermore, IPD permit using information unavailable from publications [13]. In the context of diagnostic accuracy studies, IPD meta-analyses have the advantage to control for selective reporting of thresholds [43, 61]. Selective reporting of thresholds is a form of selection bias. It occurs because it is more likely that authors will report results from thresholds associated with higher sensitivity and/or specificity. A traditional meta-analysis that relies on results from these reported thresholds may produce biased (too high) pooled sensitivity and specificity due to this selective reporting [43, 61]. By using raw data, IPD meta-analyses rely on results from all thresholds and are not affected by selective reporting of thresholds. IPD meta-analyses also come with disadvantages [63]. Conducting an IPD meta-analysis can be much more time consuming and expensive than traditional meta-analyses. IPD meta-analyses require contacting study authors, obtaining their data, standardizing all acquired datasets and going through laborious ethic procedures. However, not all study authors are willing or able to share their raw data. This partial availability might introduce a bias in the IPD meta-analysis if it is associated with the study results [63, 69, 70]. Methods exist to meta-analyze aggregated data with IPD data [62, 65, 71] such that the potential partial availability bias can be counterbalanced. Also, the quality of an IPD meta-analysis still depends on the quality of included studies. An IPD meta-analysis based on poor quality primary studies may be as defective as a traditional meta-analysis based on these studies 12

23 [63]. Finally, IPD meta-analyses are still prone to publication bias since systematic reviews based on IPD are not meant to seek unpublished sources more than traditional meta-analyses [63, 70] Meta-analysis of diagnostic accuracy studies In the context of diagnostic accuracy studies, traditional and IPD meta-analyses present particular statistical challenges. Traditional meta-analytic methods of diagnostic accuracy studies depend on the type of published data available [24]. The most commonly reported measure is a pair of sensitivity and specificity of a test for a single threshold [24] and we will focus on this situation. In this context, traditional meta-analyses summarize pairs of sensitivity and specificity for that specific threshold across studies to produce a bivariate summary estimate. Such a meta-analysis should account for the correlation between sensitivity and specificity across studies. This correlation can be explained by a threshold effect [58, 60], which is closely related to clinical and methodological heterogeneity. The threshold effect can be explicit: if studies report results for different thresholds, everything else being kept equal, sensitivity and specificity are expected to be different [49]. However, if studies report results for a common threshold, implicit variations still arise due to differences in measurements, observers, equipments, laboratories or other methodological factors [49, 58]. Implicit variations are also a consequence of differences in study population say clinical heterogeneity [14, 49]. Unlike other sources of variation, the threshold effect has a specific effect on 13

24 the relationship between sensitivity and specificity across studies. Assume that an increasing score on a test is associated with a more severe disease. Lowering the threshold allows more patients with a positive result, and therefore catches more true positive patients but also more false positive patients. Lowering the threshold then results in increasing sensitivity at the expense of decreasing specificity [49, 58]. Similarly, making the threshold stricter allows fewer patients with a positive result and leads to increasing specificity at the expense of decreasing sensitivity. This tradeoff between sensitivity and specificity is fully captured in the ROC curve [49, 58]. Therefore, sensitivity and specificity will negatively correlated across studies since they mutually depend on a threshold [60]. This correlation between sensitivity and specificity across studies when considering a specific threshold is referred to as between-study correlation. Meta-analytic methods for such data should also consider the possible association between sensitivity and specificity within a study, say within-study correlation [59]. In the context of diagnostic accuracy when we focus on one specific threshold, it is reasonable to assume that within-study correlation is zero given that sensitivity and specificity estimates are derived from two independent populations, diseased and non-diseased patients, respectively [45, 59]. Simulation studies and empirical comparisons of traditional meta-analyses of diagnostic accuracy studies have shown that ignoring the within- and between-study correlation of sensitivity and specificity for a given threshold by analyzing them via two separate univariate meta-analyses is inappropriate [29, 49, 59]. The popular bivariate random-effects model is recommended in the current literature of traditional 14

25 meta-analyses methods of diagnostic accuracy studies [14, 24, 29, 50, 61]. This model will be described in detail in section 2.3. When IPD are available, there are two ways to conduct a meta-analysis of diagnostic accuracy [62]. First, one could focus on a particular threshold of interest. From IPD, it is possible to estimate sensitivity and specificity for that specific threshold for all studies and conduct a meta-analysis to obtain a pair of pooled sensitivity and specificity. This method is useful when there already exists a well-established threshold for this diagnostic test [62]. Second, IPD allow varying the threshold across all possible values of a test. For a continuous scale test, one could then estimate a ROC curve for each study and conduct a meta-analysis to obtain a pooled ROC curve [37]. For an ordinal scale test, varying the threshold allows estimating a pair of sensitivity and specificity for all possible thresholds [24]. A meta-analysis then produces pooled pairs of sensitivity and specificity for all possible thresholds. Methods to meta-analyze results from all thresholds should account for correlation between sensitivities and specificities across studies as well as their within study correlation. Between-study correlation of sensitivity and specificity still arise due to the effect of the threshold, as in traditional meta-analyses. This correlation should be taken into account by modelling sensitivities and specificities for all thresholds simultaneously. Given how sensitivities (specificities) are estimated, we also expect sensitivities (specificities) to be correlated across thresholds within a study [56]. In traditional meta-analyses, the within-study correlation only concerned the possible correlation 15

26 between sensitivity and specificity for a given threshold and this correlation was estimated by zero. However, when considering results from all thresholds, sensitivities (specificities) for different thresholds within a study are derived from the same observed results of the same diseased (healthy) population using different rules. Therefore, the within-study correlation between sensitivities for different thresholds is not zero anymore. Similarly, specificities for different thresholds within a study will be correlated. However, it is still reasonable to think that sensitivities are not correlated with specificities within a study because their estimates are derived from two independent groups of patients. Methods have been recently developed to meta-analyze results from multiple thresholds simultaneously. The ordinal multivariate random-effects model from Hamza et al. [24] is an extension of the bivariate random-effects model when results are available for the same set of thresholds across studies. This model will be described in Section 2.3. Another approach introduced by Riley et al. [66] considers the analysis of continuous scale tests when results are not reported for the same set of thresholds. This method is a missing data approach where results from missing thresholds are assumed to be missing at random i.e. the method is not designed to deal with selective reporting of thresholds. Other methods have been proposed [5, 15, 37, 55, 56]. We do not consider these methods for various reasons. Two of these methods [5, 55] do not directly concern diagnostic accuracy studies and are designed for diseases with more than two stages. Dukic et al. [15] develop a model in a bayesian framework and we do not consider this approach. Kester et al. s model [37] is similar to the ordinal multivariate random-effects model so we decide to consider the most recent 16

27 version of it. Putter et al. [56] propose a very interesting model using the framework of survival analysis. 2.3 Models IPD meta-analyses of diagnostic accuracy studies can be conducted via three methods. We will now thoroughly describe these methods Bivariate random-effects model The bivariate random-effects model(brem) was first introduced by Van Houwelingen et al. [75] in the context of meta-analyses of two-dimensional treatment groups in clinical trials. In the context of diagnostic accuracy studies, the BREM applies when each primary study reports a pair of sensitivity and specificity for a specific threshold J. Let the subscript i identify primary studies included in a meta-analysis, i = 1,...,I. The bivariate random-effects model was first described in the framework of linear mixed model (LMM) [58, 74]. The logit transformed sensitivities from primary studies are assumed to be approximately normally distributed around a pooled mean value with a certain amount of variability around this mean [58]. This is a randomeffects approach: differences in study population and any other implicit variations are incorporated in the model via the variability assumed around the pooled logit sensitivity. A similar reasoning is applied for modelling the specificities. The two normally distributed outcomes are modelled simultaneously to account for possible between-study and within-study correlations, leading to the bivariate normal 17

28 random-effects model. Define the parameters η ij as the true logit sensitivity and ξ ij as the true logit (1-specificity) for threshold J in study i. We assume that the true logit sensitivities from the primary studies follow a normal distribution centered around the pooled logit sensitivity η J with between-study variance ση. 2 Similarly, we assume that the true logit (1-specificities) from the primary studies follow a normal distribution centered around the pooled logit (1-specifity) ξ J with between-study variance σξ 2. Given the random effects u iηj and u iξj, the observed logit sensitivity ˆη ij and the observed logit (1-specificity) ˆξ ij in study i follow a bivariate normal distribution ˆξ ijξ ˆη ij N η ij,c i with C i = ξ ij s2 η J 0 0 s 2 ξ J where C i is the within-study covariance matrix. As mentioned before, it is assumed that the correlation between sensitivity and (1-specificity) within a study is zero given that they are estimated from two independent populations, which explains the off-diagonal terms in C i. Using the identity link, E(ˆη ij E(ˆξ ijξ u iηj ) = η ij = η J +u iηj u iξj ) = ξ ij = ξ J +u iξj where the random effects u iηj and u iξj are allowed to be correlated with bivariate normal distribution as 18

29 u iηj N 0, σ2 ηj u iξj 0 ρ J σ ηj σ ξj ρ J σ ηj σ ξj (2.1) σ 2 ξj We thus have σηj 2,σ2 ξj and ρ J denoting the between-study variability of logit sensitivity, the between-study variability of logit (1-specificity) and the between-study correlation of logit sensitivity and logit (1-specificity), respectively. Of interest is to estimate the pooled logit sensitivity η J and the pooled logit (1-specificity) ξ J along with their between-study variability and correlation. Chu and Cole [12] proposed to extend the normal bivariate model to accommodate non-continuous outcomes. Instead of assuming a bivariate normal distribution for the logit sensitivity and logit (1-specificity) within a study, they used the exact binomial distribution to model the number of true positive and true negatives patients in each study. Define TP ij and FP ij as the number of true positive and false positive patients, respectively, in study i when using threshold J. Define n i1 and n i0 as the total number of truly diseased and truly non-diseased patients in study i, respectively. Given the random effects u iη and u iξ, TP ij and FP ij follow two independent binomial distributions TP ij Binomial(n i1,expit(η ij )) FP ij Binomial(n i0,expit(ξ ij )) 19

30 and ( logit E ( logit E ( TPiJ n ( i1 FPiJ n i0 u iηj u iξj )) = η ij = η J +u iηj (2.2) )) = ξ ij = ξ J +u iξj where the random effects u iηj and u iξj come from a bivariate normal distribution with non-zero covariance as (2.1). Study-level covariates (e.g. type of population, mean age, type of diagnostic test, publication year) can be incorporated to the BREM model [12, 26, 58] by replacing η J and ξ J in (2.2) by η J = α 0 +β 1 X β p X p ξ J = α 0 +β 1 X β p X p where X 1,...,X p are covariates measured at the study-level. The linear combination can be adapted to the situation where a covariate is only available for the diseased or the non-diseased patients across studies or where a covariate is not available for a specific study [26]. The covariance matrix of the random effects (2.1) can also depend on the covariates [26]. Simulation studies [12, 27] and empirical comparisons [25, 27] have shown that the generalized bivariate random-effects model is preferable to the normal bivariate model. The normal bivariate random-effects model is biased because the binomial within-study exact distribution is approximated by the normal distribution [12, 27]. 20

31 This approximation is more problematic when sensitivity or specificity are close to one or zero [27]. Further references to the BREM will be for the model based on the binomial distribution. The BREM is traditionally used to meta-analyze pairs of sensitivity and specificity for a specific threshold. Methods have been developed to derive a summary ROC curve (sroc) from the pair of pooled estimates [3, 36, 49, 67]. These methods assume different parametrizations for the sroc curve and it is not clear which parametrization is preferable [3, 24]. For example, one could choose to regress ξ (logit(1-specificity)) on η (logit sensitivity). Since ξ and η are interchangeable, the regression line could as well be η on ξ [3, 24]. When the BREM is only applied to one threshold, the sroc curve make an assumption on the existence and on the shape of study specific ROC curves, which is untestable with published data only [24]. When IPD are available, the BREM may be applied threshold by threshold to produce a pooled pair of sensitivity and specificity for all possible thresholds. From these pooled estimates, a sroc curve can as well be derived but the choice of parametrization for the regression line remains a problem. However, the resulting sroc curve has the advantage of being identifiable since it relies on results from multiple thresholds [24]. Study- or patient-level covariates may be incorporated in thebremandthesemayleadtosroccurvesthatvarybylevelofthecovariate [26] Ordinal multivariate random-effects model The ordinal multivariate random-effects model (ordinal MREM) was introduced by Hamza et al. [24] precisely in the context of meta-analysis of diagnostic accuracy 21

32 studies. Their model is very similar to a method developed to meta-analyze ROC curves [37]. The ordinal MREM applies when all primary studies report pairs of sensitivity and specificity for the same set of multiple thresholds. This method is designed to support ordinal scale tests. Again, let the subscript i identify primary studies included in a meta-analysis, i = 1,...,I. The ordinal MREM is a direct extension of the exact binomial BREM when results are reported in more than one category [24]. Roughly, the model is based on the following lines of reasoning. Consider an ordinal test with J categories, corresponding to J-1 (relevant) thresholds. Within each study, the distribution of truly diseased patients across the test s categories follows a multinomial distribution. The corresponding probability parameters are functions of the J-1 true study-specific sensitivities. The random-effects approach assumes that these true sensitivities are drawn from a multivariate normal distribution centered around the J-1 pooled sensitivities. Reasoning is similar for the distribution of truly non-diseased patients (specificities) for the J-1 thresholds. The multinomial within-study modelling accounts for correlation across thresholds within study. Furthermore, sensitivities and specificities are modelled simultaneously, and thus the model correctly accounts for the correlation between sensitivity and specificity across studies. Let the subscript j identify the threshold, j = 1,...,J 1, for a test with J categories. Define η ij as the logit sensitivity and ξ ij as the logit (1-specificity) in study i with threshold j. Again, define n i1 and n i0 as the total number of truly diseased and truly non-diseased patients in study i, respectively. Finally, define x ij1 and x ij0 as the number of truly diseased and truly non-diseased patients in study 22

33 i with test results falling in category j, respectively. For a given threshold j, the observed sensitivity and specificity are respectively given by j k=1 x ik1 n i1 j k=1 and 1 x ik0 n i0 Given the random effects, the within-study model assumes that the number of diseasedsubjects(x i11,...,x ij1 )andthenumberofnon-diseasedsubjects(x i10,...,x ij0 ) follow two independent multinomial distributions with parameters (n i1 ;π i11,...,π ij1 ) and (n i0 ;π i10,...,π ij0 ), respectively. The probability parameters π ij1 are functions of the logit sensitivities η ij as ξ ij as π ij1 = exp(η ij ) 1+exp(η ij ), for j=1 exp(η ij ) 1+exp(η ij ) exp(η i,j 1) 1+exp(η i,j 1 ) 1 exp(η i,j 1) 1+exp(η i,j 1 ), for j=2,...,j-1, for j=j Similarly,theprobabilityparametersπ ij0 arefunctionsofthelogit(1-specificities) π ij0 = exp(ξ ij ) 1+exp(ξ ij ), for j=1 exp(ξ ij ) 1+exp(ξ ij ) exp(ξ i,j 1) 1+exp(ξ i,j 1 ) 1 exp(ξ i,j 1) 1+exp(ξ i,j 1 ), for j=2,...,j-1, for j=j The between-study model is specified in two steps. First, a random intercept/fixed slope linear relationship is assumed between ξ ij and η ij : η ij = α+u iα +βξ ij (2.3) 23

34 where the random effects u iα follow a normal distribution centered around zero with variance σα 2 and β is a fixed effect. It is also possible to introduce a random-effect term u iβ for the slope [24]. This linear relationship assumption is motivated by the necessity of producing a pooled ROC curve as well as simplifying the model [24]. Second, since test categories have a natural order, the proportional odds logit model [2, 5] is used to link ξ ij to the linear predictor as log ( j k=1 π ik1 J k=j+1 π ik1 i,δ ij ) = ξ ij = ξ j + i +δ ij (2.4) where i denotes the random effects associated to study clustering and δ ij denotes the random effects associated with the threshold value j, j = 1,...,J 1, within study [54]. It is assumed that the i follow a normal distribution as i N(0,σ 2). At this point, several assumptions are made on the distributions of the random effects u iα, i and δ ij in order to simplify the model. The i are allowed to be correlated with u iα, and their covariance is given by σ α. The δ ij s are assumed to be independent and identically distributed (i.i.d) with a normal distribution given by δ ij N(0,σδ 2). It is further assumed that the δ ijs are independent from i and u iα. The resulting multivariate normal distribution of the random effects is given by 24

35 u iα i +δ i1 i +δ i2. σα 2 σ α σ α σ α σ 2 +σ2 δ σ 2 σ2 N 0,. σ σ 2 i +δ i,j 1 σ α σ 2 σ 2 σ2 +σ2 δ Previous assumptions made on the random effects define a compound symmetric covariance structure. The δ ij being i.i.d., the variance of i + δ ij is constant for j = 1,...,J 1. Furthermore, since i and δ ij are assumed to be independent, we have Cov( i +δ ik, i +δ il ) = Var( i )+Cov( i,δ ik )+Cov( i,δ il )+Cov(δ il,δ ik ) = Var( i ) = σ 2 for l k, l,k = 1,2,...,J 1. As a practical interpretation, the compound symmetric structure assumes that the correlation between any pair of logit (1-specificities) ξ ik and ξ il is equal, l k, l,k = 1,2,...,J 1. This assumption might not be very appropriate since it is expected that the correlation between consecutive thresholds may be larger compared to non-consecutive thresholds [24]. Ofinterestistoestimatethefixedeffectsparametersα, β and ξ 1,..., ξ J 1 aswell as the variance and covariance of the random-effects terms σα, 2 σ 2, σ2 δ and σ α. From these estimates, one can derive estimates of the pooled logit sensitivities η j = α+β ξ j 25

36 along with their standard errors. A sroc curve is directly derived from the model estimates α and β. The ordinal MREM also easily accommodates study-level covariates [24]. Studyorpatient-levelcovariatescanbeincorporatedbyreplacingη ij in(2.3)andξ ij in(2.4) by η ij = α i +u iα +βξ ij +β 1 X 1 + +β p X p ξ ij = β 1 X 1 + +β p X p + i +δ ij where X 1,...,X p are covariates measured at the study- or patient-level. One could also decide to only adjust η ij and test the interaction between the covariates and ξ ij [24]. Again, different levels of a covariate will produce different sroc curves [24] Normal multivariate random-effects model The normal multivariate random-effects model (normal MREM) was introduced by Riley et al. [66] in the context of meta-analysis of diagnostic accuracy studies. The model applies when primary studies report pairs of sensitivity and specificity for multiple thresholds, but not necessarily for the same sets of thresholds. It can be viewed as a missing data approach. The model will be described in the context of IPD, that is when results from all thresholds are available in all primary studies. Ordinal tests with a finite number of thresholds will be considered. Imputation of missing data is out of the scope of this project and is discussed elsewhere 26

37 [61, 66]. Again, let the subscript i identify primary studies included in a metaanalysis, i = 1,...,I, and j identify the threshold value, j = 1,...,J 1. The model is a direct extension of the normal bivariate random-effects model [58, 75] just as the ordinal MREM was a direct extension of the binomial bivariate random-effects model. The model is based on the following lines of reasoning. Within each primary study, the logit sensitivities and the logit (1-specificities) for all J-1 thresholds are assumed to be normally distributed around the true study-specific logit sensitivities η ij and logit (1-specificity) ξ ij. Following the framework of linear mixed model, these true logit sensitivities and logit (1-specificities) are assumed to be drawn from a normal distribution centered around the pooled logit sensitivities η j and pooled logit (1-specificities) ξ j, for j = 1,...,J 1. Define x ij1 and x ij0 as the number of truly diseased and truly non-diseased patients in study i with test results falling in category j, respectively. Within each study, the number of diseased subjects (x i11,...,x ij1 ) and the number of non-diseased subjects(x i10,...,x ij0 )follow twoindependentmultinomialdistributionswithparameters (n i1 ;π i11,...,π ij1 ) and (n i0 ;π i10,...,π ij0 ), respectively. At this point, instead of using the framework of generalized linear mixed model (GLMM), the authors suggest to estimate the probability parameters of the two distributions by maximum likelihood. The multinomial likelihoods are given by 27

38 ( ) J 1 xij1 l i (1) (π i11 ) x i11 (π i21 ) x i21 (π i(j 1)1 ) x i(j 1)1 1 π ik1 ( ) J 1 xij0 l i (0) (π i10 ) x i10 (π i20 ) x i20 (π i(j 1)0 ) x i(j 1)0 1 π ik0 Maximization of the log-likelihood can easily be done with any statistical software [66]. From the estimates of the π ij1 s and π ij0 s, we can subsequently derive estimates of the logit sensitivity ˆη ij at threshold j in study i by ( ) j ˆη ij = logit 1 ˆπ ik1 and, similarly, estimates of the logit (1-specificity) at threshold j in study i by ( j ) ˆξ ij = logit ˆπ ik0. Estimates of the variance of the logit sensitivities s 2 ij1 and of their covariance s i(l,k)1, l k,l,k = 1,...,J 1, can be derived from estimates of the variance and covariance of the π ij1 s, for example using the delta method [8, 66]. Similarly, estimates of the variance of the logit (1-specificities) s 2 ij0 and of their covariance s i(l,k)0, l k,l,k = 1,...,J 1, can be derived from estimates of the variance and covariance of the π ij0 s. It is impossible to obtain an estimate of π ij1 (or π ij0 ) if there are no truly diseased (or truly non-diseased) patients with test result in category j [66]. The logit sensitivity (or logit (1-specificity)) estimated with that threshold j will then be equal to the estimated logit sensitivity (or 1-specificity) from the previous threshold k=1 k=1 k=1 k=1 28