A novel approach to assess diagnostic test and observer agreement for ordinal data. Zheng Zhang Emory University Atlanta, GA 30322

Transcription

1 A novel approach to assess diagnostic test and observer agreement for ordinal data Zheng Zhang Emory University Atlanta, GA Corresponding address: Zheng Zhang, Ph. D. Department of Biostatistics Rollins School of Public Health Emory University 1518 Clifton Road NE Atlanta, GA Tel: (404) Fax: (404)

2 Abstract Statistical evaluation of medical diagnostic tests traditionally focuses on the cases when the disease classification is of two states (healthy vs. diseased). The two-state disease classification results from a binary gold standard. However, when the gold standard tests are ordinal, we have a multi-state disease classification system. There is a lack of suitable statistical methods to evaluate diagnostic test performance when the disease classification is more than two states, especially when the investigational test is also ordinal. This manuscript develops a novel method to assess the performance of ordinal tests when the gold standard is also ordinal. We propose two conditional probability-based indices to evaluate test performance which are averages of sensitivities and specificities at different cut-off points. Furthermore, the ratio and average of the two new indices are used to measure observer agreement when the data are ordinal. An estimating procedure is presented and simulation studies are used to investigate the small sample behavior of these indices. We illustrate our method via two real data examples. KEY WORDS: ordinal test; diagnostic test; observer agreement; kappa statistic; sensitivity; specificity. 1 Introduction Accurate diagnosis of disease is essential for appropriate treatment and/or surveillance. Often the earlier a disease is diagnosed the more successful is its treatment. The rigorous evaluation of a diagnostic test is a high priority for research. Most research in this area focus on the two-state disease classification system where the patients are classified as diseased or not-diseased (healthy). Under this system, sensitivity and specificity are used to quantify the performance of binary tests and the ROC curve is used for ordinal or continuous tests. Briefly, sensitivity is defined as Pr(test positive diseased) and specificity is defined as Pr(test negative healthy). For ordinal and continuous tests, test positive is defined at each cut-off point c as Y > c where Y denotes test results, and sensitivity(c) is defined as P r(y > c diseased) and specificity(c) is defined as Pr(Y c healthy), then the ROC curve is defined as a plot of sensitivity(c) versus 1-specificity(c) for all possible values 1

3 of c. Two-state disease classification results from a binary gold standard, however, not all gold standard tests are binary, some of them are ordinal. With an ordinal gold standard test, we have a multi-state disease classification system. Under this system, the diagnostic test can be either ordinal or continuous. There have been a few studies attempting to evaluate a continuous test performance for multi-state disease classification [1,2]. In contrast, we are not aware of any study that targets the performance of ordinal tests for multi-state disease. This manuscript develops a novel method to assess the performance of ordinal tests when the gold standard test is also ordinal, specifically when the categories for both gold standard test and investigational test are the same. For example, when screening for cervical cancer, patients often are classified into four categories: no disease; low grade squamous intra-epithelial lesions (SIL); high grade SIL and invasive carcinoma histologically from colposcopy (a gold standard test for cervical cancer). Since colposcopy requires highly trained colposcopist and specific facilities, it is rarely done for population screening. Other inexpensive or more convenient tests are more likely to be used for screening, such as a Papanicolaou (Pap) smear (an ordinal test, generating five categories slightly different from those of colposcopy). Currently, the prevailing approach is to first collapse the four categories derived from colposcopy into two by combining no disease and low grade SIL or combining high grade SIL and invasive carcinoma, and then calculate sensitivity and specificity for the Pap smear. This simplistic approach only captures a partial picture of test performance and this practice of collapsing categories ignores valuable information concerning disease progression. Different grades of disease are usually associated with different levels of morbidity and disability. Thus, the consequences of misdiagnosing or misreporting the disease may be vastly dissimilar according to the stage of the disease. Misclassifying high grade SIL as none is a much graver mistake than misclassifying low grade SIL as none because there are fewer treatment options available to a woman with high grade SIL and less time to implement them. In addition, many medical studies are based on the outcome resulting from the diagnostic test as its major endpoint. Studies with an ordinal outcome will usually have greater statistical power to detect a significant effect whether it be for a treatment or a risk factor for the disease than corresponding studies with a binary outcome ( disease versus no disease ). Clearly, new methodology is needed to evaluate multi-class diagnostic 2

4 tests adequately. The evaluation of observer agreement when gold standard is available is closely related with the assessment of diagnostic tests when the categories are the same for both the gold standard test and the investigational test. For example, suppose we have two radiologists, each rates a set of images on an ordinal scale, further assume one of the radiologists is highly experienced and his/her ratings are treated as gold standard. We are interested in calculating the agreement between those two raters. In this setting, the kappa statistic [3] and the weighted kappa statistic [4] are the most popular indices for measuring agreement. The kappa statistic measures the agreement beyond random chance. As noted by several authors [5,6], the kappa statistic has a major drawback in that the magnitude of the statistic depends heavily on the raters marginal distributions of measurements. Situation arises when the kappa statistic suggests low agreement but the raw data suggests a high degree of agreement. We propose two conditional probability-based indices that could be applied to the evaluation of both diagnostic tests and observer agreements when the data are measured on the ordinal scale. In section 2 we present the definitions of two indices, the right agreement fraction(raf) and the left agreement fraction(laf) and two quantities derived from RAF and LAF, agreement ratio(ar) and spread (SP). Theoretical implications of those quantities are also presented in section 2. Associated estimating procedures appear in section 3. We will investigate the small sample behavior of those indices via simulation studies in section 4. We illustrate our method with a cervical cancer screening data and a cervical ectopy data in section 5. Section 6 provides a summary and some closing remarks. 2 Agreement Fractions and Agreement Ratio We propose to evaluate ordinal multi-class diagnostic tests by two new quantities, the right agreement fraction(raf) and the left agreement fraction(laf). RAF and LAF are the extensions of sensitivity and specificity for ordinal data. Consider a diagnostic test with K classes, let D be the result from the gold standard test and Y denote the result from the investigational test. For each fixed class k, k = 1, 2,...K 1, define the right agreement probability at k, RAP(k), as RAP(k) = P(Y > k D > k) (1) 3

5 and the left agreement probability at k, LAP(k), as LAP(k) = P(Y k D k) (2) We call the overall RAP(LAP) across all possible values of k as RAF(LAF) and define RAF and LAF as similarly, RAF = 1 K 1 RAP(k) = 1 K 1 P(Y > k D > k) (3) K 1 K 1 k=1 LAF = 1 K 1 LAP(k) = 1 K 1 P(Y k D k) (4) K 1 K 1 k=1 The above definitions imply that RAF and LAF are in fact the averages of sensitivities and specificities at different cut-off points. Notice if Y and D are both binary and take values from the set {1, 2}, RAF is equivalent to sensitivity and LAF is equivalent to specificity if D = 2 represents disease and Y = 2 represents test positive. Since agreement has two intrinsic components, namely, accuracy and precision, we want to derive some measures that intuitively address these two components. Lin et.al [7] suggests lack of accuracy reflects in disagreement between the two marginal distributions as location and/or scale shift, where precision is related with the magnitude of within-sample variation. To address the accuracy issue, we notice that when there is a location or mean shift in the two marginal distributions, there would be a discrepancy between RAF and LAF. We hereby propose a measurement named agreement ratio(ar), which is defined as k=1 k=1 AR = RAF/LAF (5) AR = 1 when there is no mean shift, AR > 1 if the readings have a mean shift to the right compared with the true values and AR < 1 if the readings have a mean shift to the left compared with the true values. In addition to AR, we propose another measurement named spread (SP) which is the average value of RAF and LAF, SP = (RAF + LAF)/2 (6) 4

6 SP reflects the random variation of Y away from D, which is related to precision. If Y and D have perfect agreement, RAF, LAF, AR and SP are all equal to 1. In addition, the situation where AR is close to 1 but SP is not implies good accuracy but poor precision. 3 Estimation Procedures Suppose both Y and D can take values 1, 2,...,K. The test results can be summarized in Table I. Table 1: Data from a hypothetical diagnostic test. Y = 1 Y = 2... Y = K Total D = 1 n 11 n n 1K n 1. D = 2 n 21 n n 2K n D = K n K1 n K2... n KK n K. Total n.1 n.2... n.k N The empirical estimators of RAF and LAF are: RAF = 1 K 1 K K i=k+1 j=k+1 n ij K 1 K k=1 i=k+1 n i. (7) LAF = 1 K 1 k k i=1 j=1 n ij K 1 k k=1 i=1 n i. We can estimate the agreement between the observed value and the true value by computing RAF, LAF, AR and SP estimates and the corresponding confidence intervals. The variance estimate can be obtained by a bootstrap process. The confidence interval can be calculated as a symmetric interval on either the original scale or an appropriate transformed scale, or as an asymmetric interval by using empirical quantiles. (8) 5

7 4 Simulation Studies 4.1 Consistency of the proposed estimators We simulate an ordinal data set where both Y and D have three levels, where they take values 1, 2 or 3. Assuming D has a multinomial distribution with probabilities (0.2,0.5,0.3) and the conditional distribution of Y given D are given in Table II. Table 2: Probability distribution of Y given D. Y = 1 Y = 2 Y = 3 D = D = D = The theoretical values are RAF = ,LAF = ,AR = ,SP = N is chosen to be 50, 100 and 200. The simulation results are listed in Table III which is based on 1000 runs. Table 3: Simulation studies. The values are bias (standard error) which are multiplied by 100. RAF LAF ÂR ŜP (7.4) 0.08(8.6) 0.91(15.4) -0.8(5.6) N (5.1) -0.07(6.1) 0.57(10.5) -1.0(3.9) (3.8) 0.02(4.3) 0.10(7.3) -0.6(2.9) The simulation results suggest that the empirical estimator is unbiased even for a modest sample size of 50. The biases are much smaller with a sample size of 100. Increasing sample size to 200 doesn t improve the estimators significantly. 4.2 Comparison with the Kappa Statistics We simulate two data sets, each represents the readings from two methods Y and D, and kappa statistics for both data sets are similar but it is obvious 6

8 the agreement between the two method are quite different. Table 4: Data from case I. Y = 1 Y = 2 Y = 3 Total D = D = D = Total Table 5: Data from case II. Y = 1 Y = 2 Y = 3 Total D = D = D = Total Table 6: Results from case I and II Case κ RAF LAF AR SP I II We can see although the kappa statistics for case I and II are very close, their AR values are very different, indicating although there is no systematic mean shift in case II, there is a substantial mean shift to the left in case I (reflected in LAF > RAF). 7

9 Table 7: Pap smear result. Pap smear Total No disease Low SIL High SIL Carcinoma No disease Gold Low SIL standard High SIL Carcinoma Total Data Analysis 5.1 Cervical cancer screening data To illustrate our proposed method, consider a cervical cancer screening data set, first published in Denny et al. [8]. The study was designed to evaluate alternative methods of cervical cancer screening for resource-poor settings. Seven hundred and fifty-two women ages years who had not been screened previously for cervical cancer, were not pregnant and had not undergone hysterectomy were recruited from Cape Town, South Africa for the study. Those women were screened using a Pap smear, an ordinal test. The gold standard test is colposcopy and cervical biopsy, which classifies the subject into four categories: no disease, low grade SIL, high grade SIL and invasive carcinoma. The Pap smear s categories are within normal limits, ASCUS, low grade SIL, high grade SIL and invasive carcinoma. For our purpose, we combine the first two categories of the Pap smear and call the combined category no disease. Hence the Pap smear result has four categories and has one-to-one correspondence with the gold standard test diagnosis. The result is shown in Table VII. Table VII can be reduced to three two-by-two tables by dichotomizing the test results at different cut-off points. The corresponding estimates of sensitivities and specificities are (0.583, 0.892), (0.579, 0.942) and (1.000, 0.991). By taking the averages, we calculated RAF =0.721(95% CI: (0.670, 0.768)), LAF=0.942(95% CI: (0.929, 0.953)), ÂR=0.765(95% CI: (0.709, 0.818)) and ŜP =0.831(95% CI: (0.807, 0.857)). The 95% confidence intervals are based on the empirical quantiles from 1000 bootstrap samples. Around 5% of boot- 8

10 strap samples have empty counts in the last category of carcinoma hence the RAF values can not be calculated. Here we simply ignored those samples. Since ÂR < 1, we conclude that the Pap smear result has a mean shift to the left compared with the gold standard, which means overall, Pap smear tends to score lower than the gold standard method. For comparison, we dichotomize the results from both Pap smear and gold standard test, combining no disease and low SIL into negative category and the last two categories into a new positive category and calculate sensitivity and specificity of the binary test, which yields sensitivity of 57.9% and specificity of 94.2%. Although the LAF estimate is close to the specificity value, the RAF estimate is much higher than the sensitivity estimate, which is resulted from the perfect classification of Pap smear method when the true category is carcinoma. We suspect the practice of dichotomization has under-estimated the classification power of Pap smear. 5.2 Cervical ectopy data Cervical ectopy refers to the presence of endocervical type columnar epithelium on the portio surface of the cervix. It occurs to a variable degree as a normal physiological process and is common in young women. The percentage of ectopy on the cervix is believed to be a possible risk factor for the heterosexual transmission of human immunodeficiency virus (HIV) infection in women, hence it is important to have a precise measurement of the amount of ectopy on a women s cervix. The data is obtained from a study [9] whose main objective is to develop a computerized planimetry method for measuring cervical ectopy and to compare the reliability of the new method with direct visual assessment (whose measurements will be treated as the target values). The outcome, The percentage of ectopy on the cervix, has been collapsed into four ordered categories (K = 4): minimal, moderate, large and excessive. There are 85 women without cervical disease in the study, whose images were assessed by three medical raters using both methods. We analyzed the data by calculating the agreement between the two methods overall and also for each rater individually. Data are displayed as a series of four-by-four tables and for each table, we calculated kappa statistic, RAF, LAF, AR and SP estimates as well as the 95% confidence intervals (empirical quantiles from bootstrap). Table VIII, IX, X and XI summarize the result. Table VIII shows that the agreement between the two methods are poor, there is a substantial mean shift to the left which means computerized planime- 9

11 Table 8: Cervical ectopy result from all raters. RAF = 0.556(0.484, 0.627), LAF = 0.905(0.865, 0.941), ÂR = 0.615(0.533, 0.701), ŜP = 0.731(0.687, 0.771) and κ = 0.271(0.185, 0.351). Computerized planimetry Total minimal moderate large excessive minimal visual moderate assessment large excessive Total Table 9: Cervical ectopy result from rater 1. RAF = 0.534(0.416, 0.643), LAF = 0.913(0.842, 0.979), ÂR = 0.584(0.450, 0.715), ŜP = 0.724(0.654, 0.788) and κ = 0.270(0.124, 0.413). Computerized planimetry Total minimal moderate large excessive minimal visual moderate assessment large excessive Total try tends to score much lower than the visual assessment. Table IX to XI show that between three raters, measurements from rater 2 offer the best agreement between the two methods. 6 Discussion We proposed a novel method to assess ordinal diagnostic test performance when the disease classification is of more than two classes, specifically when the categories for both diagnostic test and gold standard test are the same. The proposed indices are averages of sensitivities and specificities at different 10

12 Table 10: Cervical ectopy result from rater 2. RAF = 0.732(0.592, 0.842), LAF = 0.868(0.804, 0.929), ÂR = 0.843(0.671, 0.992), ŜP = 0.800(0.719, 0.863) and κ = 0.396(0.226, 0.552). Computerized planimetry Total minimal moderate large excessive minimal visual moderate assessment large excessive Total Table 11: Cervical ectopy result from rater 3. RAF = 0.464(0.323, 0.593), LAF = 0.972(0.917, 1.000), ÂR = 0.478(0.330, 0.614), ŜP = 0.718(0.640, 0.785) and κ = 0.154(0.016, 0.289). Computerized planimetry Total minimal moderate large excessive minimal visual moderate assessment large excessive Total cut-off points. The advantage of the proposed method is that we obtain separate sensitivity and specificity values and the proposed indices are just summary measures for those individual estimates. Here we define RAF and LAF as simple averages of different sensitivity and specificity values, however, it is possible to consider weighted averages of those values. The choice of weights warrants further research. Although we require the categories of the investigational test and the gold standard match, if the categories of the investigational test are more than that of the gold standard test, the proposed method can still be useful as long as one-to-one correspondence can be achieved by combining some of the 11

13 categories of the test. It is possible to extend our method to the unequal categories scenario although special care has to be taken for inference. Our method extends to observer agreement evaluation for ordinal data when gold standard is available. This method fills a gap in the medical diagnostic testing area and provides an alternative approach to assess observer agreement for ordinal data. We can also extend the method to assess observer agreement for continuous data (unpublished results). There are still a lot of issues remain here. For example, it is important to develop a formal procedure to compare agreement and to select the best when several methods are compared with a gold standard. We will also explore different ways to model AR and/or SP for covariate effects. Acknowledgement We would like to thank Drs. Michael Haber and Lance Waller from Emory University and Drs. Huiman X. Barnhart and Andrzej S. Kosinski from Duke University for helpful discussions and comments. References [1] Nakas CT, Yiannoutsos CT. Ordered multiple-class ROC analysis with continuous measurements. Statistics in Medicine 2004; 23: [2] Mossman D. Three-way ROCs. Medical Decision Making 1999; 19(1): [3] Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 1960; 20: [4] Cohen J. Weighted kappa: norminal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin 1968; 70: [5] Maclure M, Willett WC. Misinterpretation and misuse of the kappa statistic. American Journal of Epidemiology 1987; 126: [6] Feinstein AR, Cicchetti DV. High agreement but low kappa. Journal of Clinical Epidemiology 1990; 43: ,

14 [7] Lin L, Hedayat AS, Sinha B, Yang M. Statistical methods in assessing agreement: models, issues, and tools. Journal of the American Statistical Association 2002; 97(457): [8] Denny L, Kuhn L, Pollack A, Wainwright H, Wright TC. Evaluation of alternative methods of cervical cancer screening for resource-poor settings. Cancer 2000; 89(4): [9] Gilmour E, Ellerbrock TV, Koulos JP, Chiasson MA, Williamson JM, Kuhn L, Wright TC. Measuring cervical ectopy: direct visual assessment versus computerized planimetry. American Journal of Obstetrics and Gynecology 1997; 176: