Assessment of Interrater Agreement for Multiple Nominal Responses Among Several Raters Chul W. Ahn, City of Hope National Medical Center

Transcription

1 Assessment of Interrater Agreement for Multiple Nominal Responses Among Several Raters Chul W. Ahn, City of Hope National Medical Center ABSTRACT An interrater agreement coefficient is computed using a proportional overlap method in the general case in which multiple raters (not necessarily fixed in number) formulate a variable number of multiple diagnoses for each subject. Correction of chance agreement is performed based on the underlying assumptions of rater-specific a priori equal probabilities of diagnoses selection, and rater-specific a posteriori probabilities of diagnoses selection based on the observed marginal frequencies of rater responses. Two chance correction strategies are compared and illustrated using examples involving dental X-ray film readings and multiple psychiatric diagnoses. Introduction Evaluating reliability, the consistency with which a measure assesses a given trait, is an important part of medical research. It covers the broad spectrum of activities ranging from measuring consistency of diagnoses to determining the reliability of rating the presence/absence, levels of symptom, prognostic, and outcome characteristics, etc. It is often and justifiably found unacceptable to require that only one response category be assigned to one subject. To force a single response category is to require unrealistic and incomplete response. The other problem arises by limiting an equal number of multiple responses. There is an increasing demand to compute an interrater agreement index in the case in which a variable number of multiple raters formulate a variable number of diagnoses for each subject. For the following reasons, an interrater agreement index to deal with multiple raters and multiple responses is increasingly needed in psychiatry. (i) DSM-III (Diagnostic and Statistical Manual of Mental Disorders: Third Edition, 1980) and DSM-III-Revised (1987) encourage clinicians to formulate as many diagnoses needed to describe the subject s condition adequately. For example, DSM-III (p.24) and DSM-III-R (p.16) state that "on both Axis I and II multiple diagnoses should be made when necessary to describe the current condition." (ii) Clinical experience shows that duality and even higher multiplicity of recognizable psychiatric disorders are not infrequent phenomena. Comorbidity patterns are treated in detail by Master and Cloninger (1990) and Mezzich et al. (1990). (iii) In addition to the formulation of multiple diagnoses, multiple raters are also common in reliability tests. There are studies which happen to involve multiple raters formulating multiple diagnoses. A detailed description of such studies are presented in Mezzich et. al (1985). The procedures used here for assessing agreement among several raters formulating multiple diagnoses involve (i) the computation of a measure of agreement between two multiple diagnoses for the same subject according to a procedure operationally defined, (ii) the computation of overall average observed intrasubject agreement, (iii) the calculation of a measure of chance agreement, and (iv) the computation of an interrater agreement value and assessment of its statistical significance. Method We suggest two different definitions for "overlap proportion" and compute the interrater agreement index under each definition. Definition 1: Overlap proportion is the ratio of the number of agreements between specific categories over the number of specific categories mentioned in the longer of two diagnostic lists. 1109

2 Definition 2: The proportion of overlap between two diagnostic formulations is defined as the ratio of the number of agreements between specific categories over the number of different specific categories mentioned in the two diagnostic lists. To illustrate the proportion of agreement under each definition, let's consider the following two diagnostic formulations. Diagnostic formulation 1 cocaine abuse dysthymia exhibitionism Diagnostic formulation 2 cocaine abuse hypochondriasis The proportion of agreement under definition 1 is the ratio of 1 (one agreement, on "cocaine abuse") over 3 (the number of specific categories mentioned in the longer diagnostic lists is 3). The proportion of agreement between these two diagnostic formulations according to the "proportional overlap" criteria under definition 2, is the ratio of 1 (one agreement, on "cocaine abuse") over 4 (four possible agreements, that is, the number of different specific categories mentioned in the two diagnostic lists: "cocaine abuse" I "dysthymia", "hypochondriasis''', and "exhibitionism"). Let Pl' and pz be the proportion of observed agreement between two diagnostic formulations under def initions 1 and 2, respectively. It can be easily seen that O$pz$Pl$l. There are two strategies to correct chance agreement. One approach is to correct chance agreement under the assumption of rater-specific a pri.ori equal probabilities of diagnosis selection. The other approach is to correct chance agreement using raterspecific a posteriori probabilities of diagnosis selection based on the observed marginal frequencies of rater responses. (1) Strategy 1 (rater-specific a priori equal probabilities) Let Ai be the number of diagnoses selected by rater A, and Bi be the number of diagnoses selected by rater B, and Xi be the number of common diagnoses chosen by a pair of raters A and B for the i th patient. Under definition 1, observed agreement, po~, between rater A and rater B for the it subject is given by Kupper and Hafner (1989) showed that Xi follows the hypergeometric distribution, if both rater A and rater B choose their subsets of diagnoses for the ith subject completely at random. Then, under definition 1, chance agreement, Pei' between rater A and rater B for the i~ subject is given by Pci where k i diagnoses observed subjects given by E(Xd = min(a" Bd/k" is the number of available' for the i th subject. Then, the and chance agreement across between rater A and rater Bare The chance corrected agreement between rater A and rater B is given by When there are more than two raters, the overall interrater agreement, K, is computed as the average of the overall pairwise agreements. For example, the interrater agreement among raters A, B, and C is given by K = (KAB + K BC + KAC> 13. It is not possible to derive explicit expressions for the covariances among the pair-specific interrater agreement in K. The jackknife technique will be used to provide the variance estimate, which is needed to construct a confidence interval for K. Details of this procedure are described in Kupper and Hafner (1989). 1110

3 Similarly, chance corrected agreement can be calculated by following the above steps under definition 2. (2) Strategy 2 (rater-specific a posteriori probabilities) Observed agreement among the several raters for a subject is measured by averaging the proportions of agreement obtained for all combinations of pairs of raters judging that subject. The overall observed proportion of agreement (Po) for the sample of subjects under consideration is the average of the mean proportions of agreement obtained for each of the N subjects in the sample. Chance agreement is calculated using rater-specific a posteriori (i.e., estimated) probabilities of diagnosis selection based on the observed marginal frequencies of rater responses. That is, the proportion of chance agreement (Pc) is calculated by computing proportion of agreement between all diagnostic formulations made by all raters for all subjects under each definition, and then averaging across them. An interrater agreement index K under the definition 1 is computed as An interesting special case of the above formula results when there are two raters and each rater need select only one attribute to describe the ith subject. Then the above interrater agreement index is equal to the kappa coefficient which was originally proposed by Cohen (1960). Let S2(Po) be the variance of observed proportions of agreement across subjects and N be the number of subjects. The standard error of kappa (Kraemer, 1981) is given by SE(K) Then, K/SE(K) has approximately t distribution with N-l degrees of freedom. A 100(1-a)% confidence interval for the true agreement index can be calculated as K ± t 1-a/2 SE(K). Similarly, an inter rater agreement index is computed under definition 2. Examples The following two examples illustrate the computation of an interrater agreement index using the procedures stated in the previous section. 3.1 Example 1 (X-Ray Film Readings) Cantor, Reskin, and Luire (1985) examined two different types of X-ray films for detection of proximal surface caries: standard film "U", and faster speed film "E" which reduces patient x ray exposure by approximately 50%. If the X-ray quality of E film is at least as good as that provided by U film, the use of E film would be strongly encouraged by the dental community. Kupper and Hafner (1989) computed the inter rater agreement statistics using' definition 1 under -strategy 1 in order to assess the comparability of two films. Three rater~ independently evaluated 21 U films and 20 E films to determine which teeth had carious lesions (cavities) from the 6 premolars and molars shown on each X-ray. The interrater agreement values were computed from the study of X-ray film readings among raters using the data set given in Kupper and Hafner (1989). Table 1 shows the interrater agreement values, standard errors, and 95% confidence intervals. A Z-statistic is used to assess if the expected agreement for two X-ray film speed types are equal. Z = (Ku - KEl/{Var(K u ) + Var(KEl}O.5 <1 The Z-statistic shows that statistically significant between agreement measures film speed types. there is no difference for the two 3.2 Example 2 (Psychiatric Cases) Mezzich et al. (1985) conducted a reliability study, in which 30 child psychiatrists were asked over mail to make independent diagnoses on 27 child and adolescent psychiatric case summaries. Each psychiatrist rated 3 cases, and each case turned out to be rated by 3 or 4 psychiatrists upon completion of the study. Table 2 shows the resulting 90 multiple diagnostic formulations. Each diagnostic formulation was composed of up to three broad diagnostic categories taken from Axis I (clinical psychiatric syndromes) of the American Psychiatric 1111

4 Association's Diagnostic and statistical Manual of Mental Disorders (DSM-III, 1980). The interrater agreement values were computed from the study of diagnostic agreement among child psychiatrists. Table 3 shows the interrater agreement values, standard errors, and 95% confidence intervals. Discussion The original kappa coefficient, an index of interrater reliability adjusted for chance agreement, is generally very useful. However, it is not directly applicable to the currently prevailing situation in which multiple diagnoses (not necessarily fixed in number) are formulated by multiple raters (not necessarily fixed in number). The paper defines the concept of observed agreement, calculates the chance agreement using two strategies, (i) rater-specific a priori equal probabilities of diagnosis selection and (ii) rater-specific a posteriori probabilities of diagnosis selection based on the observed marginal frequencies of rater responses. These two approaches (i) and (ii) are expected to provide similar overall pairwise agreement values if the rater-specific marginal distributions are not highly skewed. Otherwise, the above two approaches will yield considerably different agreement values because the latter approach gives more weights on disagreements when the base rate (or prevalence rate) is very low or very high (Shrout et al., 1987). The following hypothetical data in Table 4 illustrates that two strategies yield considerably different agreement values when the base rate is very rare. The overall observed agreement between rater A and rater B is P o =0.98. The chance agreements are P c =O.5 and P c =O.96 under strategies 1 and 2, respectively. Therefore, the chance corrected agreement takes the values K=O.96, and K=O.49, using strategies 1 and 2, respectively. the knowledge of prevalence rate, the strategy 2 is more appropriate. Otherwise, the strategy 1 is suggested in practice. Regarding the choice of proportional overlap between definition 1 and def inition 2, both observed agreement and chance agreement are higher in definition 1 than those in definition 2; however, the interrater agreement is larger under definition 1 in some cases, and isn't larger in the other cases. In general, it is expected that overall agreement values will not differ much between definition 1 and definition 2. The interrater agreement estimates under strategy 1 are quite similar to those under strategy 2 in X-ray film readings and psychiatric cases examples. Under strategy 2, the obs'erved agreement among several raters for a single subject is computed by averaging the proportions of agreement obtained for. all combinations of pairs of raters evaluating that subject. The overall observed agreement value and its variance are ca.lculated using the observed agreement values obtained for each subject. Thus, the calculation of the variance of interrater agreement is straightforward. However, under strategy 1, the observed agreement value for each subject is obtained for a pair of raters, and then the overall observed agreement value between a pair of raters is computed by ave rag ing the observed agreement values obtained for each subject. Next, the interrater agreement between a pair of raters is computed. Overall interrater agreement values are calculated by averaging the interrater agreement values obtained for all combinations of pairs of raters. Since it is not possible to derive explicit expressions for the covariances among the pair-specific inter rater agreement, the use of jackknife is suggested to compute the variance estimate under strategy 1. The computation of variance estimate is much easier under strategy 2 than under strategy 1. The choices of the strategies should depend on the rater's prior knowledge on the prevalence rate of diagnoses or attributes. If the prevalence rate of diagnoses is very low or very high, and the rater selects the diagnosis bas~d on 1112

5 REFERENCES Ahn C, and Mezzich J. PROPOV-K: A Fortran program for computing a kappa coefficient using a proportional overlap procedure. Comput. Biomed. Res. 22, 415 (1989). American Psychiatric Association. "Diagnostic and Statistical Manual of Mental Disorders." 3rd ed. DSM-lll, American Psychiatric Association, Washington, DC, Table 1. Interrater agreement estimates and associated inferential statistics for each film speed U Film Oef. 2 Strategy 1 Estimates std. Dev strategy 2 95% C.l (0.359, 0.668) (0.343, 0.656) American Psychiatric Association. "Diagnostic and Statistical Manual of Mental Disorders." 1st ed. DSM-lll-R, American Psychiatric Association, Washington, DC, Oef. 2 E Film Estimates ~S~t~d~.~D7e~v~. 95% C.l (0.376, 0.703) (0.362, 0.678) Cohen J. A coefficient of agreement for nominal scales. Educ. Psych. Measure (1960). Kraemer H. Extension of the kappa coefficient. Biometrics 36, 207 (1980). Kupper L, and Hafner K. On Assessing interrater agreement for multiple attribute responses. Biometrics 45, 957 (1989). Strategy 1 Estimates ~S~t~d~.~o~e~v~. 95% C.l (0.314, 0.626) Oef (0.324, 0.642) Def. 2 Strategy 2 Est ima te s "s",t",d",.,---,d",e",v:w,' 9 5 % C. I (0.258, 0.640) (0.275, 0.647) Master J, and Cloninger C. Comorbidity in anxiety and mood disorders. American Psychiatric Press, Mezzich A, Mezzich J, and Coffman G. Reliability of DSM-IIl VB. DSM-ll in child psychopathology. J. of Amer. Academy of Child Psychiatry. 24, 273 (1985). Mezzich J, Ahn C, Fabrega H, and Pilkonis P. Psychiatric co-morbidity patterns in a large population presenting for care. in Comorbidity in Anxiety and Mood Disorders edited by Masters J and Cloninger C, American Psychiatric Press, washington, DC, Shrout P, Spitzer R, and Fleiss J. Quantification of agreement in psychiatric diagnoses revisited. Archives of General Psychiatry. 44, 172 (1987). 1113

6 Table 2. formulations cases using categories Cases S Multiple diagnostic for 27 child psychiatric DSM-III Axis I broad Rat." n ,7,7 14,16 14, II, II , II ,9 14, Note. 1. Organic mental disorders; 2. substance usc disorders: 3. scbizophrenic and paranoid disorders: 4. scbizoaffeetive disorden; S. affective disorders; 6, psychoses nol elsewhere classi 6cd; 7. anxiety factitious. somatofonn. and dissociative disorders; 8, psychosexual disorder; 9, mental retardation: 10. pervasive developmental disorder; 11, anention deficit disorders; 12. conduct wsorders; 13. anxiety disorders of childhood or adolescence; 14, other disorders of childhood or adolescence, speech and stereotyped movement disorders. disorders characteristic of late adolescence; 15. eating disorders; 16. ru.ctive disorders not elsewhere ejassified~ 17, disorders of impulse con trol not elsewhere classified; 18, sleep and other disorders; 19. conditions not anributable to a mental disorder: 20. no diagnosis on Axis 1. Table 3. Interrater agreement estimates and associated inferential statistics strategy 1 Def. 2 Def. 2 Estimates Strategy 2 Estimates Table 4. Hypothetical data Rater B Std. Dev. 95% C.r (0.13, 0.53) (0.14, 0.45) Std. Dev. 95% C.I (0.18, 0.38) (0.16, 0.38) Dx Rater A No Ox DX No Dx