NICHOLAS BELLAMY, ALEXANDER KLESTOV, KENNETH MUIRDEN, PETRA KUHNERT, KIM-ANH DO, LOUISE O'GORMAN, and NICHOLAS MARTIN

Transcription

1 Perceptual Variation in Categorizing Individuals According to American College of Rheumatology Classification Criteria for Hand, Knee, and Hip Osteoarthritis (OA): Observations Based on an Australian Twin Registry Study of OA NICHOLAS BELLAMY, ALEXANDER KLESTOV, KENNETH MUIRDEN, PETRA KUHNERT, KIM-ANH DO, LOUISE O'GORMAN, and NICHOLAS MARTIN ABSTRACT. Objective. The American College of Rheumatology (ACR) classification criteria for osteoarthritis (OA) permit the categorization of individuals for hand, knee, and hip OA and are of defined sensitivity and specificity. They depend on both clinical and radiographic aspects of ~A. The clinical diagnosis of OA in the peripheral skeleton is dependent on the skilled examination of several clinical features characteristic of the condition, while the interpretation of radiographs is a perceptual skill based on appreciating specific structural features on plain radiographs. We investigated the interrater reliability of the ACR classification criteria for OA when applied in a community based sample. Methods. The study was part of a multifaceted diagnostics protocol, evaluating methodologic issues, in the conduct of genetic research in OA. From a cohort of 8 pairs of twins registered with the Australian Twins Registry (ATR), standard clinical examinations of hands, knees, and hips were performed on 74 complete and incomplete pairs of twins over age 50 years. The pairs were selected to represent both twin pairs who had previously self-reported a diagnosis of OA, as well as those who had not. Rheumatologists who performed the assessments were blind to the original self-report. All subjects were examined independently by one of 2 pairs (NB/AK or NBIKM) of consultant rheumatologists, blind to one another's assessments. Each rheumatologist separately assessed the hands, knees, and hips, rating them clinically by ACR criteria for ~A. The observations were made without reference to any radiographic or serologic information. Results. Interrater agreement was different for the 3 different anatomic areas and was different for the 2 pairs of rheumatologists. The actual (observed) interrater agreements based on ACR clinical criteria were as' follows: hand OA NB/AK = 0.92, NBIKM = 1.00; knee OA NB/AK = 0.94, NBIKM = 0.92; hip OA NBI AK = 0.98, NBIKM = lnterrater agreement based on ACR clinical criteria, as assessed by the adjusted kappa statistic, was as follows: hand OA NB/AK = 0.84, NBIKM = 1.00; knee OA NB/AK = 0.87, NBIKM = 0.84; hip OA NB/AK = 0.95, NB/KM = Conclusion. Since clinical agreement was extremely high in all 3 anatomic sites, and for both pairs of assessors, we conclude that for genetic epidemiology purposes, subjects can be examined by a single experienced rheumatologist using the ACR classification criteria. (J Rheumatol 1999;26:2654-8) Key Indexing Tenns: OSTEOARTHRITIS ACR CRITERIA TWIN OBSERVER AGREEMENT From the Department of Medicine, The University of Western Ontario, London, Canada; Queensland Institute of Medical Research and the University of Queensland, Brisbane; and Melbourne University, Melbourne, Australia. N. Bellamy, MD, MSc, FRCP (Glas, Edin, C), FAC?, Professor, Department of Medicine, The University of Western Ontario and Queensland Institute of Medical Research; A. Klestov. MD, FRACP, University of Queensland; K. Muirden, MD, Melbourne University; P. Kuhnert, BSc; K-A. Do, PhD; L. O'Gonnan, PhD; N. Martin, PhD, Queensland Institute of Medical Research. Address reprint requests to Prof N. Bellamy, Centre of National Research on Disability and Rehabilitation Medicine, University of Queensland Department of Medicine, Clinical Sciences Bldg., Royal Brisbane Hospital, Herston Rd., Queensland 4029, Australia. . nbellamy@medicine.uq.edu.au Submitted December 18,1998 revision accepted April 1, Musculoskeletal (MSK) clinical examination is a perceptual skill originally acquired over a period of supervised training and followed by many years of application in clinical practice and for some rheumatologists also in clinical research. Even those rheumatologists who have engaged in clinical research, and who may have given added consideration to the importance, not only of intrarater reliability but also of interrater reliability, do not completely agree with one another even after a period of training in the use of standardized examination techniques 1 - JO However, it is possible to train assessors to acceptable levels of reliability. Consistency in performing the MSK examination is critical II, since misclassification when 2654 The Journal of Rheumatology 1999; 26:12

2 severe can invalidate the results of an epidemiologic investigation. It is therefore essential to quantitate interrater agreement in categorizing individuals as to the presence of osteoarthritis (OA) prior to performing analysis on epidemiologic data sets. We are investigating the genetic determination of ~A. This study requires a radiographic as well as a clinical assessment of each participating twin pair. Although we have performed a separate evaluation of categorization by clinical criteria, for joints not captured by American College of Rheumatology (ACR) criteria, the ACR criteria are also based in part (knee I2, hip13) or entirely (hand I4 ), on aspects of the MSK examination. Furthermore, most observer agreement studies are based only on patients with the disease of interest. In such studies there is a high expectation of observing commonly occurring features. The purpose of this study was to assess observer agreement for the ACR classification criteria for OAI5-17, when applied by experienced rheumatologists, in a community based sample derived from the Australian Twin Registry (ATR). MATERIALS AND METHODS The study used the Australian National Health and Medical Research Council Twin Registry. The ATR is a volunteer registry started in Twins were recruited through schools, community groups, and by media advertising throughout Australia. About 25,000 pairs of twins of all types and ages are registered. This represents about 10% of the expected number of twin pairs in Australia. From November 1993 to July 1995, a questionnaire was sent to a cohort of 78 pairs over age 50 years. Of this group, 1533 individuals returned completed questionnaires, with complete data being obtained on 602 pairs. The questionnaire was extensive and included items on age, sex, zygosity, birth order, general health, attitudes to health, life events, coping, smoking, alcohol, exercise, emotions, personality, bones and joints, vitamins, and sun exposure. In the bones and joints section, registrants were questioned separately about the following: pain, swelling, and stiffness in joints; prior diagnosis of OA (degenerative arthritis), rheumatoid arthritis, and other forms of arthritis or rheumatism; prior bone fracture or joint injury; and radiographs of hands, knees, or hips taken in the last 5 years. In addition, registrants indicated on a homunculus any joints affected by pain or swelling. Registrants were asked to respond to these questions first considering themselves, and then to provide information regarding their co-twin. From a combination of self-reported OA and involvement of target joints for OA without prior history of joint trauma, twins potentially affected by OA were identified. In contrast, those twins not identifying joint problems were categorized as being non-oa. Because it was necessary to examine as well as take radiographs of and blood samples from study subjects, it was reasoned given age and condition that participants would be unlikely to travel more than 50 km from home. Since the examinations were to be performed in Brisbane and Melbourne, we invited 63 OA pairs (41 discordant and 22 concordant pairs) and an additional 55 non-oa pairs, who met the aforementioned selection criteria, to participate. On the day of study, subjects were examined independently by 2 consultant rheumatologists, blood was taken by venipuncture, a skin mold was made, and radiographs taken of hands, knees, and hips. Not all twins attended in pairs, although many did. They were not examined in any set order and clinical examinations were carried out in separate rooms. No discussion was allowed regarding individual examinations. The clinical portion of the ACR criteria was completed for each subject. Pain, morning stiffness, crepitus, and bony enlargement were elicited using standard clinical techniques. Range of movement was measured using a Baseline long-arm goniometer, applied in a standard fashion. No radiographic or erythrocyte sedimentation rate (ESR) data were available at the time of the clinical assessments. They were obtained later the same day, but were not reviewed until after completion of the clinical portion of the study. Data analysis was conducted using S_PIUS 18 In addition to estimating the actual level of agreement for each pair of assessors, Cohen's kappa19 (unweighted), a statistic of agreement beyond chance, also was calculated separately for each anatomic area. Since Cohen's kappa can be affected adversely by both bias and prevalence, we have also calculated the bias index (BI), the prevalence index (PI), and the adjusted kappa (K adj ), which takes into account both the BI and pro. The adjusted kappa is the preferred estimate of agreement beyond chance and closely parallels the variations in the actual observed levels of agreement. RESULTS A total of 159 subjects (74 complete pairs, incomplete pairs) agreed to participate and were examined. The demographic characteristics of the subjects, including age, sex, height, and weight, are illustra~ed in Tables 1 and 2. There were 28 MZF, 14 MZM, 20 DZF, 3 DZM, 4 DZFM, and 5 DZMF complete pairs (Table 1). The mean age of the 74 complete pairs was years (SD 6.99) and of the incomplete pairs (Table 2) years (SD 7.76). The mean ESR was (range 1-61, SD 10.78, median 10). One examiner (NB) was present for all examinations. However, the second examiner was different in Brisbane (AK) and Melbourne (KM). In reporting interrater agreement, we have provided agreement coefficients for actual (i.e., observed) agreementobs' Cohen's kappa COHEN ' and adjusted kappaadjusted and the BI and PI for each anatomic area and pair of assessors (Table 3). Bias index and prevalence index. BI and PI values were different for different criteria and different pairs of assessors as follows (Table 3). Table 1. Demographics for 74 complete twin pairs who participated in the Australian 1\vins Registry OA study. Zygosity OverAll, MZF MZM DZF DZM DZFM DZMF 105F,43M Variable N Mean (SD) N Mean (SD) N Mean (SD) N Mean (SD) N Mean (SD) N Mean (SP) N Mean (SD) Age,yrs (6.99) (7.36) (6.23) (6.81) (5.72) (6.32) (2.2) Height, em (8.56) (6.43) (8.07) (7.57) (4.38) (4.88) (8.39) Weighl,kg (13.56) (13.24) (10.92) (13.96) (9.67) (15.33) (l0.82) City, Brisbane City, Melbourne Bellamy, et al: ACR criteria in OA 2655

3 BI values PI values NB vsak to to 0.98 NBvsKM to to 0.95 Hands. The following agreement coefficients were noted for the ACR hand criteria (Table 3). NB vsak NB vskm OBS 0.78 to to 1.00 COHEN to to 1.00 ADJUSTED 0.55 to to 1.00 OBS COHEN ADJUSTED Hips. The following agreement coefficients were noted for the ACR bip criteria (Table 3). NB vsak NBvsKM OBS COHEN ADJUSTED 0.69 to to to to to to 0.93 OBS COHEN ADJUSTED Knees. The following agreement coefficients were noted for the ACR knee criteria (Table 3). NBvsAK NBvsKM OBS 0.66 to to 0.98 COHEN to to 0.93 ADJUSTED 0.31 to to 0.97 OBS COHEN ADJUSTED Table 2. Demograpbics for incomplete twin pairs who participated in the Australian Twins Registry OA study. Variable Age, yrs Height, cm Weight, kg City, Brisbane City, Melbourne N 8 3 Overall 8 Women, 3 Men Mean (SD) (7.76) (.69) (23.93) DISCUSSION. The correct categorization of individuals is an essential component of genetic epidemiology research. Such categorizations may be based on clinical, radiographic, or serologic information, or on a combination21 We have assessed the extent to which 2 pairs of experienced rheumatologists agree on the presence or absence of OA in 68 peripheral joints. It is important to recognize that while assessors may agree, they may nevertheless both be incorrect. Therefore, the clinical skill and experience of the participating rheumatologists and radiologists is paramount. For this study we chose 3 senior academic consultant rheumatologists, each with at least 15 years' experience in performing and teaching the MSK examination. In addition, one participant (NB) had previously conducted a number of observer variability studies l-4,9, in which he had been standardized against 5 other Canadian rheumatologists, and whose performance in the assessment of patients with OA, rheumatoid arthritis, ankylosing spondylitis, fibromyalgia, and scleroderma had been quantitated. The 2 academic radiologists were also highly experienced in reporting MSK films. We believe that statistics of agreement are appropriate to addressing the issue of the necessity for replicate assessments when performing genetic epidemiology studies. The issue of which statistic to use is contentious, since there are several reliability statistics, each of which examines a different aspect of reliability. For discrete variables, Cohen's kappa statistic19 is appropriate and is commonly used. However, this statistic can be capricious, and some knowledge of factors that might affect kappa is important. In this study, the BI was very small, suggesting that the effect of bias on kappa values was small, was different for different joints, and was similar for the different pairs of assessors. Bias, therefore, is not a significant problem. In contrast, the PI was large for most joint assessments. It was large because this was a community based sample in which only some participants had OA, and was even greater for non-target joints because of the especially low prevalence in those joints even in affected individuals. The PI, therefore, has a profound influence on unadjusted kappa values. As a consequence, we do not believe the Cohen's kappa is a good estimate of agreement in this study. In contrast, the adjusted kappa 20 provides an estimate of agreement beyond chance that closely parallels the actual observed agreement. Based on the observed agreement and adjusted kappa values, we conclude that for non-target joints agreement is almost perfect. The ACR classification criteria were simple to apply in this community based sample. Extremely high levels of agreement were observed for both pairs of clinical assessors for all 3 anatomic areas, even though agreement coefficients for component criteria were sometimes modest. Nevertheless, when incorporated in the algorithm they collectively resulted in high levels of actual observed agreement and adjusted kappa values., we conclude that the agreement between experienced rheumatologists in applying ACR classification criteria for hip, knee, and hand OA is excellent in all 3 anatomic areas. These observations suggest that for genetic epidemiology purposes such examinations could be performed by a single similarly skilled individual. The requirement for only a single assessor has important time and cost implications for such studies The Jouma! of Rheumatology 1999; 26:12

4 Table 3. Agreement between 2 pairs of rheurnatologists in rating the presence of OA based on the ACR criteria Feature NB vs AK Clinical Assessment NB vs KM Clinical Assessment Po (Ppos- Pneg) K BI PI Kadj Po (Ppos- Pneg) K BI PI Kadj HAl'IDS Pain, aching, or stiffness 0.89 (-0.24) '1.00 (0.00) Hard tissue enlargement of 2 or more of 10 selected hand joints 0.78 (-0.09) (-0.24) MCP swelling in < 2 joints 0.81 (0.90) (0.50) Hard tissue enlargement involving 2 or more DIP 0.81 (-0.10) (-0.25) Defonnity of 1 or more of 10 selected hand joints 0.93 (-0.34) (-0.42) D (-0.32) (0.00) HIPS Pain 0.94 (-0.70) (-0.18) Internal rotation 0.69 (-0.01) (0.00) Flexion 0.77 (-0.09) (-0.16) Morning stiffness 0.98 (0.99) (0.97) Pain on internal rotation 0.94 (-0.97) (-0.40) ESR N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A Radiographic osteophytes N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A Radiographic JSN N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A (clinical) 0.98 (-0.99) (-0.32) (clinicalllab/rad) 0.95 (-0.64) (-0.13) KNEES Pain 0.88 (-0.37) (-0.05) Crepitus 0.66 (-0.23) (-0.) Morning stiffness 0.99 (0.00) (0.00) 0.91 om Bony enlargement 0.96 (-0.98) (-0.99) Radiographic osteophytes N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A Synovial fluid NID NID NID NID NID NID NID NID NID NID (clinical) 0.94 (-0.32) (-0.34) Overall (clinicalllab/rad) 0.91 (-0.38) (-0.25) Po: observed agreement, K: Cohen's kappa, BI: bias index, PI: prevalence index, Kadj: adjusted kappa, MCP: metacarpophalangeal, DIP: distal interphalangeal, ESR: erythrocyte sedimentation rate, JSN: joint space narrowing. REFERENCES 1. Bellamy N, Anastassiades TP, Buchanan WW, et al. Rheumatoid arthritis antirheumatic drug trials. 1. Effects of standardization procedures on observer dependent outcome measures. J Rheumatol 1991;18: Bellamy N, Carette S, Ford PM, et ai. Osteoarthritis antirheumatic drug trials. 1. Effects of standardization procedures on observer dependent outcome measures. J RheumatoI1992;19: Bellamy N, Buchanan WW, Esdaile 1M, et al. Ankylosing spondylitis antirheumatic drug trials. 1. Effects of standardization procedures on observer dependent outcome measures. J Rheumatol 1991;18: Bellamy N, Bell MJ, Carette S, et al. Estimation of observer reliability and sample size calculation parameters for outcome measures in fibromyalgia clinical trials. Inflammopharmacology 1994;2: Bellamy N, Muirden KD, Bendrups A, et al. Rheumatoid arthritis measures. InfIammopharmacology 1997;5: Bellamy N, Baclmieier C, Brooks PM, et al. Osteoarthritis measures. Inflammopharmacology 1997;5: Bellamy N, Muirden KD, Boyden K, et al. Ankylosing spondylitis measures. Inflammopharmacology 1997;5: Bellamy N, Buchbinder R, Hall S, et al. Fibromyalgia measures. Inflammopharmacology 1997;5: Pope JE, Baron M, Bellamy N, et al. Variability of skin scores and clinical measurements in scleroderma. J Rheumatol 1995;22: Green S, Buchbinder R, Forbes A, Bellamy N. A standardized protocol for measurement of range of movement of the shoulder using the plurimeter-v inclinometer and assessment of its intrarater and interrater reliability. Arthritis Care Res 1998; : Altman RD, Meenan RF, Hochberg MC, et al. An approach to Bellamy, et al: ACR criteria in OA 2657

5 developing criteria for the clinical diagnosis and classification of osteoarthritis. J RheumatoI1983;1O: Altman R, Asch E, Bloch D, et al. Development of criteria for the classification and reporting of osteoarthritis. Classification of osteoarthritis of the knee. Arthritis Rheum 1986;29: Altman R, Alarcon G, Appelrouth D"et al. The American College of Rheumatology criteria for the classification and reporting of osteoarthritis of the hip. Arthritis Rheum 1991;34: Altman R, Alarcon G, Appelrouth D, et al. The American College of Rheumatology criteria for the classification and reporting of osteoarthritis of the hand. Arthritis Rheum 1990;33: Altman RD. Criteria for classification of clinical osteoarthritis. J Rheumatol 1991;18 SuppI27: Altman RD. The classification of osteoarthritis. J Rheumatol 1995;22 Suppl 43: Schouten JSAG, Valkenburg HA. Classification criteria: methodological considerations and results from a 12 year following study in the general population. J Rheumato995;22 Suppl 43: S-Plus Guide to Statistical and Mathematical Analysis, Version 3.3. Seattle: StatSci, A division of Mathsoft Inc., Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas 1960;20: Byrt T, Bishop J, Carlin m. Bias, prevalence and kappa. J Clin Epidemiol 1993;46: Dougados M, Nakache J-P, Gueguen A. Criteria for generalized and focal osteoarthritis. Rev Rhum (Engl Ed) 1996;63: The Journal of Rheumatology 1999; 26:12