Reliability of the Spinal Instability Neoplastic Scale

Transcription

1 Musculoskeletal Imaging Original Research Fisher et al. Reliability of the Spinal Instability Neoplastic Scale Musculoskeletal Imaging Original Research Charles G. Fisher 1,2 Anne L. Versteeg 3 Rowan Schouten 4 Stefano Boriani 5 Peter P. Varga 6 Laurence D. Rhines 7 Manraj K. S. Heran 8,9 Norio Kawahara 10 Daryl Fourney 11 Jeremy J. Reynolds 12 Michael G. Fehlings 13,14 Ziya L. Gokaslan 15 Fisher CG, Versteeg AL, Schouten R, et al. Keywords: metastasis, neoplasm, radiologist reliability, spinal instability, spinal instability neoplastic scale (SINS), spine, validity DOI: /AJR Received November 13, 2013; accepted after revision February 19, L. D. Rhines received grant support for another work from Stryker. J. J. Reynolds received grant support for other works from Globus, DePuy Synthes, and Medtronic. Z. L. Gokaslan received grant support for other works from AOSpine, NREF, DePuy, US Spine and Spinal Kinetics, and the AO Foundation. This study received financial support from AOSpine International through the AOSpine Knowledge Forum Tumor. AOSpine is a clinical division of the AO Foundation, which is an independent medically guided not-for-profit organization. AJR 2014; 203: X/14/ American Roentgen Ray Society Reliability of the Spinal Instability Neoplastic Scale Among Radiologists: An Assessment of Instability Secondary to Spinal Metastases OBJECTIVE. The spinal instability neoplastic scale (SINS) is a new classification system for tumor-related spinal instability. The SINS may prove to be a valuable tool for radiologists to communicate with oncologists and surgeons in a standardized evidence-based manner. The objective of this study was to determine the inter- and intraobserver reliability and validity of the SINS among radiologists. MATERIALS AND METHODS. Thirty-seven radiologists from 10 international sites used the SINS to categorize the degree of spinal instability in 30 patients with spinal tumors. To assess validity, we compared the SINS scores assigned by the radiologists with the SINS scores of 11 spine oncology surgeons (reference standard). Each total SINS score (range, 0 18 points) was converted into one of the following three clinical categories: 0 6 points, stable; 7 12 points, potentially unstable; and points, unstable. In addition, each total SINS score was converted into a binary scale: 0 6 points was defined as stable, and 7 18 points was considered a current or possible instability for which surgical consultation is recommended. RESULTS. Radiologists using the SINS binary scale showed excellent (κ = 0.88) validity, substantial (κ = 0.76) interobserver agreement, and excellent (κ = 0.82) intraobserver reproducibility. Radiologists rated all unstable cases and 621 of 629 (98.7%) potentially unstable cases with a SINS score of 7 or more points, thus appropriately initiating a referral for surgical assessment. CONCLUSION. SINS is a reliable tool for radiologists rating tumor-related spinal instability. It accurately discriminates between stable and potentially unstable or unstable lesions and, therefore, can guide the need for surgical consultation. R ecent advances in systemic treatments have improved the survival rates of cancer patients, which has resulted in an increased incidence of spine metastatic disease [1 4]. Spinal metastases can create spinal column insta- bility and neural compression that can potentially result in significant pain, devastating neurologic consequences, or both. Early diagnosis and appropriate management are therefore crucial. The typical goals of spinal metastasis treatments are pain relief, the res- 1 Department of Orthopaedics, Division of Spine, University of British Columbia and Vancouver General Hospital, Vancouver, BC, Canada. 2 Blusson Spinal Cord Centre, 818 W 10th Ave, 6th Fl, Vancouver, BC V5Z 1M9, Canada. Address correspondence to C. G. Fisher (charles.fisher@vch.ca). 3 Department of Orthopaedics, University Medical Center Utrecht, Utrecht, The Netherlands. 4 Department of Orthopaedic Surgery and Musculoskeletal Medicine, Christchurch Hospital, Christchurch, New Zealand. 5 Department of Degenerative and Oncological Spine Surgery, Rizzoli Institute, Bologna, Italy. 6 National Center for Spinal Disorders and Buda Health Center, Budapest, Hungary. 7 Department of Neurosurgery, The University of Texas M. D. Anderson Cancer Center, Houston, TX. 8 Department of Radiology, University of British Columbia, Vancouver, BC, Canada. 9 Department of Diagnostic & Therapeutic Neuroradiology, Vancouver General Hospital, Vancouver, BC, Canada. 10 Department of Orthopedic Surgery, Kanazawa Medical University, Kahoku-gun, Japan. 11 Department of Surgery, Division of Neurosurgery, University of Saskatchewan, Royal University Hospital, Saskatoon, SK, Canada. 12 Spinal Division, Oxford University Hospital, NHS Trust, Oxford, UK. 13 Department of Neurosurgery, University of Toronto, Toronto, ON, Canada. 14 Krembil Neuroscience Centre, Toronto Western Hospital, University Health Network, Toronto, ON, Canada. 15 Department of Neurosurgery, The Johns Hopkins University School of Medicine, Baltimore, MD. AJR:203, October

2 Fisher et al. toration or preservation of neurologic function, and establishment of spinal stability. The latter is often the primary intention of the treatment. A range of therapeutic options provided by various subspecialists including medical and radiation oncologists, radiologists, and surgeons exists. Therefore, it is critical that the terminology used for the diagnosis and treatment of spinal instability due to metastases is standardized and evidence based so that the subspecialists can work together to facilitate optimal patient care. High-grade neurologic compression and mechanical instability are considered absolute indications for surgery [5, 6]. Although neurologic issues are relatively straightforward from a diagnostic perspective, spinal instability related to metastases is not, as evidenced by the facts that there is no established reference standard and that there are inconsistencies among spine oncology surgeons, the specialty most familiar with the anatomy and treatment of spine instability [7]. The diagnosis of spinal oncologic instability is even more difficult for the nonsurgical members of the multidisciplinary care team, possibly resulting in the underrecognition and underreferral of patients who may benefit from surgical stabilization [8, 9]. Diagnosing spinal stability is made difficult by the lack of consensus guidelines rating the relevance and validity of specific clinical and imaging findings. In response, the Spine Oncology Study Group (SOSG), an international cohort of spine oncology experts including oncology spine surgeons and oncologists, developed a tool to assess and grade spinal stability to assist in making treatment decisions and identifying patients requiring surgical assessment. SOSG defined spinal instability caused by metastatic disease as follows [7]: [the] loss of spinal integrity as a result of a neoplastic process that is associated with movement-related pain, symptomatic or progressive deformity, and/or neural compromise under physiologic loads. SOSG then integrated the best available literature and expert opinions to develop the spinal instability neoplastic scale (SINS) [7]. The SINS classifies spinal stability by adding together six radiographic and clinical components to create a total SINS score ranging from 0 to 18 (Table 1). The total SINS score is further divided into the following three clinical categories of stability: 0 6 points denotes the spine as stable; 7 12 points, as potentially unstable (Fig. 1); and points, as unstable TABLE 1: The Spinal Instability Neoplastic Scale (SINS) Classification According to Fisher et al. [7] Parameters Score (Points) Location Junctional (occiput C2, C7 T2, T11 L1, L5 S1) 3 Mobile spine (C3 C6, L2 L4) 2 Semirigid (T3 T10) 1 Rigid (S2 S5) 0 Pain a Yes 3 Occasional pain but not mechanical 1 Pain-free lesion 0 Bone lesion Lytic 2 Mixed (lytic/blastic) 1 Blastic 0 Radiographic spinal alignment Subluxation/translation present 4 De novo deformity (kyphosis/scoliosis) 2 Normal alignment 0 Vertebral body collapse > 50% collapse 3 < 50% collapse 2 No collapse with > 50% body involved 1 None of the above 0 Posterolateral involvement of spinal elements b Bilateral 3 Unilateral 1 None of the above 0 Note The scores for the six radiographic and clinical components were added together to yield a total SINS score ranging from 0 to 18. Reprinted with permission from [7]: Fisher CG, DiPaola CP, Ryken TC, et al. A novel classification system for spinal instability in neoplastic disease: an evidence-based approach and expert consensus from the Spine Oncology Study Group. Spine (Phila Pa 1976) 2010; 35:E1221 E1229. a Pain improvement with recumbency, pain with movement or loading of spine, or both. b Facet, pedicle, or costovertebral joint fracture or replacement with tumor. (Fig. 2). Based on a recommended threshold for surgical consultation, a binary scale was also created: stable (0 6 points) or a current or possible instability for which assessment by a spine surgeon is recommended (7 18 points). SOSG members using SINS showed 95.7% and 79.5% sensitivity and specificity, respectively, for the detection of potentially unstable or unstable spinal metastases [10]. Near-perfect interobserver reliability and intraobserver reliability were noted for distinctions among the three clinical categories of stability [10]. Although the SINS has not yet undergone prospective clinical validation, it represents the only classification system to date for diagnosing neoplastic spinal instability derived from evidence-based medicine. It is therefore critical that key physicians who will use it and advise on it specifically, radiologists assess the tool from a fundamental diagnostic test perspective before prospective clinical evaluation. The aim of this study was to assess the interand intraobserver reliability of the SINS score among radiologists who play a pivotal role in the evaluation of patients with spine metastases. Materials and Methods Patients This study received institutional review board approval before its commencement. The records for 30 cases of metastatic spine disease with patient-identifying information removed were reviewed at 10 international sites. The patients in the study group were 13 women (mean age, 60 years; 870 AJR:203, October 2014

3 Reliability of the Spinal Instability Neoplastic Scale age range, years) and 17 men (mean age, 64 years; age range, years). Lung (n = 6, 20%), prostate (n = 6, 20%), breast (n = 5, 17%), and myeloma (n = 4, 13%) were the most common primary malignancies, with 12 different tumor types represented overall. Lesions were spread across the cervical (C0 C7) spine (n = 6, 20%), thoracic (T1 T10) spine (n = 11, 37%), thoracolumbar (T11 L2) junction (n = 8, 27%), and low lumbar (L3 L5) regions (n = 5, 17%). For each case, demographic data, histopathologic diagnosis, and a pain description with A D B E Fig year-old man with multiple myeloma who presented with mild low back pain that is not aggravated by activities or movement. A E, Selected CT images of L5 lesion are provided: left parasagittal (A), midline sagittal (B), right parasagittal (C), coronal (D), and axial cut (E). Total spinal instability neoplastic scale (SINS) score was calculated as follows: junctional location, 3 points; pain but not mechanical, 1 point; lytic, 2 points; spinal alignment preserved, 0 points; less than 50% vertebral body collapse, 2 points; unilateral replacement of right pedicle with tumor, 1 point. Thus, total SINS score is 9 points ( = 9), which indicates potential instability for which surgical referral is recommended. an emphasis on movement-related pain were provided to simulate information typically provided to the radiologist. Representative preoperative radiographs, CT scans, and MR images followed. CT scans were provided for each assessment. After reviewing an instructional booklet and case examples on the application of the SINS, 37 radiologists from North America, Europe, and Asia independently assigned a SINS score to each case. From 6 to 8 weeks later, after the order of the cases was changed to minimize recall bias, the same radiologists scored the same cases a second time. All radiologists were working at academic centers and had been in practice for an average of 15 years (SD = 12 years) at the time of the study, and 36% had completed subspecialty training in neuroradiology. Although a true reference standard has not yet been determined, the best possible validity reference or reference standard was established by 11 experienced spine oncology surgeons, all of whom are familiar with the SINS classification and are members of the SOSG. They reviewed and rated each case C AJR:203, October

4 Fisher et al. once. For each case, the most frequent SINS score (i.e., the score assigned by most surgeons) became the reference standard score. In cases in which ties occurred, the two most frequent scores were averaged. For each case, the total SINS score, the three clinical categories (stability, potential instability, and instability), and the binary scale were analyzed. The mean number of years in practice for the spine surgeons was 13 years (SD, 12 years). A D Statistical Analysis Interobserver agreement (Fleiss kappa) for the radiologists and surgeons and intraobserver reproducibility (Cohen s kappa) among the radiologists were calculated for the total SINS score, the three clinical categories, and the binary scale [11, 12]. Validity was defined as the average kappa (Cohen s kappa) for agreement between the radiologists and the reference standard. The kappa coefficients were interpreted and translated into levels of agreement using the Landis and Koch [13] grading system (Table 2), where a kappa value of 1.0 implies perfect agreement. To produce CIs for the kappa estimates, we used a sampling distribution for the kappa statistic by means of bootstrapping, simulating 1000 B E Fig year-old woman with known metastatic sarcoma who presented with neck pain that was exacerbated by any movement but that improved with application of cervical collar. A E, Selected CT images of C4 lesion are provided: left parasagittal (A), midline sagittal (B), right parasagittal (C), coronal (D), and axial cut (E). Total spinal instability neoplastic scale (SINS) score was calculated as follows: mobile spine location, 2 points; mechanical pain, 3 points; lytic lesion, 2 points; kyphotic deformity, 2 points; more than 50% collapse, 3 points; bilateral posterolateral involvement, 3 points. Total SINS score is 15 points ( ), which indicates potential instability for which surgical referral is recommended. kappa estimates, and using the 2.5 and 97.5 percentiles for the CIs. All statistical analyses were performed using Stata software (version 12.0, StataCorp) for Microsoft Windows. Results Interobserver Agreement Interobserver agreement was substantial (κ = 0.76) and moderate (κ = 0.53) among radiologists for the binary scale and three clinical categories, respectively (Table 3). An analysis of the separate SINS components showed excellent agreement for location (κ = 0.94); sub- C 872 AJR:203, October 2014

5 Reliability of the Spinal Instability Neoplastic Scale stantial agreement for pain (κ = 0.73), bone lesion quality (κ = 0.65), and vertebral body collapse (κ = 0.61); and moderate agreement for spinal alignment (κ = 0.49) and posterolateral involvement of the spinal elements (κ = 0.55). The interobserver agreement among surgeons was excellent (κ = 0.83) and substantial (κ = 0.65) for the binary scale and three clinical categories, respectively. Intraobserver Reproducibility For the radiologists, intraobserver reproducibility was excellent (κ = 0.82) and substantial (κ = 0.69) for the binary scale and three clinical categories, respectively (Table 3). An analysis of the separate SINS components showed that intraobserver reproducibility was excellent for location (κ = 0.96) and pain (κ = 0.85) and was substantial for bone lesion quality (κ = 0.75), spinal alignment (κ = 0.69), vertebral body collapse (κ = 0.71), and posterolateral involvement of the spinal elements (κ = 0.68). Validity The level of agreement between radiologists and reference standard was excellent (κ = 0.88) for the binary scale and substantial (κ = 0.71) for the three clinical categories. Table 4 outlines the three clinical categories (0 6 points, stable; 7 12 points, potentially unstable; points, unstable) scored by the spine surgeons cross-tabulated with the scores by the radiologists. The radiologists recorded SINS scores of 7 points or greater for all the unstable cases and for 621 of the 629 (98.7%) potentially unstable cases. In other words, all unstable and 98.7% of potentially unstable patients would have been appropriately identified as requiring surgical assessment. Twenty percent of the cases that the surgeons identified as stable were considered potentially unstable by the radiologists. Discussion Given the multidisciplinary approach toward the treatment of patients with metastatic spine disease, it is important to establish a clear framework for proper referral among health care professionals to ensure the best supportive care is provided. Before the SINS, reliable and validated guidelines to assess tumor-related spinal instability did not exist, and this lack of a standard created variability in interpretations, referral patterns, and consequently patient management. Radiologists and the advanced imaging modalities (e.g., CT and MRI) they interpret have a critical role in the evaluation of spinal metastases and the treatment decisions for these TABLE 2: Levels of Agreement for Kappa Values Kappa Value Level of Agreement Slight Fair Moderate Substantial > 0.80 Excellent Note Adapted with permission from [13] (Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33: ). patients. It is therefore essential that radiologists have a reliable and valid tool on which to evaluate neoplastic instability something that they currently do not have. Although SINS has not undergone prospective clinical evaluation, it was derived using evidence-based medicine and is the closest thing to a reference standard that we have. This study has shown that, among radiologists, the SINS binary scale provides a reliable tool for rating tumor-related spinal instability with excellent validity, substantial interobserver agreement, and excellent intraobserver reproducibility. These results indicate that SINS is a tool that can accurately discriminate between stable and potentially unstable lesions, an important decision when considering the need for surgical consultation. Using radiographic features to help clinicians select the most appropriate treatment of metastatic spinal disease has been previously reported. Snyder et al. [14] used a CT-based structural rigidity analysis, which they referred to as CTRA, as a noninvasive method to predict vertebral fracture risk in breast cancer patients with spinal metastases and suggested that surgical stabilization be considered for patients identified with a high fracture risk. CTRA, however, requires additional software and is time-consuming, with an analysis time of 2 4 hours per patient, thereby currently limiting its use. Furthermore, unlike fractures in the appendicular skeleton, not all fractures in the spine are unstable; therefore, a more comprehensive approach to the assessment of spinal stability is necessary. In contrast, the SINS framework for assessing spinal instability is based on six common radiographic and clinical parameters and does not rely on sophisticated analysis methods. One study assessed the interrater reliability of a range of imaging features of spinal metastases and noted moderate or poor agreement for most criteria between musculoskeletal radiologists and orthopedic surgeons [15]. The set of radiologic components included in the SINS were selected by the SOSG by combining the best available literature and consensus expert opinion [7], an evidence-based medicine process. Using the surgeons consensus assessment as the reference for validity or reference standard is a limitation because SINS must be prospectively evaluated; however, in the early phases of the development of a classification system, these assumptions are necessary to ensure SINS is TABLE 3: Reliability Analysis of the Spinal Instability Neoplastic Scale (SINS) Among Radiologists and Surgeons (Reference Standard) Observers TABLE 4: Cross Tabulation of Clinical Categories Based on Spinal Instability Neoplastic Scale (SINS) Scores for 30 Cases Determined by Surgeons (Reference Standard) and 37 Radiologists Clinical Category Assigned by Radiologists a Interobserver Agreement (κ) Three Clinical Categories (95% CI) Clinical Category Assigned by Surgeons (No. of Interpretations) Stable Binary Scale (95% CI) Potentially Unstable Unstable Total Stable Potentially unstable Unstable Total a The radiologists first assessments were used for this analysis. Intraobserver Reproducibility (κ) Three Clinical Categories (95% CI) Binary Scale (95% CI) Radiologists 0.53 ( ) 0.76 ( ) 0.69 ( ) 0.82 ( ) Surgeons 0.65 ( ) 0.83 ( ) NA NA Note NA = not applicable. AJR:203, October

6 optimized before expensive and time-consuming prospective evaluation. Obtaining opinions from radiologists across seven international sites has enhanced the generalizability of this study s findings. In addition, an a priori power analysis ensured that a sufficient number of cases (n = 30) were included [16], and rearranging the case order before the second review minimized recall bias. Despite these strategies, some results of this study may have been skewed because the mock cases may not fully reflect real-life clinical settings. Radiologists depend on the referring clinician to describe the patient s pain characteristics and, in particular, its relationship to movement. For the radiologist-assigned SINS score to be accurate, this detail needs to be routinely provided as part of the radiology consultation. We are hopeful that as oncologists more commonly use SINS, this important characteristic will be provided. Furthermore, with the involvement of an integrated multidisciplinary team of specialists (oncologists, radiologists, pathologists, surgeons) becoming more prevalent in the care of oncology patients, the provision of the pain component of SINS to radiologists should be improved. The limited selection of images available for each case scenario may have negatively impacted on the reliability of measures such as discriminating between lytic and blastic lesions. The availability of complete sets of reconstructed CT and MR images and ability to scroll through these images may translate into better SINS reliability in the true clinical setting. The lack of images is a limitation of this study, but one that probably biases toward the null hypothesis. The statistical method for testing reliability also has limitations. The kappa statistic used in this study is influenced by different factors, such as the rater s prevalence, weighting, and bias. A low kappa value might reflect the inability of the tested classification or the raters to distinguish between two different categories that are closely related [17, 18]. Although a kappa value of between 0.81 and 1.0 arbitrarily translates into excellent agreement according to the widely accepted Landis and Koch [13] scale, this level of agreement may reflect an unacceptable difference in agreement in the clinical setting. Depending on the purpose of the score and the influence of the result, the margin of error that will be allowed must be determined [17, 18]. The reliability of SINS has been previously established among surgeons, and with this study, radiologists who play a pivotal role in advising oncologists about potentially serious problems related to neurologic compression and spinal instability. Although the current iteration of SINS has sound diagnostic Fisher et al. test parameters, it requires prospective clinical validation and may indeed undergo modification based on the results of these studies. For now, physicians involved in the evaluation and care of these complex patients have a reliable and provisionally valid tool to evaluate spinal instability secondary to metastatic disease something we have desperately needed for some time. Acknowledgments We are grateful to the radiologists who contributed to and participated in this study: Nafi Aygun, Sachin Gujar, Izlem Izbudak, Haris Sair, Ugo Albisinni, Eugenio Rimondi, Daniel Vanel, Alberto Zerbi, Mitesh Ganhdi, Greg Cowderoy, John McGuire, Jennifer Sommerville, Maria Gergely, Maria Puhl, Judit Sandor, Ildiko Takasc, Vince Kumar, Norman Leeds, Matthew Debnam, Komal Shah, Ramy Mansour, Eugene McNally, Simon Ostlere, James Teh, Tasha Ellchuk, Derek Fladeland, Haron Obaid, Sheldon Wiebe, Jason Chew, Jason Shewchuk, Talia Vertinsky, Mariko Doai, Toshifumi Gabata, Masataka Kitadate, Julia Fairbairn, Khalid Latief, and Rachel Musson. We thank our fellow spine surgeons who assisted in establishing the reference standard: Nicolas Dea, Adrienne Kelly, and Dennis Hartig. We also thank the following participating spine centers and their clinical personnel: University of British Columbia, BC Cancer Agency, and Vancouver Coastal Health, Vancouver, BC, Canada; University of Saskatchewan, Royal University Hospital, Saskatoon, SK, Canada; University of Toronto and Toronto Western Hospital Krembil Neuroscience Centre, Toronto, ON, Canada; The Johns Hopkins University School of Medicine, Baltimore, MD; The University of Texas M. D. Anderson Cancer Center, Houston, TX; Rizzoli Institute, Bologna, Italy; Oxford University Hospital NHS Trust, Oxford, UK; National Center for Spinal Disorders and Buda Health Center, Budapest, Hungary; Princess Alexandra Hospital, Brisbane Spine Reference Center, Brisbane, Australia; Kanazawa Medical University, Kahoku-gun, Japan; and Nottingham University Hospital NHS Trust Queen s Medical Centre, Nottingham, UK. We also thank AO Clinical Investigation and Documentation, especially Monica Daigl for performing the statistical analysis, and the AOSpine Knowledge Forum Tumor Steering Committee for managing and supporting this study. References 1. Hayat MJ, Howlader N, Reichman ME, Edwards BK. Cancer statistics, trends, and multiple primary cancer analyses from the Surveillance, Epidemiology, and End Results (SEER) program. Oncologist 2007; 12: Harel R, Angelov L. Spine metastases: current treatments and future directions. Eur J Cancer 2010; 46: Ortiz Gómez JA. The incidence of vertebral body metastases. Int Orthop 1995; 19: Falicov A, Fisher CG, Sparkes J, Boyd MC, Wing PC, Dvorak MF. Impact of surgical intervention on quality of life in patients with spinal metastases. Spine 2006; 31: Bilsky M, Smith M. Surgical approach to epidural spinal cord compression. Hematol Oncol Clin North Am 2006; 20: Bilsky MH, Azeem S. The NOMS framework for decision making in metastatic cervical spine tumors. Curr Opin Orthop 2007; 18: Fisher CG, DiPaola CP, Ryken TC, et al. A novel classification system for spinal instability in neoplastic disease: an evidence-based approach and expert consensus from the Spine Oncology Study Group. Spine (Phila Pa 1976) 2010; 35:E1221 E Eastley N, Newey M, Ashford RU. Skeletal metastases: the role of the orthopaedic and spinal surgeon. Surg Oncol 2012; 21: Galasko CS, Norris HE, Crank S. Spinal instability secondary to metastatic cancer. J Bone Joint Surg Am 2000; 82: Fourney DR, Frangou EM, Ryken TC, et al. Spinal instability neoplastic score: an analysis of reliability and validity from the Spine Oncology Study Group. J Clin Oncol 2011; 29: Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas 1960; 20: Fleiss JL. Measuring nominal scale agreement among many raters. Psychol Bull 1971; 76: Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33: Snyder BD, Cordio MA, Nazarian A, et al. Noninvasive prediction of fracture risk in patients with metastatic cancer to the spine. Clin Cancer Res 2009; 15: Khan L, Mitera G, Probyn L, et al. Inter-rater reliability between musculoskeletal radiologists and orthopedic surgeons on computed tomography imaging features of spinal metastases. Curr Oncol 2011; 18:e282 e Scheffe H. The analysis of variance. New York, NY: Wiley, Kottner J, Audigé L, Brorson S, et al. Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. J Clin Epidemiol 2011; 64: Sim J, Wright CC. The kappa statistic in reliability studies: use, interpretation, and sample size requirement. Phys Ther 2005; 85: AJR:203, October 2014