The Journal of TRAUMA Injury, Infection, and Critical Care

Transcription

1 The Journal of TRAUMA Injury, Infection, and Critical Care Harborview Assessment for Risk of Mortality: An Improved Measure of Injury Severity on the Basis of ICD-9-CM T. Al West, MD, MPH, Frederick P. Rivara, MD, MPH, Peter Cummings, MD, MPH, Gregory J. Jurkovich, MD, and Ronald V. Maier, MD Background: There have been several attempts to develop a scoring system that can accurately reflect the severity of a trauma patient s injuries, particularly with respect to the effect of the injury on survival. Current methodologies require unreliable physiologic data for the assignment of a survival probability and fail to account for the potential synergism of different injury combinations. The purpose of this study was to develop a scoring system to better estimate probability of mortality on the basis of information that is readily available from the hospital discharge sheet and does not rely on physiologic data. Methods: Records from the trauma registry from an urban Level I trauma center were analyzed using logistic regression. Included in the regression were Internation Classification of Diseases-9th Rev (ICD-9- CM) codes for anatomic injury, mechanism, intent, and preexisting medical conditions, as well as age. Two-way interaction terms for several combinations of injuries were also included in the regression model. The resulting Harborview Assessment for Risk of Mortality (HARM) score was then applied to an independent test data set and compared with Trauma and Injury Severity Score (TRISS) probability of survival and ICD-9-CM Injury Severity Score (ICISS) for ability to predict mortality using the area under the receiver operator characteristic curve. Results: The HARM score was based on analysis of 16,042 records (design set). When applied to an independent validation set of 15,957 records, the area under the receiver operator characteristic curve (AUC) for HARM was This represented significantly better discrimination than both TRISS probability of survival (AUC , p 0.005) and ICISS (AUC , p 0.001). HARM also had a better calibration (Hosmer-Lemeshow statistic [HL] 19.74) than TRISS (HL 55.71) and ICISS (HL ). Physiologic data were incomplete for 6,124 records (38%) of the validation set; TRISS could not be calculated at all for these records. Conclusion: The HARM score is an effective tool for predicting probability of in-hospital mortality for trauma patients. It outperforms both the TRISS and ICD- 9-CM Injury Severity Score (ICISS) methodologies with respect to both discrimination and calibration, using information that is readily available from hospital discharge coding, and without requiring emergency department physiologic data. J Trauma. 2000;49: The quantitative characterization of injury severity has gained increasing attention over the last 20 years. The uniform application of accurate severity scales to the trauma patient population is essential for several reasons. First, comparison of therapeutic modalities and evaluations of patient outcomes can be meaningful only if objective measures of their injury severity can be assigned. Effective use of prehospital and interhospital triage depends on reliable estimates of the severity of a patient s injuries. Quality improvement efforts and prevention programs rely on such objective data to make policy decisions and set priorities. Similarly, trauma research requires a tool by which to assure that study groups are compared with counterparts with equivalent degrees of injury. Submitted for publication October 4, Accepted for publication May 11, Copyright 2000 by Lippincott Williams & Wilkins, Inc. From the Department of Surgery, University of Texas Southwestern Medical Center, Dallas, Texas (T.A.W.) and Department of Surgery, University of Washington (R.V.M., G.J.J.) and Harborview Injury Prevention and Research Center (F.P.R., P.C.), Seattle, Washington. Address for reprints: T. Al West, MD, MPH, Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Boulevard, Dallas, TX The modern era of injury severity scoring began in the early 1950s with the pioneering work of DeHaven. 1 Since then, there have been numerous attempts to develop a scoring system that can accurately reflect the severity of a patient s injuries, particularly with respect to the effect of those injuries on survival. 2 9 However, these scoring systems have attributes that limit their ability to reliably predict mortality. First, there is often a loss of information. Physiologic and anatomic data from the patient are combined into intermediate scores, which are then combined (often in several complex steps) to achieve a final probability of survival score. Second, injury combinations are modeled as if the effects of the injuries are independent, although it is well recognized clinically that some combinations may be more lethal than a simple additive or multiplicative model would predict. Third, there are no current injury scoring systems that take preexisting disease, a widely recognized contributor to outcome, 10,11 into account when assigning probability of survival. Most scoring systems rely on injury descriptions and physiologic data that are not easily and economically obtainable from standard discharge data. Last, prehospital or emergency department physiologic data is often missing, making calculation of probability of survival impossible. 530 September 2000

2 HARM: A New Probability of Survival Score The Committee on Injury Prevention and Control at the Institute of Medicine has recommended validation of existing models, as well as development of new models, using common databases and standard measures of scale performance. 12 We propose an alternative approach to injury severity scoring in an effort to rectify several drawbacks of the existing injury coding schemes. This injury severity classification method attempts to explicitly address the possibility that certain injury combinations might contribute to mortality beyond their independent effects. In addition, it takes comorbid disease into account when predicting mortality. Finally, this method uses data that are readily available for all patients, without relying on missing or inaccurate physiologic data. METHODS Data were obtained from the Harborview Medical Center Trauma Registry. This computerized database contains information concerning all patients evaluated for traumatic injury at Harborview Medical Center who were either admitted to the hospital or died in the emergency department. The registry is used for quality improvement and clinical research and provides data to the Washington State Trauma Registry. Harborview Medical Center is a Level I trauma center and is a tertiary referral center for trauma, including burns and pediatric injuries, for the states of Washington, Alaska, Montana, and Idaho. Information was obtained for all trauma admissions and emergency room deaths between July 1, 1985, and December 31, No treatment-related variables were included in the analysis; only those variables that were determined upon or before the individual s arrival at the hospital were used in initial model development. The data set was divided into two groups for the purposes of developing and testing the model. A 50% random sample from the entire set was selected as the design set; the remaining records were analyzed separately as the validation set. The design set was used to determine the best prediction model, and the validation set was used to test the accuracy of the model on an independent set of data. Statistical analysis was performed using the program Stata (College Station, TX). We first reclassified the 2,034 different Internation Classification of Diseases-9th Rev (ICD-9-CM) 13 diagnosis codes representing injuries (codes ) into 109 injury categories. Each category represented a group of anatomically similar injuries. Every effort was made to balance the need for fine enough separation of different injury types (more categories) with the need for simplicity in modeling (fewer categories). In this classification scheme, many distinct ICD- 9-CM codes were collapsed into a broader category. However, there were many individual ICD-9-CM codes that contributed information to several different injury categories (Fig. 1). ICD-9-CM codes that corresponded to Abbreviated Injury Scale (AIS) 1 severity scores less than 3 were excluded, Fig. 1. This single ICD-9-CM code for fracture of skull base, open with cerebral laceration or contusion, with moderate (1 24 hours) loss of consciousness and return to preexisting conscious level, would contribute to four injury categories: skull fracture, open intracranial wound, cerebral laceration or contusion, and moderate-length loss of consciousness. parallel to the approach taken by the authors of the ASCOT model. 8 These codes represent minor injuries, such as superficial wounds and minor orthopedic injuries, that have been found not to contribute significantly to mortality predictions. 8 Burns and burn-related injuries were also excluded from the reclassification scheme, as the predictive model was not intended to estimate severity of burns nor predict burn mortality. A list of the injury categories and their corresponding ICD-9-CM codes can be found in Table 1. We also sought to include comorbid conditions in the model as well as injuries. ICD-9-CM codes were combined into 11 categories of comorbid disease on the basis of the earlier work of Morris and MacKenzie 10 (Table 1). The mechanism of injury and its associated intent were classified on the basis of the primary External Cause of Injury code, defined within ICD-9-CM. These codes were classified according to recommendations from the Centers for Disease Control and Prevention. 14 Sixteen specific mechanisms and five classes of intent were represented, as defined in Table 1. A panel of experienced trauma surgeons produced a list of 25 two-way interactions among the injury categories defined above that were judged to be clinically significant. To maintain a balance between specificity and simplicity, interactions were created from categories that were broader than Volume 49 Number 3 531

3 The Journal of TRAUMA Injury, Infection, and Critical Care Table 1 Anatomic Diagnoses, Comorbid Conditions, Mechanism Categories, and Intent Categories Considered for the Model Anatomic Diagnoses Considered for the Model Injury ICD-9-CM Codes Included Skull fracture , Open intracranial wound (.4.9), (.4.9), (.1,.3,.5,.7,.9) Facial fracture 802 Concussion (.9), (.9), 850, (.9) Cerebral laceration or contusion (.1,.6), (.1,.6), (.0.3,.8) Cerebellar or brainstem hemorrhage (.4.7,.9) Subarachnoid hemorrhage 852(.0,.1) Epidural hemorrhage 852(.4,.5) Subdural hemorrhage 852(.2,.3) Unspecified intracranial hemorrhage (.2,.3,.7,.8), (.2,.3,.7,.8) Unspecified intracranial injury (.4,.9), (.4,.9), 854 Loss of consciousness for 24 hours (.x2,.x3), (.x2,.x3), 850(.x1,.x2), (.x2,.x3) Loss of consciousness for 24 hours (reversible) (.x4), (x4), 850(.x3), (.x4) Loss of consciousness for 24 hours (irreversible) (.x5), (x5), 850(.x4), (.x5) Unspecified period of loss of consciousness (.x6), (x6), 850(.x5), (.x6) Complete spinal cord injury C4 or above 806(.01,.11), Incomplete spinal cord injury C4 or above 806(.02.04,.12.14), 952(.02.04) Unspecified spinal cord injury C4 or above 806(.00,.10), Complete spinal cord injury C5 T12 806(.06,.16,.21,.26,.31,.36), 952(.06,.11,.16) Incomplete spinal cord injury C5 T12 806(.07.09,.17.19,.22.24,.27.29,.32.34,.37.39), 952(.07.09,.12.14,.17.19) Unspecified spinal cord injury C5 T12 806(.05,.15,.20,.25,.30,.35), 952(.05,.10,.15) Lumbar/sacral/cauda equina cord injury 806(.4.7), 952(.2.4) Unspecified spinal cord injury 806(.8.9), 952(.8.9) Fracture of larynx or trachea 807(.5.6) Uncomplicated open wound of larynx 874(.00.02) Complicated open wound of larynx 874(.10.12) Uncomplicated open wound of pharynx Complicated open wound of pharynx Common or internal carotid artery injury 900(.01,.03) External/unspecified carotid artery injury 900(.00,.02) Internal jugular vein injury Other/unspecified neck vessel injury 900(.8.9) Crush injury to neck Flail chest Fracture of 1 3 ribs 807(.00.03,.10.13) Fracture of 3 ribs 807(.04.09,.14.19) Pneumothorax 860(.0.1) Hemothorax 860(.2.3) Hemopneumothorax 860(.4.5) Sternal fracture 807(.2.3) Cardiac contusion 861(.01,.11) Laceration of heart 861(.02,.12) Full-thickness cardiac laceration 861(.03,.13) Unspecified cardiac injury 861(.00,.10) Pulmonary contusion 861(.21,.31) Pulmonary laceration 861(.22,.32) Unspecified lung injury 861(.20,.30) Diaphragm injury 862(.0.1) Bronchus injury 862(.21,.31) Esophagus injury 862(.22,.32) Other/unspecified intrathoracic injury 862(.29,.39,.8.9) Thoracic aorta or great vessels 901(.0.1) SVC or innominate vein 901(.2.3) Pulmonary vessels 901(.40.42) Unspecified thoracic vessels 901(.81.89) Open thoracic wound (.1x,.3x,.5x,.7x,.9x) Stomach injury 863(.0.1) Duodenal injury 863(.21,.31) Small intestine injury 863(.20,.29,.30,.39) SVC, superior vena cava; GI, gastrointestinal; IVC, inferior vena cava; E-codes, external cause of injury codes; NEC, not elsewhere classifiable. 532 September 2000

4 HARM: A New Probability of Survival Score Table 1 continued Anatomic Diagnoses Considered for the Model Injury ICD-9-CM Codes Included Colon injury 863(.40.44,.46,.49,.50.54,.56,.59) Rectal injury 863(.45,.55) Unspecified GI tract injury 863(.80,.89,.90,.99) Injury to head of pancreas 863(.81,.91) Injury to body of pancreas 863(.82,.92) Injury to tail of pancreas 863(.83,.93) Multiple/unspecified pancreatic injury 863(.84,.94) Hematoma/contusion/minor hepatic laceration 864(.01.02,.11.12) Moderate laceration of liver 864(.03,.13) Major laceration of liver 864(.04,.14) Other/unspecified liver injury 864(.00,.09.10,.19) Hematoma or capsular tear of spleen 865(.01.02,.11.12) Moderate laceration of spleen 865(.03,.13) Major laceration of spleen 865(.04,.14) Other/unspecified splenic injury 865(.00,.09.10,.19) Minor kidney injury 866(.01.02,.11.12) Major kidney injury 866(.03,.13) Other/unspecified renal injury 866(.00,.10) Bladder injury 867(.1.2) Ureter injury 867(.2.3) Uterus injury 867(.4.5) Other/unspecified abdominal organ injury 867(.6.9), 868, 869 Open abdominal wound 863(.1x,.3x,.5x,.9x), (.1x), 867(.1x,.3x,.5x,.7x,.9x), (.1x) Popliteal artery injury Traumatic amputation above the knee 897(.2.3) Bilateral lower extremity amputation 897(.6.7) Crush injury above the knee 928(.00.01,.8) Femoral artery injury 904(.0.1) Open pelvic fracture 808(.1,.3,.5x,.9), Closed pelvic fracture 808(.0,.2,.4x,.8), Hip fracture 820 Abdominal aortic injury IVC injury 902(.11,.19) Hepatic vein injury Celiac/mesenteric artery injury 902(.2x) Portal vein injury Mesenteric venous injury 902(.31.32) Splenic vein injury Iliac vessel injury 902(.50.54,.59) Renal vessel injury 902(.4x) Other/unspecified abdominal vessel injury 902(.55.56,.8x,.9) Comorbid Conditions Considered for the Model Comorbid Condition Chronic obstructive pulmonary disease Congenital coagulopathy (excluding 286.6, 287.4) Diabetes mellitus 250 Cirrhosis 571 Ischemic heart disease Hypertension Psychoses Epilepsy 345 Obesity 278 Alcohol or drug dependence Neurologic degenerative disease ICD-9-CM Diagnosis Codes Mechanism Categories Mechanism Cut/pierce 920, 956, 966, 986, 974 Fall , 888, 957, 968.1, 987 Firearm 922, 955(.0.4), 985(.0.4), 970 ICD-9-CM E-Codes Volume 49 Number 3 533

5 The Journal of TRAUMA Injury, Infection, and Critical Care Table 1 continued Mechanism Categories Mechanism ICD-9-CM E-Codes Machinery 919 Motor vehicle collision occupant (.0.1) Motorcycle collision (.2.3), (.2.3) Motor vehicle/bicycle collision (.6) Motor vehicle/pedestrian collision (.7) Motor vehicle collision unspecified (.9) Bicycle collision (.3), (.6), 826(.1,.9) Pedestrian (.2), (.7), (.0) Other transport (.0.1,.8.9), (.0.5,.8.9), 826(.2.8), (.2.9), 83, , 958.6, Struck by, against , 960.0, 968.2, 973, 975 Other specified, classifiable , , 918, 921, 923, , 929, 955(.5,.9), 958(.0,.4), 910.1, 965(.5.9), 967, 968.4, 985.5, 988(.0,.4), 971, 978, , 996, 997(.0.2) Other specified, NEC 928.8, 929.8, 958.8, 959, 968.8, 969, 988.8, 989, 977, 995, 997.8, Unspecified 887, 928.9, 929.9, 958.9, 968.9, 988.9, 976, Intent Categories Intent Unintentional , Self-inflicted Assault Undetermined Legal intervention , ICD-9-CM E-Codes any main effects categories in the model. A detailed description of the interaction terms is found in Table 2. Forward stepwise logistic regression analysis was carried out on the design set, with in-hospital death as the dependent variable. Each of the 109 injury categories was included as a binary variable (representing presence or absence of injury in that category), as were terms for each of the mechanism and intent categories. Two-way interaction terms, as noted above, were also included in the stepwise regression. Age was modeled as a quadratic spline, 15 with knots at ages 12, 25, and 65 years. Terms were included in the final model if the Wald statistic reached a significance of p All main effects that contributed to a significant interaction term were included in the model. The prediction model determined from the design set was then applied to the validation set. The coefficients for the predictor variables were used to calculate the Harborview Assessment for Risk of Mortality (HARM) score. This score takes the form of a probability, between 0 and 1, of inhospital mortality: P m 1/(1 e ). 0 1 (age) 2 (age2) (chest spinal cord), where 0 is a constant and n is the regression coefficient associated with each term in Table 3. The presence or absence of each of the mechanisms and injury categories is represented by 1 or 0 in this equation. Abbreviated Injury Scale severities were derived from the ICD-9-CM data for each of the patients using the ICDMAP program. 16 These AIS scores were used to calculate an ISS and then combined with age and Revised Trauma Score data using published coefficients. 17 An ICISS score 9 was also derived for each validation set patient, using a dictionary of the survival risk ratios derived from the North Carolina Hospital Discharge database. All injury ICD-9-CM codes between 800 and were used for this calculation. To compare the ability of HARM, ICISS, and Trauma and Injury Severity Score (TRISS) to discriminate between survivors and nonsurvivors, a receiver operator characteristic (ROC) curve 18 was derived for each scoring system on the basis of the validation set. The area under the curve (AUC) was calculated and compared using a nonparametric method. 19 The calibration characteristics of the three scores were compared using the Hosmer-Lemeshow (HL) statistic. Sensitivity and specificity were calculated for each score as well. RESULTS A total of 33,990 admissions for blunt or penetrating trauma were recorded in the Harborview Medical Center Trauma Registry between July 1, 1985, and December 31, To avoid analyzing readmissions associated with the same injury for a given patient, only the first admission for any particular patient was included, resulting in 1,683 excluded stays. The final data set thus consisted of 32,307 admissions or emergency department deaths; 16,185 admis- 534 September 2000

6 HARM: A New Probability of Survival Score Table 2 Definitions of Interaction Categories and Interaction Terms Considered for the Model Definitions of Interaction Categories Broad Injury Category Included Injury Categories Head Skull fracture Open intracranial wound Facial fracture Concussion Cerebral laceration or contusion Cerebellar/brainstem laceration Subarachnoid hemorrhage Epidural hemorrhage Subdural hemorrhage Unspecified intracranial hemorrhage Unspecified intracranial injury Neck Fracture of larynx or trachea Uncomplicated open wound of larynx Uncomplicated open wound of larynx Complicated open wound of pharynx Uncomplicated open wound of pharynx Common/internal carotid artery injury External/unspecified carotid artery injury Internal jugular vein injury Other/unspecified vessel in neck Spinal cord Complete cord injury C4 or above Incomplete cord injury C4 or above Unspecified cord injury C4 or above Complete cord injury C5 T12 Incomplete cord injury C5 T12 Unspecified cord injury C5 T12 Lumbar/sacral/cauda equina injury Unspecified spinal cord injury Chest wall/pleura Flail chest Fracture of 1 3 ribs Fracture of 3 ribs Pneumothorax Hemothorax Hemopneumothorax Sternal fracture Thoracic vascular Thoracic aorta or great vessel injury SVC or innominate vein injury Pulmonary vessel injury Unspecified thoracic vessel injury Lung Pulmonary contusion Pulmonary laceration Unspecified lung injury Heart Cardiac laceration Full-thickness cardiac laceration Cardiac contusion Unspecified cardiac injury Liver Minor liver laceration/contusion/hematoma Moderate liver laceration Major liver laceration Other/unspecified liver injury Hepatic vein injury Spleen Hematoma or capsular tear of spleen Moderate laceration of spleen Major laceration of spleen Pancreas Injury to head of pancreas Injury to body of pancreas Injury to tail of pancreas Multiple/unspecified pancreatic injuries SVC, superior vena cava; COPD, chronic obstructive pulmonary disease. Table 2 continued Definitions of Interaction Categories Broad Injury Category Abdominal vascular Kidney Pelvis Interaction Terms Considered for the Model Head chest wall Head thoracic vascular Head liver Head pelvis Head spinal cord Head age 65 Chest wall spinal cord Chest wall liver Chest wall pelvis Chest wall COPD Lung COPD Heart ischemic heart disease Liver colon Liver cirrhosis Liver abdominal vascular Colon abdominal vascular Colon pancreas Colon spleen Pancreas abdominal vascular Pancreas small intestine Pancreas kidney Kidney abdominal vascular Hip fracture age 65 Pelvis age 65 Pelvis rectum Table 3 Characteristics of study population Characteristic Design Set (n 16,185) Included Injury Categories Abdominal aortic injury Inferior vena cava injury Celiac/mesenteric artery injury Portal vein injury Mesenteric venous injury Splenic vein injury Iliac vessel injury Renal vessel injury Other/unspecified abdominal vascular injury Minor kidney injury Major kidney injury Other/unspecified kidney injury Open pelvic fracture Closed pelvic fracture Validation Set (n 16,122) p Value Mean age (y) Male gender (%) 73.8% 74.4% 0.2 Blunt injury (%) 81.6% 81.6% 0.3 Mean ISS Mortality (%) 6.7% 6.9% 0.3 sions were in the design set, and 16,122 were in the validation set. The study population was predominantly young and male, and most injuries were the result of blunt trauma (Table 3). The mean ISS was 10.9 (median 9). There were no significant differences between the design and validation sets Volume 49 Number 3 535

7 The Journal of TRAUMA Injury, Infection, and Critical Care Table 4 Variables Included in Final Model Independent Variable Adjusted Odds Ratio Regression Coefficient Constant Age-related variables Age Age Quadratic spline term, age Quadratic spline term, age Quadratic spline term, age Mechanism of injury Fall Firearm Machinery Motor vehicle collision, unspecified Motor vehicle collision, occupant Motor vehicle/motorcycle collision Motor vehicle/bicycle collision Motor vehicle/pedestrian collision Bicycle collision Pedestrian Other transport Struck by, against Other specified, classifiable Other specified, not elsewhere classified Unspecified Injury categories Loss of consciousness 24 hours (irreversible) Loss of consciousness 24 hours Skull fracture Open intracranial wound Facial fracture Concussion Cerebral laceration or contusion Cerebellar or brainstem hemorrhage Subarachnoid hemorrhage Epidural hemorrhage Subdural hemorrhage Unspecified intracranial hemorrhage Unspecified intracranial injury Complete spinal cord injury C4 or above Incomplete spinal cord injury C4 or above Unspecified spinal cord injury C4 or above Complete spinal cord injury C5 T Incomplete spinal cord injury C5 T Unspecified spinal cord injury C5 T Lumbar/sacral/cauda equina cord injury Unspecified spinal cord injury Complicated open wound of pharynx Other/unspecified neck vessel injury Flail chest Fracture of 1 3 ribs Fracture of 3 ribs Pneumothorax Hemothorax Hemopneumothorax Cardiac contusion Cardiac laceration Full-thickness cardiac laceration Unspecified cardiac injury Pulmonary contusion Pulmonary laceration Thoracic aorta or great vessels Superior vena cava or innominate vein September 2000

8 HARM: A New Probability of Survival Score Table 4 continued Independent Variable Adjusted Odds Ratio Regression Coefficient Colon injury Injury to tail of pancreas Moderate laceration of liver Major laceration of liver Hepatic vein injury Major laceration of spleen Minor kidney injury Other/unspecified abdominal organ Traumatic amputation above the knee Open pelvic fracture Closed pelvic fracture Abdominal aortic injury Inferior vena cava Mesenteric venous injury Comorbidities Congenital coagulopathy Cirrhosis Ischemic heart disease Hypertension Psychoses Alcohol or drug dependence Interaction terms Head spinal cord Pelvis (age 65) Chest spinal cord with regard to age, gender, mortality, mean ISS, or proportion of injuries attributable to blunt trauma. The final logistic regression model contained 80 terms (Table 4). Of the 100 total injury categories considered, 51 were included in the model. Sixteen were not significant individually but were included as main effects terms for the modeled interactions. Six of the 11 terms representing comorbid conditions were included in the final model. All 16 mechanism categories were represented as 15 dummy variables. Three interaction terms were included. Finally, modeling age as a quadratic spline with three knots required five age-related variables. When applied to the validation set, the HARM model was able to calculate a probability of mortality in 16,097 of the 16,122 admissions (99.9%). ICISS was successful in 15,820 cases (98.1%). Because of missing physiologic data, a TRISS probability of survival score could be calculated for 9,923 (61.4%) of the patients in the validation set. HARM had the highest AUC at (Fig. 2, Table 5). For ICISS, the AUC was TRISS was in between, with an AUC of HARM had a better fit to the validation data (HL statistic 21.37, p ) than ICISS (HL 712.4, p ) or TRISS (HL 59.54, p 0.005). Note that a smaller HL statistic connotes a better fit to the actual data. Because the TRISS score could be calculated only for 61.4% of the total validation set, ROC curves and HL statistics were derived for each model in the subset of records for which all three scores could be calculated. Nonparametric derivation of the AUC allows for a calculation of standard Fig. 2. Comparison of ROC curves for HARM, TRISS, and ICISS. error, thus allowing relevant comparisons to be made. 19 The HARM score maintains its larger AUC in this subset, and the difference is statistically significant (Table 6). To calculate the sensitivity and specificity of each score for predicting survival, an arbitrary cutoff was chosen for each score so that sensitivity was fixed at 95%. In other words, 95% of survivors had a predicted probability of survival above this cutoff value. Specificity was then determined Volume 49 Number 3 537

9 The Journal of TRAUMA Injury, Infection, and Critical Care Table 5 Predictive Power for Survival Using HARM, TRISS, and ICISS Model N a Area under ROC HL Statistic p Value for HL Statistic HARM 16, TRISS 9, ICISS 15, a Number of patients for which score is calculable. Table 6 Comparison of Area under ROC Curves for HARM, TRISS, and ICISS on the Basis of Identical Validation Sets (n 9923) Model Area under ROC p Value for Comparison with HARM HARM TRISS ICISS Table 7 Comparison of Specificities for HARM, TRISS, and ICISS Model Sensitivity Cutoff Probability Specificity 85% % HARM 90% % 95% % 85% % TRISS 90% % 95% % 85% % ICISS 90% % 95% % Table 8 Ten Most Lethal Injuries in HARM Model Independent Variable Adjusted Odds Ratio Loss of consciousness for 24 hours (irreversible) 95.2 Full-thickness cardiac laceration 67.2 Unspecified cardiac injury 32.0 Complete spinal cord injury C4 or above 30.9 Superior vena cava or innominate vein 28.4 Pulmonary laceration 27.3 Cardiac contusion 22.4 Traumatic amputation above the knee 21.4 Major laceration of liver 14.6 Thoracic aorta or great vessels 13.5 at that cutoff point for each score and compared (Table 7). The specificity for HARM was 83.4%, for ICISS was 78.20%, and for TRISS was 72.1%. DISCUSSION Although TRISS has been the de facto standard for the prediction of mortality among trauma patients for many years, it has several limitations. 4,8 9,20 21,26 28 In addition to theoretical issues, perhaps the most important limitation is in its real-life applicability to patients with missing physiologic data. The goal of this study was to develop a scoring system that can predict mortality by using patient data that are reliably available by extracting as much information as possible out of that data. The resulting HARM score possesses excellent power in discriminating between survivors and nonsurvivors. In addition, it has better calibration than either TRISS or ICISS. The accurate assignment of AIS and ISS scores to patients requires the time and effort of an individual who understands the procedures for deriving the scores, be that a nurse coordinator or a trauma registrar. Many trauma centers, in an effort to obviate that need, rely on the ICDMAP program 16 to convert ICD-9-CM injury diagnosis codes into AIS scores for use in calculation of ISS and TRISS. However, there is not a one-to-one correspondence for most of the ICD-9-CM diagnoses, and many injuries are inaccurately scored by this method. There have been promising results from several studies assessing the ability to predict survival on the direct basis of ICD-9-CM codes. 9,23 26 A recent study concludes that an ICD-9-CM based score is equally accurate whether on the basis of assignment by a dedicated trauma registrar or discharge data from the standard hospital information system. 22 The HARM score coefficients are based on logistic regression analysis of more than 16,000 trauma patients at a Level I trauma center. The basis for the score is a modification of the ICD-9-CM coding scheme, which has been shown in several studies to be useful in stratifying trauma patients. Logistic regression was chosen for the statistical method in this study because of its ability to model the interactions between independent variables and because it does not require the assumption of independence among variables. Variables were selected for inclusion in the final model on the basis of their ability to add predictive power to the final score. The adjusted odds ratios for the 10 most significant injury categories are listed in Table 4. Examination of these top 10 odds ratios reveals a familiar pattern to those who are involved in clinical trauma care. Prolonged coma is overwhelmingly the most important variable in the model, underscoring the devastating effects of major head injury on survival. Cardiac injuries, both specific ones, such as a fullthickness laceration, and unspecified ones, were also important in the model. Major liver injury, pulmonary laceration, and various abdominal vascular injuries are also on the list. 538 September 2000

10 HARM: A New Probability of Survival Score Several comorbid factors were also found to be important, most notably, cirrhosis, congenital coagulopathy, and ischemic heart disease (IHD). The most important determinants of survival in this model seem to reflect clinical experience. Although many different interactions were explored, there were only three that were significant enough for inclusion in the final model. Pelvic fracture was associated with a higher mortality in older patients ( 65 years) than in younger patients after adjustment for the presence of all other injuries. Spinal cord injuries had a positive interaction with both head and chest injuries. It is unclear why the interaction factors did not play more of a role in the model. It may have been because of the overwhelming influence of coma and other injury categories. Another explanation may be that the broader categories defined for the interactions led to a watering-down effect as more severe injuries were included in the same category with less severe injuries. Different modeling methodologies, and perhaps larger design data sets, may make it easier to capture the clinically recognized importance of synergism between injuries. HARM assigns a probability of mortality without the use of physiologic data. The underlying hypothesis is that given enough information about patients physiologic reserves, mechanisms of injury, and anatomic injury patterns, their responses to an insult can be predicted. One potential weakness of HARM, therefore, is that two patients with the same injuries, mechanism, comorbidities, and age would have the same score regardless of their vital signs in the emergency room. However, the point of HARM is to predict survival on the basis of factors established at the time of the injury itself. This approach avoids the inherent problems with physiologic data: there is often no way to know the time elapsed from the injury until the vital signs were taken, and inconsistent use of prehospital versus emergency room data will yield different physiologic scores. TRISS fails to assign a probability score to a significant percentage of patients. In this study, the ROC curves were recalculated for HARM and ICISS on the subset of patients for which TRISS did assign a score. The AUCs remain unchanged for each score for this subset, and it seems that HARM clearly outperforms the other two scores. HARM achieved this high level of calibration without the use of physiologic data and predicted a score in nearly every patient in the data set. In using any probability score in an applied setting, such as for quality assurance, it is important to use the proper cut point as a divider between predicted survivors and nonsurvivors. The sensitivity and specificity for any particular score vary depending on what cutoff is used. Sensitivity will increase only at the expense of specificity and vice versa. Therefore, to compare the sensitivity and specificity of any group of scores, one of the two values must be fixed. For the purposes of this comparison, the sensitivity was fixed at 95%, and the specificity at that point was compared. It can be seen from Table 7 that the price (in misclassification of nonsurvivors) paid by HARM for its high sensitivity is less than that for both TRISS and ICISS. The split sample technique used in this study addresses the issue of validity by applying the HARM model to a second set of unique data and determining how well the model predicts for a population different from that for which it was designed. HARM was found to work well in this second sample. However, this does not assure the ability to generalize the results of this study to other settings. The Harborview Medical Center patient population may be homogeneous in comparison with other hospital populations. In addition, because the basis of the HARM is in ICD-9-CM codes, its ability to predict mortality in the Harborview data set may be related to local consistencies in coding; other hospitals with slightly different coding conventions may not fare as well with the present HARM coefficients. Application of the HARM model to a data set from a different hospital with different coding conventions may not achieve the same results. True verification of the model will require testing it with a completely independent data set, preferably one with a heterogeneous collection of data sources. ICD-10-CM coding will soon be adopted for use in coding hospitals discharges. If the HARM method is validated at other institutions, this suggests that it will be worthwhile creating a new set of coefficients on the basis of the ICD-10-CM codes, using a method similar to that used to develop the HARM score. CONCLUSION This study confirms earlier work that indicates that injury severity scores based on ICD-9-CM codes can predict mortality with as much or more accuracy than those based on the AIS rubric with considerably less effort and expense. Furthermore, it suggests that this predictive power can be gained without the use of physiologic data from the emergency department. The HARM score, as outlined here, is a predictive score that can accurately predict outcome on the basis of data that are derived entirely from factors that are established before or at the time of an injury and easily abstracted from standard hospital discharge data. Perhaps more importantly, it can be used not only for patients for whom physiologic data are available but for any patient whose injuries are known. REFERENCES 1. DeHaven H. The Site, Frequency and Dangerousness of Injury Sustained by 800 Survivors of Light Plane Accidents. New York: Crash Injury Research, Cornell University Medical College; Association for the Advancement of Automotive Medicine. The Abbreviated Injury Scale,1990 Revision. Des Plaines, IL: AAAM; Teasdale G, Jennett B. Assessment of coma and impaired consciousness: a practical scale. Lancet. 1974;2: Baker SP, O Neill B, Haddon W, et al. The injury severity score: a method for describing patients with multiple injuries and evaluating emergency care. J Trauma. 1974;14: Volume 49 Number 3 539

11 The Journal of TRAUMA Injury, Infection, and Critical Care 5. Champion HR, Sacco WJ, Carnazzo AJ, et al. Trauma score. Crit Care Med. 1981;9: Champion HR, Sacco WJ, Copes WS, et al. A revision of the Trauma Score. J Trauma. 1989;29: Copes WS, Sacco WJ, Champion HR, et al. Progress in characterizing anatomic injury severity. Paper presented at the 33rd Annual Proceedings, Association for the Advancement of Automotive Medicine, 1989; Des Plaines, IL. 8. Champion HR, Copes WS, Sacco WJ, et al. A new characterization of injury severity. J Trauma. 1990;30: Osler T, Rutledge R, Deis J. ICISS: an International Classification of Disease-9 based injury severity score. J Trauma. 1996;41: Morris JA, MacKenzie EJ, Edelstein SL. The effect of preexisting conditions on mortality in trauma patients. JAMA. 1990;263: Milzman DP, Boulanger BR, Rodriguez A, et al. Pre-existing disease in trauma patients: a predictor of fate independent of age and injury severity score. J Trauma. 1992;32: Bonnie RJ, Fulco CE, Liverman CT, eds. Reducing, the Burden of Injury: Advancing Prevention and Treatment. New York: National Academy Press; Commission of Professional Hospital Activities. International Classification of Disease, 9 th Revision, Clinical Modification. Ann Arbor, MI: Edwards Brothers; Centers for Disease Control, and Prevention. Recommended framework for presenting injury mortality data. MMWR. 1997;46: Greenland S. Dose-response and trend analysis in epidemiology: alternatives to categorical analysis. Epidemiology. 1995;6: Mackenzie EJ, Steinwachs CM, Shankar B. Classifying trauma severity based on hospital discharge diagnoses: validation of an ICD- 9 CM to AIS-85 conversion table. Med Care. 1989;27: Champion HR, Sacco WJ, Copes WS. Injury severity scoring again [editorial]. J Trauma. 1995;38: Metz CE. Basic principles of ROC analysis. Semin Nucl Med. 1978; 8: DeLong E, DeLong D, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44: Markle J, Cayten CT, Burne DW, et al. Comparison between TRISS and ASCOT methods in controlling for injury severity. J Trauma. 1992;33: Hannan EL, Mendeloff J, Farrell LS, et al. Validation of TRISS and ASCOT using a non-mtos trauma registry. J Trauma. 1995;38: Osler T, Cohen M, Rogers FB, et al. Trauma registry injury coding is superfluous: a comparison of outcome prediction based on trauma registry International Classification of Diseases, 9th Revision (ICD- 9), and hospital information ICD-9 codes. J Trauma. 1997;43: Rutledge R. Injury severity and probability of survival assessment in trauma patients using a predictive hierarchical network model derived from ICD-9 codes. J Trauma. 1995;38: Rutledge R, Fakhry S, Baker C, et al. Injury severity grading in trauma patients: a simplified technique base upon ICD-9 coding. J Trauma. 1993;35: Rutledge R, Hoyt D, Eastman AE, et al. Comparison of the injury severity score and ICD-9 diagnosis codes as predictors of outcome in injury: analysis of 44, 032 patients. J Trauma. 1997;42: Rutledge R, Osler T, Emery S, et al. The end of the ISS and TRISS: ICISS outperforms both ISS and TRISS as predictors of trauma patient survival, hospital charges, and hospital length of stay. J Trauma. 1998;44: Osler T. Injury severity scoring: perspectives in development and future directions. Am J Surg. 1993;165:43 51S. 28. Champion HR, Copes WS, Sacco WJ, et al. Improved predictions from ASCOT over TRISS: results of an independent evaluation. EDITORIAL COMMENT In this issue of the Journal, West et al. examine the Harborview Assessment for Risk of Mortality (HARM), a new predictor of trauma patient mortality based on ICD-9- CM. The HARM methodology departs substantially from most earlier efforts to predict mortality for trauma patients in several ways. First, it does not use any physiologic measures. Second, it takes preexisting disease (comorbidities) into account. Third, it includes numerous (15) mechanisms of injury in its final logistic regression model rather than merely separating patients on the basis of blunt and penetrating injuries, or on the basis of more aggregated mechanisms such as MVCs, low falls, other blunt injuries, stab wounds, and gunshot wounds. Fourth, it reclassifies 2,034 ICD-9-CM injury diagnosis codes into 109 injury categories, 51 of which were used in its final statistical model. Fifth, it includes interactions of injury categories. The final HARM statistical model utilizes a total of 80 variables, with 5 representing age, 15 for mechanisms of injury, 51 for injury categories, 6 for comorbidities, and 3 interaction terms. The authors conclude that HARM outperforms Trauma and Injury Severity Score (TRISS) and International Classification of Diseases, Ninth Revision (ICD-9)-based Injury Severity Score (ICISS) with respect to both discrimination and calibration, the two most commonly used measures of fit for logistic regression models. The fact that HARM does not require emergency department physiologic data is claimed as an additional advantage. We believe that there are a few caveats that should be expressed regarding these findings. First, for TRISS, the authors used published coefficients derived from the MTOS study which is at least 10 years old. ICISS scores were derived using a dictionary of survival/risks ratios derived from the North Carolina Hospital discharge database. By validating HARM using a split sample method from one hospital and then comparing the results to TRISS and ICISS with coefficients derived from vastly different databases, the authors have biased the study in terms of favoring their new model. The truest test of whether this model outperforms TRISS and ICISS would be to derive TRISS coefficients and ICISS risk ratios from the Harborview data set, and then compare all three models using either the Harborview or an independent data set. 1 In particular, the comparisons of calibration among HARM, TRISS, and ICISS are inappropriate when the underlying population mortality rates (in the initial MTOS [for TRISS], North Carolina [ICISS] and Harborview [HARM] are different. 1 Regarding the specific statistical model that is presented by West et al., there are several factors/variables that do not have face validity in that they appear to be protective of mortality, yet clearly should not be. Examples are comorbidities such as alcohol or drug dependence, psychoses, and 540 September 2000

12 HARM: A New Probability of Survival Score hypertension. For example, the interpretation of the contribution of psychoses to the model is that patients with psychoses have odds of dying in the hospital that are 0.16 times (roughly one sixth) the odds of patients without psychoses dying in the hospital, all other risk factors being the same. This clearly makes no sense and is perhaps just a function of a relatively small number of patients with psychoses in the study (or a relatively small number whose psychoses were coded). If this methodology is transported to other sites that do not have sufficient data to identify their own significant variables and associated coefficients, factors without face validity should be removed. One could also argue that these factors should not be used internally at Harborview. Also, there are some other investigations that would be of interest in exploring the relative advantages/disadvantages of HARM. For example, the contention that HARM outperforms methodologies that include physiologic measures would have been more convincing if the Revised Trauma Score (RTS) data elements had been added to ICISS as they were to the Injury Severity Score (ISS) to form TRISS. As indicated by Rutledge et al., 2 ICISS is a better predictor when physiologic measures are added. It would also have been interesting to see whether HARM could have been substantially improved by adding physiologic measures, or whether they would have been superfluous given the large number of other variables present. Despite the caveats presented above, we believe that the paper by West et al. is a valuable addition to the literature on predicting adverse outcomes for trauma patients. The authors have done some creative and valuable out of the box thinking in proposing a major departure from previous methodologies. Their ideas of subdividing broader categories of mechanisms of injury and explicitly using numerous injury categories in the model are particularly appealing and intriguing. We anxiously look forward to other studies that compare HARM with TRISS and ICISS/RTS, and applaud West et al. for their efforts to enhance our ability to understand the factors that influence mortality among trauma patients. Edward L. Hanan, PhD Department of Health Policy, Management, and Behavior School of Public Health State University of New York, University at Albany Albany, New York C. Gene Cayten, MD, MPH Department of Surgery New York Medical College and Our Lady of Mercy Medical Center Bronx, New York REFERENCES 1. Hannan E, Farrell L, Gorthy S, et al. Predictors of mortality in adult patients with blunt injuries in New York State: a comparison of the Trauma and Injury Severity Score (TRISS) and the International Classification of Disease, Ninth Revision-based Injury Severity Score (ICISS). J Trauma. 1999;47: Rutledge R, Osler T, Emery S, et al. The end of the ISS and TRISS: ICISS outperforms both ISS and TRISS as predictors of trauma patient survival, hospital charges, and hospital length of stay. J Trauma. 1998;44: Volume 49 Number 3 541