Comparison of Prediction Rules and Clinician Suspicion for Identifying Children With Clinically Important Brain Injuries After Blunt Head Trauma

ORIGINAL CONTRIBUTION Comparison of Prediction Rules and Clinician Suspicion for Identifying Children With Clinically Important Brain Injuries After Blunt Head Trauma Shireen M. Atabaki, MD, MPH, John D. Hoyle, Jr., MD, Jeff E. Schunk, MD, David J. Monroe, MD, Elizabeth R. Alpern, MD, MSCE, Kimberly S. Quayle, MD, Todd F. Glass, MD, MS, Mohamed K. Badawy, MD, Michelle Miskin, MS, Walton O. Schalick, MD, PhD, Peter S. Dayan, MD, MSc, James F. Holmes, MD, MPH, and Nathan Kuppermann, MD, MPH Abstract Objective: Children with minor head trauma frequently present to emergency departments (EDs). Identifying those with traumatic brain injuries (TBIs) can be difficult, and it is unknown whether clinical prediction rules outperform clinician suspicion. Our primary objective was to compare the test characteristics of the Pediatric Emergency Care Applied Research Network (PECARN) TBI prediction rules to clinician suspicion for identifying children with clinically important TBIs (citbis) after minor blunt head trauma. Our secondary objective was to determine the reasons for obtaining computed tomography (CT) scans when clinical suspicion of citbi was low. Methods: This was a planned secondary analysis of a previously conducted observational cohort study conducted in PECARN to derive and validate clinical prediction rules for citbi among children with minor blunt head trauma in 25 PECARN EDs. Clinicians recorded their suspicion of citbi before CT as <1, 1 5, 6 10, 11 50, or >50%. We defined citbi as 1) death from TBI, 2) neurosurgery, 3) intubation for From the Department of Pediatrics and Emergency Medicine, George Washington University School of Medicine (SMA), Washington, DC; the Department of Emergency Medicine, Michigan State University School of Medicine (JDH), Grand Rapids, MI; the Departments of Emergency Medicine and Pediatrics, Western Michigan University Homer Stryker School of Medicine (JDH), Kalamazoo, MI; the Department of Pediatrics, University of Utah School of Medicine (JES, MM), Salt Lake City, UT; the Department of Emergency Medicine, Howard County General Hospital (DJM), Columbia, MD; the Department of Pediatrics, University of Pennsylvania School of Medicine (ERA), Philadelphia, PA; the Department of Pediatrics, Feinberg School of Medicine, Northwestern University (ERA), Chicago, IL; the Department of Pediatrics, Washington University School of Medicine (KSQ), St. Louis, MO; the Department of Pediatrics, University of Cincinnati College of Medicine (TFG), Cincinnati, OH; the Department of Pediatrics, Nemours Children s Hospital (TFG), Orlando, FL; the Departments of Emergency Medicine and Pediatrics, University of Rochester School of Medicine and Dentistry (MKB), Rochester, NY; the Department of Emergency Medicine, University of Texas, Southwestern Medical Center (MKB), Dallas, TX; the Departments of Orthopedics Rehabilitation, University of Wisconsin School of Medicine (WOS), Madison, WI; the Department of Pediatrics, Columbia University College of Physicians and Surgeons (PSD), New York, NY; and the Department of Emergency Medicine (JFH, NK) and the Department of Pediatrics (NK), University of California, Davis School of Medicine, Sacramento, CA. Participating centers and site investigators are listed in Appendix A. Received September 22, 2015; revision received December 23, 2015; accepted December 28, 2015. Presented in part at the Pediatric Academic Societies Annual Meeting, Vancouver, Canada, May 2010; and the Society for Academic Emergency Medicine, Phoenix, AZ, June 2010. This work was supported by a grant from the Health Resources and Services Administration/Maternal and Child Health Bureau (HRSA/MCHB), Division of Research, Education, and Training (DRTE) and the Emergency Medical Services of Children (EMSC) Program (R40MC02461). This project was also supported in part by the Health Resources and Services Administration (HRSA), Maternal and Child Health Bureau (MCHB), Emergency Medical Services for Children (EMSC) Network Development Demonstration Program under cooperative agreements U03MC00008, U03MC00001, U03MC00003, U03MC00006, U03MC00007, U03MC22684, and U03MC22685. The information or content and conclusions are those of the author and should not be construed as the official position or policy of, nor should any endorsements be inferred by HRSA, HHS, or the U.S. Government. The authors have no relevant financial information or potential conflicts to disclose. Supervising Editor: Michelle L. Macy, MD, MS. Address for correspondence and reprints: Shireen M. Atabaki, MD, MPH; e-mail: satabaki@childrensnational.org. 566 ISSN 1069-6563 566 2016 by the Society for Academic Emergency Medicine 566 PII ISSN 1069-6563583 doi: 10.1111/acem.12923

ACADEMIC EMERGENCY MEDICINE May 2016, Vol. 23, No. 5 www.aemj.org 567 more than 24 hours for TBI, or 4) hospital admission of 2 nights or more associated with TBI on CT. To avoid overfitting of the prediction rules, we performed comparisons of the prediction rules and clinician suspicion on the validation group only. On the validation group, we compared the test accuracies of clinician suspicion > 1% versus having at least one predictor in the PECARN TBI age-specific prediction rules for identifying children with citbis (one rule for children <2 years [preverbal], the other rule for children >2 years [verbal]). Results: In the parent study, we enrolled 8,627 children to validate the prediction rules, after enrolling 33,785 children to derive the prediction rules. In the validation group, clinician suspicion of citbi was recorded in 8,496/8,627 (98.5%) patients, and 87 (1.0%) had citbis. CT scans were obtained in 2,857 (33.6%) patients in the validation group for whom clinician suspicion of citbi was recorded, including 2,099/7,688 (27.3%) of those with clinician suspicion of citbi of <1% and 758/808 (93.8%) of those with clinician suspicion >1%. The PECARN prediction rules were significantly more sensitive than clinician suspicion >1% of citbi for preverbal (100% [95% confidence interval {CI} = 86.3% to 100%] vs. 60.0% [95% CI = 38.7% to 78.9%]) and verbal children (96.8% [95% CI = 88.8% to 99.6%] vs. 64.5% [95% CI = 51.3% to 76.3%]). Prediction rule specificity, however, was lower than clinician suspicion >1% for preverbal children (53.6% [95% CI = 51.5% to 55.7%] vs. 92.4% [95% CI = 91.2% to 93.5%]) and verbal children (58.2% [95% CI = 56.9% to 59.4%] vs. 90.6% [95% CI = 89.8% to 91.3%]). Of the 7,688 patients in the validation group with clinician suspicion recorded as <1%, CTs were nevertheless obtained in 2,099 (27.3%). Three of 16 (18.8%) patients undergoing neurosurgery had clinician suspicion of citbi <1%. Conclusions: The PECARN TBI prediction rules had substantially greater sensitivity, but lower specificity, than clinician suspicion of citbi for children with minor blunt head trauma. Because CT ordering did not follow clinician suspicion of <1%, these prediction rules can augment clinician judgment and help obviate CT ordering for children at very low risk of citbi. ACADEMIC EMERGENCY MEDICINE 2016;23:566 575 2016 by the Society for Academic Emergency Medicine There are more than 1.2 million emergency department (ED) visits for blunt head trauma annually in the United States, of which more than 500,000 are in children younger than 15 years. 1 Since the 1970s, computed tomography (CT) has emerged as the reference standard test for emergent identification of traumatic brain injury (TBI). However, CT use has associated risks and costs, the most significant of which is the lifetime attributable cancer mortality. This risk is estimated to be 1 in 1,400 imaged infants, 2,3 in addition to a two- to threefold increase in lifetime cancer risk associated with CT in the first 22 years of life. 4 Although the need for selective CT use in children has been recognized for years, 5 9 CT use almost doubled over the two decades preceding 2006, 3,10 with a stabilization in general EDs 11 and slight decline in pediatric EDs in recent years. 3,12,13 However, substantial variation in CT use among practitioners in both general and pediatric hospitals persists, 11 14 demonstrating the need for strategies to reduce variation in practice and further reduce unnecessary CT use. Decisions for cranial CT use for pediatric head trauma are frequently made by emergency clinicians, based on clinical suspicion of TBI and demonstrate a great deal of variability. 11 17 Furthermore, clinician suspicion of pathologic findings from common ED presenting traumatic complaints, such as blunt head trauma 15 17 and cervical spine trauma, 18 may not be sufficiently sensitive. The Pediatric Emergency Care Applied Research Network (PECARN) derived and validated two clinical prediction rules for detection of clinically important TBI (citbi) in children. 19 Each rule consists of six variables with one rule for children <2 years old and the other for children 2 years (Figure 1). Per the PECARN rules, a child is considered to be not at very low risk for citbi in the presence of any of the age-appropriate predictive variables; both rules have excellent performance characteristics. 19 Understanding the performance of prediction rules in comparison with clinician suspicion may aid clinicians in adoption of such rules and may enhance appropriate CT use in children with blunt head trauma. 20 22 In fact, there have been strong recommendations that clinical prediction rules be compared to unstructured clinician suspicion at the time of derivation/validation before widespread implementation. 23,24 Prior comparisons of TBI decision rules with clinician suspicion have been limited by the single-site nature or lower sensitivity of the derived prediction rules. 15,17 The objectives of the current study were to: 1) compare the test characteristics of clinician suspicion with those of a prediction rule to identify children with citbis after minor blunt head trauma and 2) determine clinicians reasons for obtaining CT scans for children with minor blunt head trauma when clinical suspicion of citbi was very low. METHODS Study Setting and Population We conducted a planned secondary analysis of data from a prospective observational cohort study at 25 PECARN centers. We derived the clinical prediction rules between June 2004 and March 2006. The current study compares clinician suspicion to the derived prediction rules on patients enrolled in the validation phase from March through September 2006. The Human Subjects Research Committee approved the study at each site. The methods have been previously described 19 and are briefly summarized here.

568 Atabaki et al. COMPARISON OF PREDICTION RULES AND CLINICIAN SUSPICION IN PEDIATRIC HEAD TRAUMA A. Patients < 2 years old Altered mental status** Temporal, parietal or occipital scalp hematoma History of loss of consciousness > 5 sec Severe mechanism of injury*** Palpable or unclear skull fracture Acting abnormally per parent B. Patients 2-18 years old Altered mental status** History of loss of consciousness History of emesis Severe mechanism of injury*** Signs of basilar skull fracture Severe headache Figure 1. Pediatric Emergency Care Applied Research Network Prediction (PECARN) rules for identifying children at very low risk of citbis* after blunt head trauma. 19 Prediction rules for citbi in children younger than 2 years (A) and in those aged 2 years and older (B). If none of the six variables in either rule were present, the risk of citbi was <0.02% for those younger than 2 years and <0.05% for those 2 18 years. *citbi definition: death, neurosurgery, intubation for more than 24 hours for TBI, or hospital admission of 2 nights or more associated with TBI on CT. **Altered mental status defined as a Glasgow Coma Scale score of 14, agitation, sleepiness, slow to respond, or repetitive questioning. ***Motor vehicle crash with patient ejection, death of another passenger, or rollover; pedestrian or bicyclist without helmet struck by a motorized vehicle; falls >3 feet (if <2 years); falls >5 feet (if 2 18 years); or head struck by a high-impact object. citbi = clinically important traumatic brain injury. Inclusion/Exclusion Criteria In the parent study, we included children <18 years with blunt head trauma and Glasgow Coma Scale (GCS) scores of 14 or 15, evaluated in any one of the PECARN EDs. For this analysis to compare clinician suspicion and the prediction rules, we used the validation population (n =8,627) to prevent unfair comparisons using prediction rules, which may be overfit to the data from which they were derived. We also included only patients for whom clinician suspicion of citbi was documented. We defined citbi using the following criteria: death from TBI, neurosurgery, intubation for more than 24 hours for TBI, or hospital admission of 2 nights or more associated with TBI on CT. 19 Patients were excluded for the following: ED presentation >24 hours after their injuries, penetrating trauma, known brain tumors, preexisting neurologic disorders complicating the assessment, neuroimaging at an outside hospital before transfer, or trivial mechanisms of injury (groundlevel falls or walking/running into stationary objects) in the absence of signs or symptoms of head injury other than scalp abrasions or lacerations. Study Protocol and Data Collection Clinicians were asked to perform standardized history and physical examinations and complete standardized data forms before CT scanning (if obtained). CT scans were obtained at the discretion of the treating physicians. We considered TBI to be present on CT in the presence of any of the following: intracranial bleeding, pneumocephalus, cerebral edema, skull fracture depressed by at least the thickness of skull, or traumatic skull diastasis. If a CT scan was obtained, the clinician recorded the most important indications influencing their decision for CT. Prior to knowledge of CT results (if obtained), clinicians prospectively recorded their suspicion of citbi. Hospitalizations for 2 or more nights for social reasons or awaiting other therapies were not considered to be citbi if no other criteria were met. Clinician suspicion of citbi was recorded on an ordinal scale as <1, 1 5, 6 10, 11 50, and >50%. Clinician suspicion was documented regardless of whether a CT was performed and prior to knowledge of CT results. Clinicians were unaware of the prediction rules during this evaluation as the rules were not yet derived. Outcome Measures Rates of citbi were determined by clinical follow-up both for admitted patients and for those discharged from the ED. For admitted patients, we performed medical record reviews and for discharged patients we performed telephone follow-up between 1 week and 3 months after the ED visit. If telephone follow-up was unsuccessful, we mailed follow-up surveys with the identical script as the telephone script. If that was unsuccessful, we reviewed the medical records, ED process improvement records, hospital trauma registries, and morgue records to ensure that no patients discharged from the ED and missing follow-up had citbis. Statistical Analysis We summarized the data using descriptive statistics with 95% confidence intervals (CIs). Using the validation dataset, we determined the sensitivities, specificities, negative and positive predictive values, and likelihood ratios for clinician suspicion of citbi of <1% and >1%. We also determined the same test characteristics for the prediction rules when they did not meet the very lowrisk criteria (i.e., had at least one predictor present). We compared the test performance of clinician suspicion of citbi to that of the two previously derived prediction rules (Figure 1) as described above. Very low clinician suspicion was selected to be a suspicion of <1% to maximize the sensitivity of clinician suspicion of citbi, while keeping this risk assessment feasible and comprehensible. We also analyzed the prevalence of citbi and documented indications for CT when clinician suspicion for citbi was <1%. Finally, we also determined the frequency of CT use by suspicion of any TBI on CT (i.e., not just citbi). All statistical analyses were performed using SAS 9.3 software (SAS Institute, Inc., Cary, NC). RESULTS There were 8,627 eligible patients enrolled in the validation group (Figure 2). Suspicion of citbi was recorded

ACADEMIC EMERGENCY MEDICINE May 2016, Vol. 23, No. 5 www.aemj.org 569 43904 Enrolled patients 35277 Excluded for present analysis 1035 GCS < 14 or unknown 340 Coagulopathy 101 Ventricular shunt 18 Missing citbi outcome data 33785 Derivation population * 2 patients had more than one exclusion criteria 8627 Validation patients 88 (1.02%) with citbi 8496 (98.5%) Patients with suspicion of citbi documented 7688 (90.5%) Patients with clinician suspicion of citbi < 1% 2099 (27.3%) CTs performed 808 (9.5%) Patients with clinician suspicion of citbi 1% 758 (93.8%) CTs performed Figure 2. Patient flow diagram. citbi = clinically important traumatic brain injury; CT = computed tomography; GCS = Glasgow Coma Scale. in 8,496 (98.5%) patients in the validation group, of whom 2,185 were younger than 2 years old (i.e., preverbal group) and 6,311 were 2 years or older (verbal group). We compared the prediction rules and clinician suspicion on these patients. Of note, in the parent study, there were 43,904 children <18 years enrolled. From that population, 42,412 had GCS scores of 14 or 15, and from these, the two age-dependent prediction rules were derived on 33,785 eligible children, and subsequently validated on the 8,627 eligible patients in the current analysis (Figure 2). There were small but non clinically meaningful differences in patient characteristics and outcomes between the derivation and validation cohorts (Table 1), with some comparisons found to be statistically significant while others were not. All subsequent analyses were limited to the 8,496 patients in the validation group for whom suspicion of citbi was recorded. For the validation group, the mean age was 6.8 years and most patients (62.6%) were male. CT scans were obtained on 33.6% of the patients, of whom 6.3% had TBI on CT scan. One percent (87/8,496) had citbis and 0.2% (16/ 8,496) had neurosurgery performed (Table 2). Of the 2,185 patients who were <2 years of age, 25 (1.1%) had citbis. Of the 6,311 patients who were 2 years of age and older, 62 (1.0%) had citbis. Among patients with clinician suspicion for citbi <1%, 2,099/7,688 (27.3%) had CT scans performed, as did 758/808 (93.8%) of those with clinician suspicion 1%. Tables 3A and 3B compare the test accuracies of clinician suspicion for citbi for children younger than 2 years and those 2 years and older with the two derived prediction rules. For the 2,185 children younger than 2 years of age, and the 6,311 children 2 years and older, the prediction rule had a substantially higher sensitivity compared to clinician suspicion of citbi. The prediction rules, however, had lower specificities than that of clinician suspicion of citbi for both preverbal children and verbal children. There were only two patients with citbis missed by the prediction rules. Neither of the two patients with citbis missed by the prediction rules required neurosurgery. We then determined the rates of CT scan ordering and of citbi according to the clinicians recorded suspicion of citbi. The rates of CT scan use and citbi both increased as the clinician s recorded suspicion of citbi increased (Table 4). Of note, clinicians obtained CT scans in 27.3% of children in whom citbi likelihood was suspected to be very low (<1%). Of substantial importance, of the 16 patients requiring neurosurgery, three had a clinician suspicion of citbi of <1%. This represents nearly 20% of total neurosurgeries. In contrast, none of the patients in the validation set who were considered low risk by the clinical prediction rules underwent neurosurgery. Next, we determined the clinicians primary indications for obtaining CT scans when their suspicion for citbi was low (<1%). Table 5 lists the three most fre-

570 Atabaki et al. COMPARISON OF PREDICTION RULES AND CLINICIAN SUSPICION IN PEDIATRIC HEAD TRAUMA Table 1 Characteristics of the Patients in the Validation Group (on Whom Clinician Suspicion of citbi Was Recorded) Versus the Derivation Group Derivation (N =33,785) Validation (N =8,496) Difference in means or % Age (y), mean (SD) 7.1 (5.6) 6.8 (5.4) 0.3 (0.2 to 0.4) Male 21,015 (62.2%) 5,322 (62.6%) 0.4% ( 1.6% to 0.7%) Cranial CT rate 11,961 (35.4%) 2,857 (33.6%) 1.8% (0.6% to 2.9%) TBI on CT 590/11,961 (4.9%) 180/2,857 (6.3%) 1.4% ( 2.3% to 0.4%) citbi 288 (0.9%) 87 (1.0%) 0.1% ( 0.4% to 0.1%) Neurosurgery 44 (0.1%) 16 (0.2%) 0.1% ( 0.2% to 0%) citbi = clinically important traumatic brain injury; CT = computed tomography. Table 2 Distribution of Types of citbi* in the Validation Cohort Overall (n =8,496), n (%; 95% CI) Clinician Suspicion <1% (n =7,688), n (%; 95% CI) Clinician Suspicion 1% (n =808), n (%; 95% CI) Type of Intervention Hospitalization 2 nights in 84 (1.0%; 0.8% to 1.2%) 32 (0.4%; 0.3% to 0.6%) 52 (6.4%; 4.8% to 8.4%) association with (+) CT Neurosurgery 16 (0.2%; 0.1% to 0.3%) 3 (0.04%; 0.01% to 0.1%) 13 (1.6%; 0.9% to 2.7%) Intubation for TBI 24 h 2 (0.02%; 0% to 0.1%) 1 (0.01%; 0% to 0.1%) 1 (0.1%; 0% to 0.7%) Death due to TBI 0 (0%; 0% to 0.04%) 0 (0%; 0% to 0.05%) 0 (0%; 0% to 0.5%) citbi = clinically important traumatic brain injury; CT = computed tomography. *citbi definition: death, neurosurgery, intubation for more than 24 hours for TBI, or hospital admission of 2 nights or more associated with TBI on CT. Fourteen patients had more than one type of intervention for citbi: three with clinician suspicion <1% and 11 with clinician suspicion 1%. Neurosurgeries for these three included craniotomy for evacuation of epidural hematoma in all three cases, one patient had multiple epidural hematomas. Table 3A Clinician Suspicion and Prediction Rule Accuracy for Detection of Children With citbi in Children <2 Years (n =2,185) Prediction Rule (i.e., at least One Factor Present) Clinician Suspicion 1% Sensitivity 25/25 (100%) (86.3 to 100%) 15/25 (60.0%) (38.7 to 78.9%) Specificity 1,158/2,160 (53.6%) 1,996/2,160 (92.4%) (51.5 to 55.7%) (91.2 to 93.5%) NPV 1,158/1,158 (100%) 1,996/2,006 (99.5%) (99.7 to 100%) (99.1 to 99.8%) PPV 25/1,027 (2.4%) 15/179 (8.4%) (1.6 to 3.6%) (4.8 to 13.4%) LR+ 2.2 (2.1 to 2.3) 7.9 (5.9 to 10.6) LR 0 (0 to 0.6) 0.4 (0.3 to 0.7) Difference % 40.0% (20.8% to 59.2%) 38.8% ( 41.2% to 36.4%) 0.5% (0.2% to 0.8%) 6.0% ( 10.1% to 1.8%) citbi = clinically important traumatic brain injury; LR+ =positive likelihood ratio; LR = negative likelihood ratio; NPV = negative predictive value; PPV = positive predictive value. quent indications recorded by the ED clinicians, by age group. There were 23 patients for whom indications for CT were not recorded by clinicians. For children younger than 2 years of age, the three most frequent indications were young age, mechanism of injury, and scalp hematoma. For children 2 years or older, the top three indications were mechanism of injury, a history of loss of consciousness, and headache. For the three patients who underwent neurosurgery despite the clinician suspicion of citbi <1%, the main indications for CT listed were as follows: patient 1 mechanism of injury and vomiting; patient 2 mechanism of injury and amnesia; patient 3 mechanism of injury, headache, and vomiting. We also analyzed other reasons which may influence a clinician s decision to obtain cranial CT scans, such as trauma team request, referring physician request, or parent expectation. We found that of the 2,099 CT scans ordered by clinicians when suspicion of citbi was <1%, trauma team request (74, 3.5%), referring physician

ACADEMIC EMERGENCY MEDICINE May 2016, Vol. 23, No. 5 www.aemj.org 571 Table 3B Clinician Suspicion and Prediction Rule Accuracy for Detection of Children With citbi in Children 2 Years (n =6,311) Sensitivity Specificity NPV PPV LR+ LR Prediction Rule (i.e., at Least One Factor Present) Clinician Suspicion 1% 60/62 (96.8%) (88.8% to 99.6%) 40/62 (64.5%) (51.3% to 76.3%) 3,635/6,249 (58.2%) 5,660/6,249 (90.6%) (56.9% to 59.4%) (89.8% to 91.3%) 3,635/3,637 (99.95%) 5,660/5,682 (99.6%) (99.80% to 99.99%) (99.4% to 99.8%) 60/2,674 (2.2%) 40/629 (6.4%) (1.7% to 2.9%) (4.6% to 8.6%) 2.3 (2.2 to 2.5) 6.8 (5.8 to 8.1) 0.1 (0.01 to 0.2) 0.4 (0.3 to 0.6) Difference % 32.3% (19.6% to 45.0%) 32.4% ( 33.8% to 31.0%) 0.35% (0.2% to 0.5%) 4.2% ( 6.1% to 2.1%) For Tables 3A and 3B, numerators for sensitivity and PPV represent the number of correct predictions of positive citbi findings and numerators for specificity and NPV represent the number of correct predictions of negative citbi findings. The denominators for sensitivity represent the total number of actual citbi findings and the denominators for specificity represent the total number of negative citbi findings. The denominators for PPV and NPV represent the total number of positive and negative citbi predictions respectively for each test. citbi = clinically important traumatic brain injury; LR+ =positive likelihood ratio; LR = negative likelihood ratio; NPV = negative predictive value; PPV = positive predictive value. Table 4 Rate of CT and citbi by Level of Clinician Suspicion of citbi Clinician Suspicion of citbi request (42, 2.0%), or parental anxiety or request (150, 7.1%) were occasionally listed. Of the 258 children with CT scans obtained for these requests (eight patients had two of these requests), two (0.8%) had citbis, and none required neurosurgery. Both patients had multiple predictors in the PECARN TBI rule. In addition clinicians were more likely to order CT scans when they had a suspicion 1% for presence of any TBI on CT (irrespective of clinical relevance; Table 6). However, clinicians still ordered CT scans for 17.1% of patients for whom their suspicion of any TBI on CT was <1% (Table 6). DISCUSSION Patients With Indicated Clinician Suspicion CT Obtained, n (%) citbi, n (%) <1% 7,688 2,099 (27.3) 32 (0.4) 1% 5% 686 637 (92.9) 28 (4.1) 6% 10% 85 85 (100.0) 16 (18.8) 11% 50% 30 29* (96.7) 8 (26.7) >50% 7 7 (100.0) 3 (42.9) citbi = clinically important traumatic brain injury; CT = computed tomography. *One patient was discharged from the ED without a CT scan. Follow-up telephone call was successful and the patient did not have a citbi or subsequent CT. In this study, which is a secondary analysis of data collected for a large, prospective multicenter study, we found that the PECARN prediction rules for citbi in children had a substantially higher sensitivity than clinician s suspicion of 1%. Although clinician suspicion of citbi had higher specificity than that of the prediction rule, clinicians often did not practice in accordance with Table 5 Most Frequent Reasons Cited for Obtaining a CT When Clinician Suspicion for citbi Was <1% Rationale for Cranial CT* Frequency Rate of citbi Rate of Neurosurgery <2 y old (n =505) Young age 346 (68.5) 7 (2.0) 0 (0) Mechanism of injury 187 (37.0) 4 (2.1) 1 (0.5) Scalp hematoma 135 (26.7) 4 (3.0) 0 (0) 2 y old (n =1,571) Mechanism of injury 693 (44.1) 18 (2.6) 2 (0.3) Loss of 466 (29.7) 7 (1.5) 0 (0) consciousness Headache 348 (22.2) 4 (1.1) 1 (0.3) Data are reported as n (%). citbi = clinically important traumatic brain injury; CT = computed tomography. *More than one reason for obtaining a CT can be cited for each patient. Indications for CT on 23 patients were not recorded. Only two patients older than 2 years underwent neurosurgery, and one had two of the reasons for CT listed above (mechanism of injury and headache). their suspicion as decisions to order cranial CT scans did not strictly follow clinician suspicion. Clinicians ordered cranial CT scans in 27.3% of children in whom clinician suspicion of citbi was <1%. In addition, if CT ordering adhered strictly to clinician suspicion, and CT scans had not been obtained for cases with clinician suspicion of <1%, three children in the validation data set who underwent neurosurgery would have been missed. All three patients had clinical predictors in the two PECARN prediction rules. One of the patients was younger than 2 years of age and had a high-risk mechanism of injury (fall more than 3 feet) and was acting abnormally per the parent. The other two were older than 2 years; one of the two had a high-risk mechanism

572 Atabaki et al. COMPARISON OF PREDICTION RULES AND CLINICIAN SUSPICION IN PEDIATRIC HEAD TRAUMA Table 6 Clinician Suspicion of Any TBI on CT and Rates of CT Obtained No n (%) ED CT performed Yes n (%) Suspicion for presence of any TBI on CT <1% 5,411 (82.9) 1,113 (17.1) 1% 5% 190 (12.8) 1,300 (87.2) 6% 10% 10 (3.3) 292 (96.7) 11% 50% 2 (2.0) 98 (98.0) >50% 0 (0) 34 (100.0) Missing clinician suspicion 26 (56.5) 20 (43.5) Data are reported as n (%). CT = computed tomography; TBI = traumatic brain injury. of injury (bicyclist without helmet struck by car) and the other had vomiting. Use of the prediction rules would have appropriately identified these children as not being at low risk of citbi while decreasing unnecessary cranial CT scans in children at very low risk by the prediction rules. This study was unique in that the outcome of interest was a combination of imaging results and clinical outcomes in a patient-oriented composite measure (citbi). Prior studies evaluating clinicians predictive abilities following pediatric blunt head trauma have assessed (only) emergency clinicians prediction of CT results as the relevant outcome. 15,17 By defining our outcome as we did, we avoided inclusion of small traumatic findings or incidental findings of CT without immediate clinical relevance. At the same time, this outcome allows for more appropriate use of clinical prediction rules and allows the rules to remain relevant as imaging technology changes and improves. In agreement with the current study, one prior study 15 trended toward increased sensitivity and another 17 demonstrated substantially greater sensitivity of prediction rules than clinician suspicion for predicting TBI on CT, despite substantially smaller sample sizes. The outcomes of these studies, however, were radiographic (i.e., TBI on CT) and therefore of lesser clinical importance. In addition, it is intuitive that identifying (by either clinician suspicion or clinical prediction rules) small radiographic findings on CT would be more difficult than identifying those that were clinically important, defined by their need for clinically meaningful interventions. One of our objectives was to uncover the rationale for CT use among children when clinician suspicion for citbi was low (<1%). We found that clinicians were most likely to obtain CT scans, despite low suspicion of citbi, based on young age, mechanism of injury and scalp hematoma in the preverbal group and mechanism of injury, loss of consciousness, and headache in the verbal group. Some clinicians may have opted to obtain CT scans despite their low suspicions of citbi because they desired to identify all TBIs regardless of their clinical importance. However, CT scans were still obtained in 17% when the clinician suspicion of any TBI on CT (clinically important or not) was <1%. This further supports the role of clinical prediction rules to assist with clinical judgment to decrease unnecessary CT scans. Most of the common reasons cited by clinicians for obtaining a cranial CT in spite of suspicion of citbi of <1% can be found among the variables in the PECARN prediction rules for both verbal and preverbal children. However, our group has also shown that when certain PECARN factors are present in isolation (i.e., without any other findings suggestive of TBI) the risk of citbi is very low. These include isolated severe mechanism of injury, 25 scalp hematomas, 26 loss of consciousness, 27 vomiting, 28 and headache. 29 Children with these isolated factors may frequently be observed for a period of time before CT decision-making. Making clinicians aware of important associations between the variables in the PECARN prediction rules and citbi may improve appropriateness of CT use by decreasing clinicians concerns for missing citbi when the risk is very small. Decision-making for cranial CT in the setting of blunt head trauma may also be influenced by patient/family request 30 or requests by referring physicians or trauma teams. The decision to obtain CT scans in this study in spite of low clinical suspicion for citbi was less frequently based on these reasons than on the clinical factors summarized previously. Centers participating in this study were primarily pediatric referral hospitals. Rates of cranial CT are lower in this setting than in nonchildren s hospitals, typically staffed by nonpediatric emergency medicine physicians. 10,31 We previously described the association of hospital setting and attending physician training with head CT use in the current study population. 14 Practicing in a children s hospital or having trained in pediatric emergency medicine was associated with a significantly lower frequency of obtaining CT scans compared to practicing in a non free-standing pediatric center or having trained in general emergency medicine. 14 As a result, the difference in accuracy between the prediction algorithms and unstructured clinician suspicion of citbi in nonchildren s hospitals (where most children with minor blunt head trauma are evaluated) may be even greater than that suggested by our study. 31,32 Our results may increase clinicians confidence in using prediction rules to augment clinician judgment regarding cranial CT for children with minor blunt head trauma. Use of the prediction rules may decrease unnecessary CT use in those at low risk of citbi compared to clinicians reliance on unstructured suspicion of citbi for decision-making, especially among those with less experience caring for children with blunt head trauma. This may come at a cost of lower specificity. However, as indicated above, having one of the PECARN prediction rule factors does not mandate a CT. Rather, it should elevate suspicion sufficiently to observe the child for a period of time in the ED or an observation unit before CT decision-making. Use of the prediction rules in these settings could also help avoid missing those with clinically important injuries, particularly those requiring neurosurgery. Prediction rules provide evidence to empower and complement clinical judgment. 33 Increased use of the prediction rules may decrease practice variation in

ACADEMIC EMERGENCY MEDICINE May 2016, Vol. 23, No. 5 www.aemj.org 573 ordering CT scans for minor blunt head trauma in children 34 and may also help mitigate racial/ethnic disparities in CT use among children with blunt head trauma. 30 Knowledge translation programs for implementation of these prediction rules are needed 35 and should take into account the effect of clinician suspicion of citbi on practice patterns regarding CT scan use for children with blunt head trauma. LIMITATIONS This study has several limitations. First, it was performed in a research network of predominantly pediatric hospitals or hospitals with dedicated pediatric EDs. The sensitivity and specificity of clinician suspicion in community EDs and hospitals with low annual pediatric volumes is unknown, but is unlikely to be better than the results reported here. It is also difficult for physicians to expressly state their probabilities of serious injury based on gestalt assessment or to correlate precisely with a specific risk number. We defined very low suspicion for citbi as that <1%, but could have selected other (lower) risk numbers. We selected <1% because this is both a sensitive threshold and a more feasible cutoff for identifying low risk by clinician suspicion than lower thresholds. For analytic purposes, we equated a suspicion of <1% as the threshold for failing to identify a patient with a citbi. However, many clinicians nevertheless obtained CT scans despite a reported <1% suspicion for citbi (and <1% suspicion for any TBI on CT regardless of clinical relevance). As we do not have timing of clinician suspicion in relation to CT decision-making, we cannot determine how a change in the patient status would affect clinical suspicion over time. Finally, CT scans were not obtained in all patients in this study. Because our outcome of interest was clinical and patient-oriented rather than radiographic, however, this should not substantially impact the results of our study. CONCLUSIONS The Pediatric Emergency Care Applied Research Network prediction rules for identifying children with clinically important traumatic brain injuries had greater sensitivity than clinician suspicion in both preverbal and verbal children with minor blunt head trauma. Although these rules were less specific than clinician suspicion, decisions to order cranial computed tomography did not strictly follow clinician suspicion. Therefore, application of the rules on children at very low risk of clinically important traumatic brain injury could result in less frequent computed tomography use. Application of accurate prediction rules can support clinical decision-making around cranial computed tomography use after pediatric blunt head trauma, enhancing appropriate use of imaging. We thank Rene Enriquez, BS, and Sally Jo Zuspan, RN, MSN, at the PECARN Data Coordinating Center (University of Utah) for their dedicated and diligent work; the research coordinators in PECARN, without whose dedication and hard work this study would not have been possible; and all the clinicians around the PECARN who enrolled children in this study. References 1. Faul M, Xu L, Wald MM, Coronado VG. Traumatic Brain Injury in the United States: Emergency Department Visits, hospitalizations and Deaths 2002-2006. Atlanta, GA: Centers for Disease Control and Prevention, National Center for Injury Prevention and Control, 2010. 2. Brenner DJ, Elliston CD, Hall EJ, Berdon W. Estimated risks of radiation-induced fatal cancer from pediatric CT. Am J Roentgenol 2001;176:289 96. 3. Miglioretti DL, Johnson E, Williams A, et al. The use of computed tomography in pediatrics and the associated radiation exposure and estimated cancer risk. JAMA Pediatr 2013;167:700 7. 4. Pearce MS, Salotti JA, Little MP, et al. Radiation exposure from CT scans in childhood and subsequent risk of leukaemia and brain tumours: a retrospective cohort study. Lancet 2012;380:499 505. 5. Committee on Quality Improvement, American Academy of Pediatrics and Commission on Clinical Policies and Research, American Academy of Family of Physicians. The management of minor closed head injury in children. Pediatrics 1999;104:1407 15. 6. National Cancer Institute. Radiation Risks and Pediatric Computed Tomography (CT): A Guide for Health Care Providers. Available at: http://cancer.- gov/cancerinfo/causes/radiation-risks-pediatric-ct. Accessed Feb 10, 2014. 7. Food and Drug Administration. FDA Public Health Notification. Pediatr Radiol 2002;32:314 6. 8. American Academy of Pediatrics. Five Things Physicians and Patients Should Question. Available at: https://www.aap.org/en-us/about-the-aap/aap-pressroom/pages/choosing-wisely-campaign.aspx. Accessed Feb 25, 2016. 9. Goske MJ, Applegate KE, Bulas D, et al. Image Gently: progress and challenges in CT education and advocacy. Pediatr Radiol. 2011;41(Suppl 2):461 6. 10. Blackwell CD, Gorelick M, Holmes JF, Bandyopadhyay S, Kuppermann N. Pediatric head trauma: changes in use of computed tomography in emergency departments in the United States over time. Ann Emerg Med 2007;49:320 4. 11. Marin JR, Weaver MD, Barnato AE, Yabes JG, Yealy DM, Roberts MS. Variation in emergency department head computed tomography use for pediatric head trauma. Acad Emerg Med 2014;21:987 95. 12. Lodwick DL, Cooper JN, Kelleher KJ, Brilli R, Minneci PC, Deans KJ. Variation in utilization of computed tomography imaging at tertiary pediatric hospitals. Pediatrics 2015;136:e1212 9. 13. Mannix R, Meehan WP, Monuteaux MC, Bachur RG. Computed tomography for minor head injury: variation and trends in major United States pediatric emergency departments. J Pediatr 2012;160:136 9.e1. doi:10.1016/j.jpeds.2011.06.024 14. Stanley R, Hoyle J, Dayan P, et al. Emergency department practice variation in computed tomog-

574 Atabaki et al. COMPARISON OF PREDICTION RULES AND CLINICIAN SUSPICION IN PEDIATRIC HEAD TRAUMA raphy use for children with minor blunt head trauma. J Pediatr 2014;165:1201 6. 15. Palchak MJ, Holmes JF, Kuppermann N. Clinical judgment versus a decision rule for identifying children at risk of traumatic brain injury on computed tomography after blunt head trauma. Pediatr Emerg Care 2009;25:61 65. 16. Klassen TP, Reed MH, Stiell IG, et al. Variation in utilization of computed tomography scanning for the investigation of minor head trauma in children: a Canadian experience. Acad Emerg Med 2000;7:739 44. 17. Atabaki SM, Stiell IG, Bazarian JJ, et al. A clinical decision rule for cranial computed tomography in minor pediatric head trauma. Arch Pediatr Adolesc Med 2008;162:439 45. 18. Bandiera G, Stiell IG, Wells GA, et al. The Canadian c-spine rule performs better than unstructured physician judgment. Ann Emerg Med 2003;42:395 402. 19. Kuppermann N, Holmes JF, Dayan PS, et al. Identification of children at very low risk of clinicallyimportant brain injuries after head trauma: a prospective cohort study. Lancet 2009;374:1160 70. 20. Graham ID, Stiell IG, Laupacis A, O Connor AM, Wells GA. Emergency physicians attitude toward and use of clinical decision rules for radiography. Acad Emerg Med 1998;5:134 40. 21. Fan E, Laupacis A, Pronovost PJ, Guyatt GH, Needham DM. How to use an article about quality improvement. JAMA 2010;304:2279 87. 22. Maguire JL, Kulik DM, Laupacis A, Kuppermann N, Uleryk EM, Parkin PC. Clinical prediction rules for children: a systematic review. Pediatrics 2011;128: e666 77. 23. Schriger DL, Newman DH. Medical decisionmaking: let s not forget the physician. Ann Emerg Med 2012;59:219 20. 24. Finnerty NM, Rodriguez RM, Carpenter CR, et al. Clinical decision rules for diagnostic imaging in the emergency department: a research agenda. Acad Emerg Med 2015;22:1406 16. 25. Nigrovic LE, Lee LK, Hoyle J, et al. Prevalence of clinically-important traumatic brain injuries in children with minor blunt head trauma and isolated severe injury mechanisms. Arch Pediatr Adolesc Med 2012;166:356 61. 26. Dayan PS, Holmes JF, Schutzman S, et al. Risk of traumatic brain injuries in children younger than 24 months with isolated scalp hematomas. Ann Emerg Med 2014;64:153 62. 27. Lee LK, Monroe D, Bachman MC, et al. Isolated loss of consciousness in children with minor blunt head trauma. JAMA Pediatr 2014;168:837 43. 28. Dayan PS, Holmes JF, Atabaki S, et al. Association of traumatic brain injuries with vomiting in children with blunt head trauma. Ann Emerg Med 2014;63:657 65. 29. Dayan PS, Holmes JF, Hoyle J, et al. Headache in traumatic brain injuries from blunt head trauma. Pediatrics 2015;135:504 12. 30. Natale JE, Joseph JG, Rogers AJ, et al. Cranial computed tomography use among children with minor blunt head trauma: association with race/ethnicity. Arch Pediatr Adolesc Med 2012;166:732 7. 31. Mannix R, Bourgeois FT, Schutzman SA, Bernstein A, Lee LK. Neuroimaging for pediatric head trauma: do patient and hospital characteristics influence who gets imaged? Acad Emerg Med 2010;17:694 700. 32. Gausche M, Rutherford M, Lewis RL. Emergency department quality assurance/improvement practices for the pediatric patient. Ann Emerg Med 1995;25:804 8. 33. Reilly BM, Evans AT. Translating clinical research into clinical practice: impact of using prediction rules to make decisions. Ann Intern Med 2006;144:201 9. 34. Hartling L, Scott-Findlay S, Johnson D, et al. Bridging the gap between clinical research and knowledge translation in pediatric emergency medicine. Acad Emerg Med 2007;14:968 77. 35. Sheehan B, Nigrovic LE, Dayan PS, et al. Informing the design of clinical decision support services for evaluation of children with minor blunt head trauma in the emergency department: a sociotechnical analysis. J Biomed Inform 2013;46:905 13. Appendix A Participating Centers and Site Investigators (in Alphabetical Order) Atlantic Health System/Morristown Memorial Hospital (M. Gerardi); Bellevue Hospital Center (M. Tunik, J. Tsung); Calvert Memorial Hospital (K. Melville); Children s Hospital Boston (L. Lee); Children s Hospital of Michigan (P. Mahajan); Children s Hospital of New York Presbyterian (P. Dayan); Children s Hospital of Philadelphia (F. Nadel); Children s Memorial Hospital (E. Powell); Children s National Medical Center (S. Atabaki, K. Brown); Cincinnati Children s Hospital Medical Center (T. Glass); DeVos Children s Hospital (J. Hoyle); Harlem Hospital Center (A. Cooper); Holy Cross Hospital (E. Jacobs); Howard County Medical Center (D. Monroe); Hurley Medical Center (D. Borgialli); Medical College of Wisconsin/Children s Hospital of Wisconsin (M. Gorelick, S. Bandyopadhyay); St. Barnabas Health Care System (M. Bachman, N. Schamban); SUNY Upstate Medical Center (J. Callahan); University of California Davis Medical Center (N. Kuppermann, J. Holmes); University of Maryland (R. Lichenstein); University of Michigan (R. Stanley); University of Rochester (M. Badawy, L. Babcock-Cimpello); University of Utah/Primary Children s Medical Center (J. Schunk); Washington University/St. Louis Children s Hospital (K. Quayle, D. Jaffe); Women and Children s Hospital of Buffalo (K. Lillis). We acknowledge the efforts of the following individuals participating in PECARN at the time this study was initiated: PECARN Steering Committee: N. Kuppermann, Chair; E. Alpern, J. Chamberlain, J. M. Dean, M. Gerardi, J. Goepp, M. Gorelick, J. Hoyle, D. Jaffe, C. Johns, N. Levick, P. Mahajan, R. Maio, K. Melville, S. Miller,* D. Monroe, R. Ruddy, R. Stanley, D. Treloar, M. Tunik, A. Walker.

ACADEMIC EMERGENCY MEDICINE May 2016, Vol. 23, No. 5 www.aemj.org 575 MCHB/EMSC Liaisons: D. Kavanaugh, H. Park. PECARN Data Coordinating Center: M. Dean, R. Holubkov, S. Knight, A. Donaldson. Data Analysis and Management Subcommittee: J. Chamberlain, Chair; M. Brown, H. Corneli, J. Goepp, R. Holubkov, P. Mahajan, K. Melville, E. Stremski, M. Tunik. Grants and Publications Subcommittee: M. Gorelick, Chair; E. Alpern, J. M. Dean, G. Foltin, J. Joseph, S. Miller,* F. Moler, R. Stanley, S. Teach. Protocol Concept Review and Development Subcommittee: D. Jaffe, Chair; K. Brown, A. Cooper, J. M. Dean, C. Johns, R. Maio, N. C. Mann, D. Monroe, K. Shaw, D. Teitelbaum, D. Treloar. Quality Assurance Subcommittee: R. Stanley, Chair; D. Alexander, J. Brown, M. Gerardi, M. Gregor, R. Holubkov, K. Lillis, B. Nordberg, R. Ruddy, M. Shults, A. Walker. Safety and Regulatory Affairs Subcommittee (SRAS): N. Levick, Chair; J. Brennan, J. Brown, J. M. Dean, J. Hoyle, R. Maio, R. Ruddy, W. Schalick, T. Singh, J. Wright. *Deceased.