UNIVERSITY OF CALGARY. Development and Validation of an ICD-10 Case Definition for Pediatric Traumatic Brain Injury

Transcription

1 UNIVERSITY OF CALGARY Development and Validation of an ICD-10 Case Definition for Pediatric Traumatic Brain Injury using Canadian Administrative Data by Jane Madison McChesney A THESIS SUBMITTED TO THE FACULTY OF GRADUATE STUDIES IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE DEPARTMENT OF COMMUNITY HEALTH SCIENCES CALGARY, ALBERTA DECEMBER, 2012 Jane Madison McChesney 2012

2 Abstract Background: Administrative data are used for disease surveillance and research. A validated case definition to study pediatric traumatic brain injury (TBI) using administrative data is not available. Methods: Using systematic review methodology, we identified studies validating TBI ICD coding. Second, a cohort of children with and without TBI from a pediatric emergency department (ER) was linked to administrative hospital and emergency databases. Nine ICD-10 based algorithms were tested. Sensitivity, specificity, and positive/negative predictive value (PPV, NPV) were calculated. Results: We identified three studies from the systematic review with varying diagnostic accuracy, and none were pediatric specific. The best algorithm to identify TBI cases was, 1 Hospital or 1 ER visit in 1 year : sensitivity 69.8%, ( ), specificity 96.7% ( ), PPV 84.2% ( ), NPV 92.7% ( ). Conclusion: This study algorithm is best to capture ER or hospitalized TBI cases. Future research should develop algorithms which include children seen in the community. ii

3 Acknowledgements I would like to thank my supervisor, Dr. Nathalie Jetté for being an incredible mentor and support to me both before, and during my graduate studies. I would like to extend my thanks to the dedicated members of my supervisory committee: Dr. Karen Barlow, Dr. Hude Quan, and Dr. Samuel Wiebe, for their guidance and incredible expertise. I would like to thank the Western Regional Training Centre for Health Services Research for their studentship support and opportunities for professional growth. A special thanks to Mr. Jim Si for his assistance with the administrative data linkage, and to Dr. Guanmin Chen for his invaluable guidance with the data analysis. iii

4 Dedication To my incredibly loving and supportive family who made me what I am. iv

5 Table of Contents Abstract... ii Acknowledgements... iii Dedication... iv Table of Contents... v List of Tables... ix List of Figures... x List of Abbreviations... xi CHAPTER 1: INTRODUCTION Overview of the Research Problem Study Purpose Study Objectives Hypotheses... 2 CHAPTER 2: BACKGROUND AND LITERATURE REVIEW Defining Brain Injury Acquired Brain Injury Traumatic Brain Injury (TBI) Mild, Moderate, and Severe Traumatic Brain Injury Head Injury versus Traumatic Brain Injury Epidemiology and Impact of Traumatic Brain Injury The Influence of Age and Sex on Traumatic Brain Injury Causes of Traumatic Brain Injury Costs Associated with Traumatic Brain Injury v

6 2.3 Surveillance for Traumatic Brain Injury What is Surveillance? Common Sources of Data for Surveillance Administrative Data International Classification of Disease System use in Administrative Data International Classification of Diseases, 10 th Edition (ICD-10) International Variations in ICD Coding Validated Case Definition for Traumatic Brain Injury Surveillance Tests of Diagnostic Accuracy Identifying Study Cohort Sensitivity Specificity Predictive Values Relationship with Prevalence Summary CHAPTER 3: METHODS AND STATISTICAL ANALYSIS Study Objectives Systematic Review Study Design Search Strategy Study Selection Data Extraction Quality Assessment vi

7 3.2.6 Statistical Analysis Validation of ICD-10 Coding and Identification of Best Case Definition Study Design Study Site Study Population and Identification of Traumatic Brain Injury Cases Additional TBI Diagnosis Validation by Telephone Interviews and Clinic Visits Data Sources Used Regional Emergency Department Information System (REDIS) Ambulatory Care Classification System Discharge Abstract Database Administrative Data Linkage ICD-10 and ICD-10-CA Codes for Traumatic Brain Injury Algorithms Used to Identify Best Case Definition Ethics Approval CHAPTER 4: RESULTS Systematic Review Development of a Validated Case Definition for Pediatric Traumatic Brain Injury Characteristics of the Study Population Additional TBI Diagnosis Validation by Telephone Interviews and Clinic Visits Validation of ICD-10 Coding ICD-10 Codes used for False-Negative TBI Cases Identification of Best Algorithm to Identify TBI Cases vii

8 CHAPTER 5: DISCUSSION Key Findings Discussion of Systematic Review Findings Strengths of Systematic Review Limitations of Systematic Review Discussion of the Validation of a Case Definition for Pediatric Traumatic Brain Injury Study Strengths Study Limitations Potential Impact of Bias Public Health Implications Areas for Future Research Conclusions BIBLIOGRAPHY APPENDIX A: Data Abstraction Form for Systematic Review APPENDIX B: Quality Assessment Tool from Benchimol et al. (2011) APPENDIX C: Full List of ICD-10 and ICD-10-CA Codes used to Identify TBI viii

9 List of Tables Table 1. Glasgow Coma Scale... 5 Table 2. Classification of TBI Severity based on the Glasgow Coma Scale... 6 Table 3. Summary of Canadian Population-Based TBI Articles... 9 Table 4. List of ICD Versions and Years of Adoption by the World Health Assembly Table 5. Comparison between ICD-9 and ICD-10 Classifications Table 6. Examples of ICD-10 Coding Variations between International Clinical Modifications 19 Table 7. Standard 2x2 Table for Calculating Diagnostic Accuracy Table 8. Relationship between Predictive Values and Disease Prevalence Table 9. Standard 2x2 Table: Effect of Specificity on Positive Predictive Value Table 10. Alberta Children s Hospital REDIS Complaint Codes List Table 11. ICD-10 Codes used to Identify TBI Table 12. Algorithms used to Test for Best Case Definition for Pediatric TBI Table 13. List of Excluded Articles and Reason for Exclusion Table 14. Summary of Sensitivity, Specificity, Positive and Negative Predictive Value, Reported by TBI Validation Studies Table 15. Quality Assessment as Outlined by Benchimol et al. (2011) Table 16. Frequency of ICD-10 Codes used in Administrative Data to Identify TBI Table 17. TBI Validation Results for Algorithms including Hospital or ER visits ix

10 List of Figures Figure 1. Timeline Figure 2. Flow-Diagram of Systematic Review Findings Figure 3. Flow-Diagram of Study Population x

11 List of Abbreviations Abbreviation AAN ABI ACCS ACH CAEP CCDSS CDC CCS CIHI CNS DAD ED EHR EMR ER GCS ICD NACRS NPHS NPV NHCC Definition American Academy of Neurology Acquired brain injury Ambulatory Care Classification System Alberta Children s Hospital Canadian Association of Emergency Physicians Canadian Chronic Diseases Surveillance System Centers for Disease Control Children s Coma Scale Canadian Institute for Health Information Central nervous system Discharge Abstract Database Emergency department Electronic health record Electronic medical record Emergency room Glasgow Coma Scale International Classification of Diseases National Ambulatory Care Reporting System National Population Health Survey Negative predictive value Neurological Health Charities of Canada xi

12 NOS PHAC PPV REDIS Sn Sp TBI WHO Not otherwise specified Public Health Agency of Canada Positive predictive value Regional Emergency Department Information System Sensitivity Specificity Traumatic brain injury World Health Organization xii

13 Chapter One: INTRODUCTION 1.1 Overview of the Research Problem Traumatic brain injury (TBI) is the leading cause of death and disability in children and young adults in the United States, with a combined estimate of 511,257 emergency department visits, hospitalizations, and deaths per year [1]. While the overall prevalence of TBI in Canada is currently unknown, the Canadian Institute for Health Information (CIHI) reported that in 2007, children (<18 years) represented almost 45% of the patients with head injury visiting emergency departments and urgent care centres, and almost 25% of the patients hospitalized with head injury [2]. Pediatric TBI is of particular concern as it can lead to significant and lifelong neurologic and cognitive deficits [3], with a substantial and permanent burden on the patient, their caregivers and society. In June 2008, the Neurological Health Charities of Canada (NHCC) was created as a collective organization of Societies, Federations, and Associations representing Canadians with neurological conditions. Due to the efforts of the NHCC, the Government of Canada identified neurological diseases as a priority in the Speech from the Throne in November 2008 [4]. Subsequently, the National Population Health Study of Neurological Conditions was launched in collaboration with the NHCC and the Public Health Agency of Canada (PHAC). Study leaders and stakeholders identified accurate epidemiological data as fundamental to planning for the needs of persons with neurological conditions. Traumatic brain injury (TBI) was identified as one of the 14 priority conditions [4]. The need to develop Canadian surveillance programs for TBI and other neurological conditions, using cost effective ascertainment sources such as administrative data, was identified as a priority [4]. In fact, the PHAC Neurological Disease Surveillance Advisory Committee was launched in 2009, using administrative data as its primary 1

14 source of ascertainment for neurological conditions [4]. However, a validated case definition to study TBI in Canada is not available. 1.2 Study Purpose The purpose of this study was to provide key information on data quality, and recommendations on best case definitions necessary to carry out TBI research in children, using Canadian administrative data. The study will also provide the methodological foundation needed for subsequent studies, aimed at studying health services utilization, and outcomes in Canadian children with TBI using administrative data. 1.3 Study Objectives The objectives of this study were to: 1. Perform a systematic review of studies which have validated TBI coding in administrative data. 2. Assess the validity of ICD-10 coding for TBI. 3. Develop a validated case definition for pediatric TBI by studying various algorithms. 1.4 Hypotheses Our two hypotheses were: 1. There will be TBI validation studies, but none which are pediatric specific. 2. It will be possible to develop an accurate, sensitive case definition, to identify pediatric TBI using ICD-10 coded administrative health data. 2

15 Chapter Two: BACKGROUND AND LITERATURE REVIEW 2.1 Defining Brain Injury Acquired Brain Injury Acquired brain injury (ABI) can occur at any point during the lifetime, including during fetal development [5]. ABI can result from many different causes, including abnormal fetal development, metabolic disorders, systemic illness, central nervous system (CNS) tumours and/or infections, toxins (such as use of alcohol during pregnancy), treatment (such as radiation therapy or CNS surgery), or trauma [5]. While there are many causes of ABI in children, traumatic brain injury is the most common and is the focus of this thesis [6] Traumatic Brain Injury (TBI) There are many different clinical definitions of traumatic brain injury (TBI) in the literature. However, a commonly used case definition for TBI was developed in 1995 by the Centers for Disease Control (CDC). They define TBI as, a craniocerebral trauma, specifically as an occurrence of injury to the head (arising from blunt or penetrating trauma or from accelerationdeceleration forces), that is associated with any of these occurrences attributable to the injury: decreased level of consciousness, amnesia, other neurologic or neurophysiological abnormalities, skull fracture, diagnosed intracranial lesions, or death [7]. Other examples of TBI definitions in the literature include: A bump, blow or jolt to the head, or a penetrating injury that disrupts the normal function of the brain [1], An alteration in brain function, or other evidence of brain pathology caused by an external force [8], and An alteration in brain function manifested as confusion, altered level of consciousness, seizure, coma, or focal sensory or motor neurologic deficit resulting from blunt or penetrating force to the 3

16 head [9]. In addition, some definitions specify that in mild TBI, subtle behavioural and neuropsychological changes may be the only symptoms [9]. Concussion is a common type of traumatic brain injury. According to the 3 rd International Conference on Concussion in Sport (2008), concussion is defined as a complex pathophysiological process affecting the brain, induced by traumatic biomechanical forces [10]. Similarly, the American Academy of Neurology (AAN) defines concussion as an alteration in mental status caused by trauma that may or may not involve loss of consciousness [11]. The AAN further identifies confusion and amnesia as hallmarks of concussion [11] Mild, Moderate, and Severe Traumatic Brain Injury Traumatic brain injury is further classified in the literature as being mild, moderate or severe. These definitions are widespread, as there are many different criteria for diagnosing TBI severity. However, a common clinical measure of neurological status in TBI is the Glasgow Coma Scale (GCS) [12]. The GCS has three scored components (eye opening, verbal response, and motor response), which assess the function of the brain and spinal cord. The GCS is based on a 15-point numeric score of which 15 indicates normal functioning, and three is the lowest possible score, indicating deep coma or death [12]. It is important to note that mental status, behaviour, and neurological function can be difficult to clinically assess in pediatric patients depending on their age and abilities [13]. A modification of the Glasgow Coma Scale called the Children s Coma Scale (CCS), has been developed and used for children under the age of three. The CCS modifies the verbal response component of the GCS, to take in to consideration responses that could be expected from a pre-verbal or non-verbal child [14]. The Glasgow Coma Scale is shown in Table 1, with the pediatric modifications for the verbal response component. 4

17 Table 1. Glasgow Coma Scale Category Response Score Eye Opening Spontaneously 4 To speech 3 To pain 2 None 1 *Modified for Infants Verbal Response Motor Response Oriented Babbles 5 Confused Irritable 4 Inappropriate words Cries to pain 3 Moans Moans 2 None None 1 Follows commands 6 Localizes to pain 5 Withdraws to pain 4 Abnormal flexion 3 Abnormal extension 2 None 1 Best Score 15 Worst Score 3 One British study looking at reported problems following TBI among children and adolescents, defined the three types as the following: Mild TBI refers to any injury causing unconsciousness for less than 15 minutes and a GCS score after initial resuscitation of 13-15; Moderate TBI refers to any injury causing unconsciousness for more than 15 minutes and a GCS after initial resuscitation of 9-12; and Severe TBI refers to any injury causing unconsciousness for more than 6 hours and a GCS after initial resuscitation of 3-8 [15]. This same definition is common among other studies in the literature (See Table 2). 5

18 Table 2. Classification of TBI Severity based on the Glasgow Coma Scale GCS Score TBI Classification Mild 9-12 Moderate 3-8 Severe Head Injury versus Traumatic Brain Injury Head injury is an outdated and non-specific term, which includes external injuries to the face, scalp, and skull, such as lacerations, contusions, abrasions, and fractures [9]. However, these injuries may or may not be associated with a TBI [9]. 2.2 Epidemiology and Impact of Traumatic Brain Injury Measuring the extent of traumatic brain injury is challenging due to the variation in TBI definitions, and the need to capture population based representative samples. It has been reported that each year in the United States, there are an estimated 1.6 million traumatic brain injuries among all ages, with approximately 1.3 million Americans treated and released from an Emergency Department (ED), 275,000 hospitalizations for non-fatal TBI, and approximately 52,000 who die as a result of their injuries [1]. Out of the 1.6 million TBIs among all ages, 511,257 occur in children 0-14 years of age [1]. Of these, there are approximately 473,947 American children treated and released from an Emergency Department (ED), 35,136 hospitalizations for non-fatal TBI, and approximately 2,174 die as a result of their injuries [1]. In total, the annual combined estimate of 511,257 ED visits, hospitalizations, and deaths makes TBI the leading cause of death and disability in children and young adults in the United States [1]. The prevalence of TBI was also estimated in a population-based birth cohort from Christchurch, New Zealand, for those aged 0-25 years [16]. Using a prospective longitudinal 6

19 study, they followed 1,265 children over a 25 year period. Based on this data, they estimated the overall prevalence of TBI to be approximately 30% [16]. At this time the prevalence of TBI in Canada is unclear [2]. Three Canadian studies have estimated the prevalence of TBI in Canada, however two were targeted at a specialized population, and one was limited to concussion. The first two studies were specific to youth and adults with disabilities, and utilized information from the 1986 to 1987 Canadian Health and Activity Limitation Survey, to estimate the prevalence of TBI in this population. The two studies reported the overall prevalence rate of TBI for adults aged 15 years and older to be 74.3/100,000 (95% CI, ) [17] and 62.3/100,000 (95% CI, ) [18] respectively. The third study utilized the Canadian National Population Health Survey (NPHS) to describe the epidemiology of reported concussion in the general population [19]. They report the annual prevalence of reported concussion to be 110/100,000 (95% CI, ) [19]. The prevalence was highest in children, with patients aged 0-14 years having the highest prevalence at 200/100,000 (95% CI, ), followed by young adults (15-34 years) at 160/100,000 (95% CI, ), and older adults (35 years and older) at 50/100,000 (95% CI, 30-80) [19]. While this information is valuable, it does not give us a clear picture of the overall prevalence of all TBI types combined in the Canadian population. Despite the limited information on the prevalence of TBI in Canada, there have been a handful of population-based studies which have estimated the incidence of TBI (see Table 3). One study from Ontario, Canada, looked at TBI related hospitalizations using data from the Discharge Abstract Database, between April 1992 and March 2002 [20]. While an overall rate was not reported, incidence was reported for each specific age group, and stratified by sex. Overall, it was noted that incidence of TBI related hospitalizations was significantly higher in 7

20 children and young adults (<25 years), as well as the elderly (>76 years) [20]. This study was also consistent with other literature showing that the incidence was higher among males [20]. A similar study by the same authors sought to determine the number of annual hospitalizations and overall episodes of care related to TBI in the province of Ontario from [21]. They found the rate of TBI hospitalizations to be fairly stable across the five year study period, ranging from /100,000 among all ages for males, and /100,000 among all ages for females [21]. In the most recent year (from ), rates of TBI hospitalizations for males were 32.3/100,000 for those aged 0-14 years, and 68.7/100,000 for those aged years. Similarly, the rates for females were 19.2/100,000 for those aged 0-14 years, and 18.5/100,000 for those aged years [21]. While this article provides extensive information on the rate of TBI hospitalizations for all ages across a five year span, only the most recent data from is presented in Table 3. Third, a population-based study was conducted to examine the incidence of heady injury among school children (ages 6-16 years) in Ontario, Canada [22]. Ninety-five percent of Ontario schools participated in this mandatory injury reporting system. This study looked at 11,068 unduplicated head injuries out of a total 67,647 total injuries reported in the year They estimated a head injury rate of 0.81/100 students during the year 2000 [22]. Finally, a study from Calgary, Canada, looked at traumas resulting in severe TBI in adults (>18 years) between April 1, 1999 and March 31, They estimated the annual incidence to be 11.4/100,000 overall, with a mortality rate of 5.1/100,000 per year [23]. Incidence was again highest in males at 17.1/100,000, compared to 5.9/100,000 for females [23]. 8

21 Table 3. Summary of Canadian Population-Based TBI Articles Article Study Year Data Source TBI Type* Age Reporting Measure Overall Incidence / Prevalence Incidence/Prevalence by Groups (per 100,000) Age in Years Year Male Female Colantonio (2009) [20] Colantonio (2010) [21] CIHI Hospital Discharge Abstract Database CIHI Hospital Discharge Abstract Database and National Ambulatory Care Resource System database All All All Ages All Ages Incidence Not Reported Incidence 52.0/100,000 for males in /100,000 for females in

22 Gordon (2006) [19] Willer (2004) [22] Zygun (2005) [23] NPHS All All Ages 2000 School Injury Reporting System Prospective Chart Review and Health Information Services Database Prevalence 110/100, /100, /100, /100, All 6-16 Incidence 0.81/ Severe >18 Incidence 114/100, * TBI type classified as Mild, Moderate, or Severe CIHI = Canadian Institute for Health Information NPHS = Canadian National Population Health Survey 10

23 2.2.1 The Influence of Age and Sex on Traumatic Brain Injury As discussed in the prior section, age and sex greatly influence the incidence and prevalence of TBI. It has been noted that TBI rates are higher among males than females in every age group [1]. In fact, a study looking at 14 US States found that the rate of TBI among men is almost twice that of the rate among women [24]. It is also commonly reported in the literature that people in the youngest and oldest age groups are most likely to sustain a TBI [1, 19-21]. A government report from the United States found that children aged 0-4 years had the highest rate of ED visits for TBI (1,256/100,000), followed by older adolescents, aged 15 to 19 years (757/100,000) [1]. In addition, when looking at all age groups, the highest rate of TBI related hospitalizations and deaths occurred in the 75 years and older age group, with 339/100,000 hospitalizations and 57/100,000 deaths [1]. Similarly, a 2007 report from the Canadian Institute for Health Information (CIHI) reported that children (<18 years) represented almost 45% of the patients with head injury visiting emergency departments and urgent care centers, and almost 25% of the patients hospitalized with head injury [2]. Pediatric TBI is of particular concern as it can lead to significant and lifelong neurologic and cognitive deficits [3] with a substantial and permanent burden on the patient, their caregivers and society. Outcomes of pediatric TBI cover a vast range including emotional, physical, intellectual, social and financial consequences [15] Causes of Traumatic Brain Injury There are many different causes of TBI across all age groups. Falls continue to be the leading cause of TBI in the United States and Canada [1, 25], with the rates being highest for those aged 0-4 years, and those 75 years and older [1]. In one Canadian study, falls were the most common 11

24 cause of TBI for children and youth in , followed closely by motor vehicle incidents, and sports and recreational activities [25]. Events in which a person was struck by a stationary or moving object, and assault are other significant causes of TBI [1, 25]. Inflicted head injury also continues to be a leading cause of death and a significant source of morbidity for children less than one year [26] Costs Associated with Traumatic Brain Injury Traumatic brain injury contributes substantially to the health care system and resource burden. One American study reported that pediatric TBI accounted for more than one billion dollars in hospital charges annually [27]. Although the prevalence of head injury (including TBI) in Canada is currently unknown, the PHAC estimates that the total direct cost of associated with head injury in was $151.7 million, with $150.7 million (99.3%) for hospital care, $0.3 million (0.2%) for physician care, and $0.7 million (0.5%) for drugs [2]. 2.3 Surveillance for Traumatic Brain Injury What is Surveillance? Surveillance is a fundamental role of public health and is used for many purposes, such as to monitor changes in disease frequency, or the prevalence of risk factors [28]. The CDC defines epidemiologic surveillance as, the ongoing systematic collection, analysis, and interpretation of health data essential to the planning, implementation, and evaluation of public health practice closely integrated with the timely dissemination of these data to those who need to know [29]. There are two types of surveillance: passive and active. In passive surveillance, data on reportable diseases or conditions are submitted by health care providers, or by another designated 12

25 professional as per specified reporting guidelines [28]. Passive surveillance is advantageous as it is generally inexpensive, and relatively easy to develop initially [28]. However, because the quality or completeness of the data submitted depends on the individual, there is often underreporting of conditions using passive surveillance. Similarly, the methods for data collection and resources cannot be as extensive as those used for active surveillance [28]. Active surveillance involves an effort to find new cases of disease in the population through different methods such as interviewing health professionals, reviewing medical records, visiting hospitals, and/or investigating areas where an index case has been reported [28]. In active surveillance, reporting is conducted by an individual who is specifically assigned to this responsibility, and therefore is often more accurate and complete [28]. Active surveillance is more expensive than passive surveillance, and is more difficult to set up initially [28] Common Sources of Data for Surveillance There are many different data sources used for disease surveillance. Some common examples include: administrative data, health survey data (such as the Behavioral Risk Factor Surveillance System in the United States [30]), and electronic medical records (such as the Canadian Primary Care Sentinel Surveillance Network [31]). At this time there are no registries or surveillance programs dedicated to TBI for the Canadian population. Although general injury registries exist, such as the National Trauma Registry (NTR) [32], Ontario Trauma Registry (OTR) [32], and Alberta Trauma Registry [33], these focus on more severely injured patients, and are not specific to TBI. Both the NTR and OTR are maintained by CIHI, and provide statistics on national injuries and provincial injuries respectively. Both registries collect their information from hospital administrative databases, and 13

26 collect data on all injuries. Some common data elements include demographic, diagnostic, and procedural data, as well as injury severity and outcomes data [34, 35]. In the NTR and OTR, injuries are grouped in to broad categories such as superficial, orthopedic, burns, head injury, spinal cord injury, internal injury, nerves, blood vessels, and other [36]. Although these registries collect information on head injury, the definition is broad. Similarly, the primary focus of this data is the cause of injury (i.e. How many head injuries occurred from motor vehicle accidents?), and differences between age and sex [36]. To qualify for the Alberta Trauma Registry, a patient must have an Injury Severity Score greater than or equal to 12, be admitted to the trauma centre, or die in the emergency department of the trauma centre [33]. While the Alberta Trauma Registry captures injuries to the head, it groups them in combination with cervical spine injuries. In addition, data on head injuries are primarily collected in regards to helmet use (i.e. How many people sustained a head injury while wearing a helmet?) [37]. A common link among all of these registries is their use of administrative data to capture the population of interest. Despite the many different resources available for epidemiological surveillance, administrative data are the most common and feasible source for surveillance Administrative Data Administrative data refers to information often collected by government for administrative purposes, for example reimbursing doctors and hospitals for services provided, or keeping track of those who are eligible for certain medical benefits [38]. Administrative data were not originally created for research purposes. However, they are valuable data sources for disease surveillance because the data are routinely collected, cover wide geographical areas, and have a 14

27 relatively complete capture of the healthcare system [39]. This is especially applicable to the single-payer Canadian system as there are comprehensive population-based administrative databases at regional, provincial, and national levels [40]. Administrative data are also cost effective, minimize biases, and offer the opportunity to study rare outcomes [41]. There are also limitations to administrative data. For example, the data may not contain important elements that you need to answer the research question, the amount of data can be overwhelming, and finally, data quality and access to data can vary [41]. Some examples of administrative databases used in the province of Alberta are, the Ambulatory Care Classification System (ACCS), the Discharge Abstract Database (DAD), and the physician claim s database. Detailed descriptions of the databases used in this study (ACCS and DAD) are given in sections and International Classification of Disease System use in Administrative Data Diseases are coded in administrative data using the World Health Organization s (WHO) International Classification of Diseases (ICD). ICD codes are used to capture cases of morbidity and mortality, as well as for epidemiological, health management, and reimbursement purposes [42]. Initially, the international classification system was used to code for death only, and in 1893 was known as the International List of Causes of Death (ILCD). Since then, it has been revised many times to include morbidity as well as mortality (see Table 4). Currently, the ICD- 9 th edition and ICD-10 th edition are the most commonly used in North America. The 11 th version of the ICD is now being developed, and is expected to be released in 2015 [43]. Unfortunately, several differences exist among the editions, which can make it hard to make comparisons between morbidity and mortality data collected around the world [44]. Table 5 outlines the differences between ICD-9 and ICD

28 Table 4. List of ICD Versions and Years of Adoption by the World Health Assembly* International Classification Year ILCD 1893 ILCD revision ILCD revision ILCD revision ILCD revision ILCD revision ICD revision ICD revision ICD revision ICD revision ICD revision ICD revision 11 (in preparation) 2015 ILCD = International List of Causes of Death ICD = International Classification of Diseases * Above table adopted with permission from Jetté et al. (2010) [44] 16

29 Table 5. Comparison between ICD-9 and ICD-10 Classifications* ICD-9 ICD-10 Name of Classification International Classification of Diseases International Statistical Classification of Diseases and Related Health Problems Date of Adoption by the World Health Organization Number of Volumes 1. Volume 1 Tabular list 2. Volume 2 Alphabetical index 1. Volume 1 Tabular index 2. Volume 2 Instruction manual 3. Volume 3 Alphabetical index Coding Format Numeric Alpha-Numeric Number of Sections vs. Chapters Supplementary Classifications 17 Sections ( ) 21 Chapters (A00-Z99) Except for U codes U00-U49: Reserved for the provisional assignment of new diseases of unknown causes U50-U99: For research purposes Two supplementary classifications 1. External causes of injury and poisoning (E800-E999) 2. Factors influencing health status and contact with health services (V01-V82) Categories 909 2,036 Subcategories 5,161 12,420 Total Codes 6,882 12,420 *Above table modified with permission from Jetté et al. (2010) [44] No supplementary classifications (prior supplementary classifications are now their own chapters) 1. Chapter XX: External causes of morbidity and mortality (V01-Y98) 2. Chapter XXI: Factors influencing health status and contact with health services International Classification of Diseases, 10 th Edition (ICD-10) The WHO ICD-10 version was released in 1994, after being endorsed by the 43 rd World Health Assembly in 1989 [42]. Although this is the most current version, ICD-10 has not yet been implemented in the United States for morbidity, or used in certain Canadian administrative databases, such as physician claim s databases, where ICD-9 is still in use. Unlike the numeric coding used in ICD-9 th version, ICD-10 uses alphanumeric coding with up to 6 characters, and contains 12,420 codes (compared to 6,882 in ICD-9) [45]. 17

30 2.3.6 International Variations in ICD Coding In addition to differences between the standard ICD versions, there are international variations which allow countries to make changes at the fifth and sixth digit level (see Table 6 for example), in order to better classify conditions that may be of special interest to that country. ICD-10-CA is the enhanced version of ICD-10 developed by CIHI for morbidity classification in Canada, and classifies diseases, injuries, causes of death, external causes of injury, and poisoning [46]. ICD-10-CA also includes codes for situations that represent risk factors to health, such as occupational, environmental, lifestyle, and psycho-social factors [46]. ICD-10-CA was first implemented in British Columbia, Newfoundland, Nova Scotia, Prince Edward Island, the Yukon, and partially in Saskatchewan in 2001, followed by Alberta, Northwest Territories, Nunavut, Ontario, and fully in Saskatchewan in New Brunswick was next to introduce ICD-10-CA in 2003, followed by Manitoba in 2004, and finally by Quebec in 2006 [47]. It is difficult to make comparisons between international versions of an ICD edition, (example: ICD-10-CA used in Canada and ICD-10-CM used in the United States), due to the differences in the interpretations of codes. See Table 6 for examples of coding variations between various international ICD-10 modifications. 18

31 Table 6. Examples of ICD-10 Coding Variations between International Clinical Modifications* Codes ICD-10-CA (Canada) ICD-10-GM (Germany) ICD-10-AM (Australia) S06.20 Diffuse brain injury without loss of consciousness S06.21 Diffuse brain injury with loss of consciousness S06.22 Diffuse brain injury with moderate loss of consciousness S06.23 Diffuse brain injury with prolonged loss of consciousness with return to pre-existing level of consciousness Diffuse cerebral and cerebellar brain injury, unspecified Diffuse brain contusions Diffuse cerebellar contusions Multiple intracerebral and cerebellar hematomas Diffuse cerebral and cerebellar injury, unspecified Diffuse cerebral contusions Diffuse cerebellar contusions Multiple intracerebral and cerebellar hematomas *Above table modified with permission from Jetté et al. (2010) [44] Validated Case Definition for Traumatic Brain Injury Surveillance In order to effectively capture cases of pediatric TBI in administrative data, it is important to develop a validated ICD-based case definition. Although validated case definitions for TBI have been published from the United States, a validated case definition to study TBI using Canadian administrative data is not currently available. It is important to validate ICD coding specifically in Canadian databases for several reasons. While the underlying coding framework is the same internationally, the United States still uses the ICD-9-CM edition (until October 1, 2013) [48] which contains fewer codes than the ICD-10-CA edition currently being used in Canada. In addition, Canada and the United States currently have different models of health care delivery, resulting in potential differences in what the administrative data capture. Finally, unlike the United States and other countries, Canadian coders undergo national standardized training, potentially leading to variations in coding standards between these countries [49]. Fortunately, administrative data have already been used successfully in Canada for surveillance of chronic conditions such as diabetes (Chronic Diseases Surveillance System) [50]. 19

32 As neurological diseases are now under the umbrella of the Canadian Chronic Diseases Surveillance System (CCDSS), there is a need to validate TBI coding to ensure it can be one of the conditions captured in the neurological disease surveillance program. 2.4 Tests of Diagnostic Accuracy When validating a case definition using administrative data, there are many different tests of diagnostic accuracy that can be used. Examples include the kappa statistic, likelihood ratios, sensitivity (Sn), specificity (Sp), positive predictive value (PPV), and negative predictive value (NPV) Identifying Study Cohort Some tests of diagnostic accuracy such as Sn, Sp, PPV, and NPV, require that you first identify those with the disease and those without the disease. Test results can fall in to four different categories within the standard 2x2 table (see Table 7). Two categories are true, and two are false. True positives are those who have the disease in question according to the reference standard, and also have the disease according to the administrative data. True negatives are those who do not have the disease in question as per the reference standard, and who are correctly identified as not having the disease according to the administrative data. False positives are those who have been identified as not having the disease in question according to the reference standard, but who have been incorrectly classified as having the disease according to the administrative data. False negatives are those who have been identified as having the disease in question according to the reference standard, but who have been incorrectly been classified as not having the disease in question according to the administrative data. 20

33 Table 7. Standard 2x2 Table for Calculating Diagnostic Accuracy Administrative Data or Diagnostic Test Reference Standard Positive (+) Negative (-) Positive (+) True Positive False Positive Negative (-) False Negative True Negative The four most common measures of the validity of a test are sensitivity, specificity, positive predictive value and negative predictive value. These four measures were the focus of this study, and will be described in detail in the following section Sensitivity Sensitivity (Sn) is a measure of the validity of a test. It is defined as the ability of a test to correctly identify those who have the disease, or the proportion of actual positives which are correctly identified as being positive. Sensitivity was calculated as: A high sensitivity indicates that the administrative data captures the cases well, and is likely capturing a true picture of the occurrence of a disease or condition. A low sensitivity in the administrative data is concerning as this indicates that the administrative data is likely underestimating the number of patients who have the disease or condition. 21

34 2.4.3 Specificity Specificity (Sp) is defined as the ability of a test to correctly identify those who do not have the disease, or the proportion of true negatives which are correctly identified as being negative. Specificity was calculated as: Predictive Values Positive predictive value (PPV) and negative predictive value (NPV) ask: if the test results are positive or negative in this patient, what is the probability that this patient has or does not have the disease? PPV and NPV were calculated as: Relationship with Prevalence The predictive value of a test is determined not only by the sensitivity and specificity, but by the prevalence of the disease in the population of interest [51]. The more sensitive a test is, the better the NPV (the more confident you can be that a negative test result rules out the disease of 22

35 interest) [51]. Alternatively, the more specific a test is, the better the PPV (the more confident you can be that a positive test rules in the diagnosis of interest) [51]. Prevalence is more important than sensitivity and specificity in determining predictive value, because prevalence can vary over a wider range than sensitivity and specificity can [51]. To highlight this concept, please refer to Table 8, which has been reproduced from Akobeng s (2007) article on understanding diagnostic tests [52]. Table 8. Relationship between Predictive Values and Disease Prevalence* [52] Prevalence (%) PPV (%) NPV (%) *Assuming a test with 80% sensitivity, and 94% specificity The table above demonstrates that higher disease prevalence is associated with a higher PPV, and a lower NPV, whereas lower disease prevalence is associated with a higher NPV and a lower PPV. Predictive value is also affected by the specificity of the test being used when the disease is infrequent [28]. This is important because generally speaking, the prevalence of the majority of diseases we deal with are much lower than 50% [28]. For the following explanation, please refer to Table 9 below. In the case of an infrequent disease, the majority of people will fall in to the negative category on the top right of the 2x2 table. Thus, any change to the right hand side of this table will affect a larger number of people than a change of the same type to the left hand 23

36 side. Therefore, a change in specificity has a greater effect on predictive value than a comparable change in sensitivity. Table 9. Standard 2x2 Table: Effect of Specificity on Positive Predictive Value Administrative Data or Diagnostic Test Reference Standard Positive (+) Negative (-) Positive (+) True Positive False Positive PPV Negative (-) False Negative True Negative NPV Sensitivity Specificity Summary TBI is the leading cause of death and disability among children and young adults in both the United States and Canada, contributing substantially to health care system and resource burden. It can potentially lead to many life lifelong deficits with a substantial and permanent burden on the patient, their caregivers and society. At this time, there are no registries or surveillance programs dedicated specifically to TBI for the Canadian population. Although general injury or trauma registries exist, they focus on more severely injured patients, and are not specific to TBI. In order meet priorities set out by the Neurological Health Charities of Canada, and the Public Health Agency of Canada to develop a Canadian surveillance program for TBI, we must first develop a validated case definition to study TBI. The focus of this thesis was to develop a validated case definition to identify children who present with a TBI in hospital and emergency room based settings. 24

37 Chapter Three: METHODS AND STATISTICAL ANALYSIS 3.1 Study Objectives To reiterate, the study objectives were to: 1. Perform a systematic review of studies which have validated TBI coding in administrative data. 2. Assess the validity of ICD-10 coding for TBI. 3. Develop a validated case definition for pediatric TBI by studying various algorithms. 3.2 Systematic Review Study Design We used standard systematic review methodology for this portion of the study Search Strategy We searched Medline (1948 to November 2010), and Embase (1980 to November 2010) for relevant articles on November 26, 2010 and November 29, 2010 respectively. Our search strategy included the following terms: administrative data, hospital discharge data, ICD-9, ICD- 10, medical record, health information, surveillance, physician claims, claims, hospital discharge, coding, codes AND validity, validation, cases definition, algorithm, agreement, accuracy, sensitivity, specificity, positive predictive value, and negative predictive value, combined with the MESH and EMTREE terms and keywords for traumatic brain injury. We limited our search to human studies which were published in the English language. 25

38 3.2.3 Study Selection Two reviewers independently assessed all abstracts for fulfillment of the eligibility criteria. To be eligible for inclusion, articles had to be original studies that validated ICD-9 or ICD-10 codes for TBI by comparing the accuracy of the code(s) with a reference standard. In addition, the study had to report at least one of the following measures of validity: sensitivity, specificity, positive predictive value, or negative predictive value. Articles that validated TBI in specialized populations (ex: TBI in athletes), or in which there were only case definitions using ICD-8 codes were excluded. Full text articles were pulled for all abstracts selected by either reviewer. The selected full text articles were then reviewed by two reviewers for fulfillment of eligibility. Disagreements were resolved by consensus. Reference lists of selected full text articles were also reviewed to ensure that no additional studies were missed in the above search strategy Data Extraction Data were abstracted by two reviewers in duplicate using a standardized data abstraction form (see Appendix A). Validated case definitions were abstracted with the specific ICD codes used, for the case definition from each paper. In addition sensitivity, specificity, positive predictive value, and negative predictive value were recorded. Additional information including study location, sample size, validation database, years of data collection, and the reference standard that was used to validate TBI was also recorded. 26

39 3.2.5 Quality Assessment A quality assessment of each included study was performed by two reviewers in duplicate. A standardized 40-item checklist (see Appendix B) was used [53], and one point was assigned for fulfilling each item on the checklist. Some items were not applicable due to the nature of the study. When it was unclear if a certain item was fulfilled, it was marked uncertain and no points were assigned. Discrepancies were resolved by consensus Statistical Analysis Descriptive statistics were used to describe the results of the systematic review. A metaanalysis was not done due to study heterogeneity. 3.3 Validation of ICD-10 Coding and Identification of Best Case Definition Study Design This was a retrospective cohort study which looked at electronic emergency department records, to identify children with and without traumatic brain injury Study Site Alberta Children s Hospital (ACH) is the primary pediatric hospital for the city of Calgary and surrounding semi-urban and rural areas. ACH also serves patients from southern Alberta, southeastern British Columbia, and southwestern Saskatchewan. ACH is one of the two accredited level one trauma centres in Calgary [33], and has a total of 133 inpatient beds, and an ER with 60,000 visits per year for those aged 0-18 years. The estimated total population of Calgary was 1,079,310 as per the 2006 Canadian census, with 272,275 aged 0-19 years [54]. 27

40 3.3.3 Study Population and Identification of Traumatic Brain Injury Cases This cohort includes 7,101 children aged 0-18 years, with primarily trauma, musculoskeletal, or central nervous system issues, captured between October 5, 2005 and June 6, 2007 at ACH through the Regional Emergency Department Information System (REDIS). These conditions were selected to increase the likelihood of capturing children who may have experienced a TBI. REDIS is an electronic information system used to collect and track information on all children who visit the ACH emergency department (ED). A combination of free text and drop down menus guide the information that is collected in each child s REDIS chart by the ED staff. For a full description of REDIS see section The diagnosis and clinical history fields of the REDIS charts were reviewed by a pediatric neurologist (Dr. K Barlow) with expertise in brain injury, to determine which children had a diagnosis of TBI. TBI was defined as any trauma to the head or body that resulted in any of the following: loss of consciousness, altered mental state, post-traumatic amnesia, neurological symptoms (e.g., headaches, dizziness, vomiting, confusion, behavioural changes, etc.) and/or focal neurological signs. The definition of mild TBI is somewhat controversial in younger children (less than five years old), in whom it can be difficult to assess an altered mental state and period of amnesia. Therefore, any young child presenting to the ED with a mild head injury without clear evidence contradicting the above criteria was included in the study. Children with simple scalp lacerations, facial injuries/fractures, or superficial injuries who did not display neuro-behavioural change were excluded, or if they had any injury other than head injury. Finally, the date the child was first seen in REDIS became the index date for all children. Descriptive statistics were used to describe the age and sex breakdown of the study cohort. 28

41 3.3.4 Additional TBI Diagnosis Validation by Telephone Interviews and Clinic Visits The quality of the REDIS chart review was further verified among a subset of patients (n = 715) who either had a telephone follow-up call to confirm TBI diagnosis (n = 670), or who visited the ACH TBI clinic (n = 45). The percentage of children who had a phone call or were seen in the clinic, and had a diagnosis of TBI confirmed was calculated. 3.4 Data Sources Used Regional Emergency Department Information System (REDIS) REDIS is an electronic information system used to collect and track information on all patients who visit the emergency department (ED). REDIS was designed specifically for the former Calgary Health Region, and is based on the existing Canadian Emergency Department Information System developed by the Canadian Association of Emergency Physicians (CAEP), the National Emergency Nurses Affiliation, and l Association des Médecins d Urgence du Québec [55]. REDIS is also used at the adult hospitals in Calgary, although the programs differ slightly to suit the needs of the patient population. Every patient who visits the ED initially gets entered in to REDIS by a trained ED staff member (ex: triage nurse, paramedic, respiratory therapist, nursing student). After a thorough initial assessment and consideration of the child s presenting symptoms, the staff member enters the best fitting complaint code from a drop down menu of 15 choices (see Table 10), and enters further details on the child s presenting complaint in to the adjacent free text cell. These 15 diagnostic choices were selected from the CAEP website, which has a total of 165 categories [56]. The only situation in which a child would not be entered in to REDIS would be in the event of a mass casualty emergency, or during software downtime. However, during the rare occasion 29

42 of downtime, paper copies of the REDIS system are used, and are consolidated once the program is back up. Once the patient has been seen by the ED physician, it is their responsibility to enter the child s official final diagnosis in their designated cell of the electronic database, regardless of whether the child is being sent home or admitted to hospital. Ideally, physicians are to use ICD codes when entering their diagnosis, however this is not standardized, and often just a free text diagnosis is entered. When the visit is complete, the child s REDIS print out gets filed with their health record. Additionally, REDIS stores all data entered indefinitely. At this time, there have not been any studies done to validate REDIS. REDIS is in the process of being replaced by a new program called Sunrise Emergency Care which will begin implementation in September (All information regarding REDIS from personal communication with Charlyn Stanley, Clinical Nurse Educator, ACH Emergency Department, November 15, 2011). 30

43 Table 10 Alberta Children s Hospital REDIS Complaint Codes List Complaint Code Definition 1. CNS Central Nervous System 2. CVS Cardiovascular System 3. EENT Eyes, Ears, Nose, Throat 4. Endocrine Endocrine 5. Fever Fever 6. Follow-Up Follow-up 7. GI Gastrointestinal 8. GU Genitourinary 9. MSK Musculoskeletal 10. Other Other 11. Psych Psychiatry/Psychology 12. RESP Respiratory 13. Skin Skin 14. Toxin Toxin 15. Trauma Trauma Ambulatory Care Classification System The Ambulatory Care Classification System (ACCS) captures information on those who are accessing facility based ambulatory care programs, such as emergency room visits, same-day surgery, day procedures, and community rehabilitation services occurring in publicly funded facilities in the province of Alberta [57]. This database contains information on the patient, provider, service, diagnosis, and procedure or intervention. The ACCS contains up to 15 coding fields for diagnoses and 5 for procedures using ICD-10-CA codes. In ACCS, the primary diagnosis field contains the diagnosis responsible for most resource use during admission [44]. The rest of the diagnosis codes are classified as secondary diagnoses. As of April 2010, Alberta began reporting all of its emergency department, day surgery, and ambulatory care clinic records to the Canadian Institute for Health Information s (CIHI) National Ambulatory Care Reporting System (NACRS) [58]. NACRS is replacing Alberta s ACCS database. 31

44 3.4.3 Discharge Abstract Database The hospital Discharge Abstract Database (DAD) is managed by CIHI, and captures all of the discharges from acute inpatient facilities (including discharges for chronic care, rehabilitation, and day surgery facilities), in all provinces and territories, except Quebec [59]. Each province/territory is responsible for reporting data directly from their facilities or their health/regional authority to CIHI [59]. The DAD contains information on the patient, provider, service, diagnosis, and procedure or intervention for people who have been discharged from an inpatient bed [57]. Since April 1, 2002, the DAD contains up to 50 coding fields for diagnoses and 20 for procedures. However, only 25 coding fields are released to researchers (since it is extremely rare for a patient to have more than 25 coding fields filled out). Prior to April 1, 2002, diagnoses were coded in the DAD using ICD-9-CM. The DAD is updated annually [57]. Similar to ACCS, the primary diagnosis field in the DAD contains the diagnosis responsible for most resource use during admission, and subsequent codes are classified as secondary diagnoses [44]. 3.5 Administrative Data Linkage The REDIS cohort was linked to two Alberta specific administrative databases (ACCS and DAD) from April 1, 2002 to March 30, 2011, by data analysts at the Data Integration Measurement and Reporting group from Alberta Health Services. Patients were linked based on regional health record numbers, first name, last name, and date of birth. The data was then deidentified and scrambled, with new randomly assigned unique numerical patient identifiers. 32

45 3.6 ICD-10 and ICD-10-CA Codes for Traumatic Brain Injury Table 11 shows the ICD-10 codes that were used to identify TBI cases in the administrative databases (see Appendix C for detailed version of ICD-10 and ICD-10-CA code list). ICD-10 codes to be tested for TBI were selected based on the following: 1) A systematic review of ICD codes to study neurological conditions [23], 2) A review of the grey literature and relevant websites (including TBI codes used by the CDC for their TBI surveillance program [1], and codes used by the Canadian National Trauma Registry [34]), 3) Expert consensus (primarily neurologists, epidemiologists, health services researchers, population health experts) from the Canadian Chronic Disease Surveillance System Neurological Conditions Working Group (unpublished) based on the above literature reviews and expert opinion. A diagnosis of TBI was considered if one of the specified ICD-10 codes was listed in any of the available diagnostic coding fields in either of the linked administrative databases (ACCS or DAD). 33

46 Table 11 ICD-10 Codes used to Identify Traumatic Brain Injury ICD-10 Code Definition S02.0 Fracture of vault of skull S02.1 Fracture of base of skull S02.3 Fracture of orbital floor S02.7 Multiple fractures involving skull and facial bone S02.8 Fractures of other skull and facial bones S02.9 Fracture of skull and facial bones, part unspecified S06 Intracranial injury S06.0 Concussion S06.1 Traumatic cerebral edema S06.2 Diffuse brain injury S06.3 Focal brain injury S06.4 Epidural hemorrhage S06.5 Traumatic subdural hemorrhage S06.6 Traumatic subarachnoid hemorrhage S06.8 Other intracranial injuries S06.9 Intracranial injury, unspecified S07 Crushing injury of the head S07.0 Crushing injury of the face S07.1 Crushing injury of the skull S07.8 Crushing injury of other parts of the head S07.9 Crushing injury of the head, part unspecified S09.9 Unspecified injury of head, including injury of ear NOS, face, NOS, and nose NOS T06.0 Injuries of brain and central nerves with injuries of nerves and spinal cord at neck level T90.5 Sequelae of intracranial injury 3.7 Algorithms Used to Identify Best Case Definition Figure 1 shows examples of the different algorithms tested over the study period. The nine unique algorithms were studied to see which produced the best case definition for children with TBI, comparing the ACH reference data against the administrative data (ACCS and DAD) (see Table 12). Tests of diagnostic accuracy (sensitivity, specificity, PPV, and NPV) were calculated for each of the different algorithms. Sensitivity refers to the proportion of children identified as having TBI according to the administrative data. Specificity refers to the proportion of children 34

47 identified as not having TBI according to the administrative data. The PPV refers to the proportion of TBI cases identified in the administrative data that were deemed true TBI cases on the basis of REDIS chart review data (reference standard). The NPV refers to the proportion of non-tbi cases identified in the administrative data that were deemed true non-tbi cases on the basis of REDIS chart review data (reference standard). 35

48 Figure 1 Timeline ER Visit* Hospital Stay * = Also REDIS Index Date 1 Year Example Algorithm: 1 Hospital or 1 ER in 1 year 36

49 Table 12. Algorithms used to Test for Best Case Definition for Pediatric TBI # Algorithm 1. 1 Hospital or 1 ER in 1 year 2. 1 Hospital or 1 ER in 2 years 3. 1 Hospital in 1 year 4. 1 Hospital in 2 years 5. 1 ER in 1 year 6. 1 ER in 2 years 7. 1 Hospital or 2 ER in 1 year 8. 1 Hospital or 2 ER in 2 years 9. Any TBI Diagnosis 3.8 Ethics Approval Ethics approval was received from the Conjoint Health Research Ethics Board at the University of Calgary. 37

50 Chapter 4: RESULTS 4.1 Systematic Review From the systematic search, 284 unique abstracts were screened, 19 full text articles were reviewed, and three full text articles met all eligibility criteria [60-62]. See Figure 2. Table 13 lists excluded articles and reasons for their exclusion. Figure 2 Flow-Diagram of Systematic Review Findings 284 Abstracts Screened 19 Full Text Articles Reviewed 3 Articles Met Inclusion Criteria 0 Articles added after Hand Searching and Expert Consultation 3 Articles Included in Final Systematic Review 38

51 Table 13 List of Excluded Articles and Reason for Exclusion* Author Date Reason for Exclusion Boake [63] 2004 No reference standard, compare two databases Cuff [64] 2007 Does not validate an ICD database Deb [65] 1999 Does not validate an ICD database Guerrero [66] 2000 Does not validate an ICD database Hill [67] 2009 Does not validate an ICD database Kashluba [68] 2006 Does not validate an ICD database Kim [69] 2000 Does not validate an ICD database Kim [70] 2009 Does not validate an ICD database Labelle [71] 2002 Does not validate an ICD database McNaughton [72] 2000 Not an ICD validation Perel [73] 2008 Does not validate an ICD database Powell [74] 2008 No ICD codes Ryu [75] 2009 Does not validate an ICD database Wirtz [76] 2008 Surveillance, No sensitivity, specificity etc. Xiang [77] 2007 Reviewers assigned ICD codes Zygun [23] 2005 Does not validate an ICD database * Table adopted with permission from St. Germaine-Smith et al. (2012) [78] All three articles included in the final systematic review were from the United States. In addition, all three articles had different reference standards to which they validated either ICD-9 or ICD-10 coding. In total, the three articles tested eight different case definitions (different ICD codes), but none required a specific number of encounters or time period. Two studies evaluated case definitions using ICD-9 coding [60, 61]. One looked at statewide inpatient and trauma registry databases from Maryland, and found that case definitions tested in the inpatient database alone, and combined with the registry had slightly higher PPV than the registry database alone [61]. Less specific codes (ex: ICD which includes some skull fractures without mention of intracranial injury) were associated with lower PPV. These authors conclude that surveillance using ICD-9-CM codes seem to under-estimate the incidence of mild TBI. 39

52 The second study compared coding of an inpatient database to real-time assessment in the emergency department, to evaluate the accuracy of mild TBI case ascertainment [60]. They found low sensitivity (45.9%) and PPV (23.9%), but high specificity (97.8%) and NPV (99.2%) [60]. This was the only study to report more than one test of diagnostic accuracy. These authors conclude that TBI surveillance based on hospital discharge data may underestimate the incidence of TBI related hospitalizations. The third study sought to describe the accuracy of death certificate surveillance for TBI mortality using ICD-10 codes. Data was obtained from the Oklahoma vital statistics database, and the Oklahoma injury surveillance system database [62]. They found a sensitivity of 78%, and a PPV of 98%. The authors state that if the results of their studies are consistent across other states, they are likely underestimating the burden of TBI related deaths. We applied a reporting guidelines checklist developed by Benchimol et al. (2011) [53] to assess the quality of these validation studies. Although the three articles did not meet all of the criteria for a strong validation study outlined by this checklist, they met several key criteria such as describing the methods of calculating/comparing tests of diagnostic accuracy, and describing the validation cohort. Table 10 shows a complete explanation of the criteria met or not met by all three articles. Table 14 summarizes the results from the three studies, and Table 15 displays the outcome of the quality assessment for the three included articles. 40

53 Table 14 Summary of Sensitivity, Specificity, Positive and Negative Predictive Value Reported by TBI Validation Studies Author, Year, Location Bazarian, 2006, USA [60] Rodriguez, 2006, USA [62] Shore, 2005, USA [61] Data Year Database Validated* 2003 Single Hospital; Inpatient, ER visits 2002 Populationbased; Death Certificates 1999 Populationbased; MHD Populationbased; MHD Populationbased; MHD Study Size ICD Version ICD Codes 516 ICD-9-CM 800.0, 800.5, 801.0, 801.5, 803.0, 803.5, 804.0, 804.5, 850.0, 850.5, 850.9, 854.0, ICD-9-CM ICD , , , , , , S01.0- S01.9, S02.0, S02.1, S02.3, S02.7-S02.9, S04.0, S06.0- S06.9, S07.0, S07.1, S07.8, S07.9, T01.0, T02.0, T04.0, T06.0, T90.1, T90.2, T90.4, T90.5, T90.8, T ICD-9-CM , ICD-9-CM , ICD-9-CM , Reference Standard ER real time clinical assessment Medical Examiner Report or Medical Chart Medical Chart Medical Chart Medical Chart Sn Sp PPV NPV

54 Populationbased; MHD Populationbased; MTR Populationbased; MTR 698 ICD-9-CM , ICD-9-CM , ICD-9-CM , Medical Chart Medical Chart Medical Chart *Population-based refers to whether the administrative database being validated is population-based even though the population being sampled may include a subgroup of patients (e.g. population based Veterans database) CM = clinical modification, ER = emergency room, ICD = International Classification of Disease, MHD = Maryland Hospital Database, MTR = Maryland Trauma Registry, NPV = negative predictive value, PPV = positive predictive value, Sn = sensitivity, Sp = specificity, TBI = traumatic brain injury 42

55 Table 15 - Quality Assessment as Outlined by Benchimol et al. (2011) [53] Question Bazarian et al [60] Shore et al [61] 1. Identifies article as study of assessing diagnostic accuracy? Identifies article as study of administrative data? States disease identification and validation as one of the goals of study? Describes validation cohort? (cohort of patients to which reference standard was applied) a. Age? b. Disease? c. Severity? d. Location/ jurisdiction? Describes recruitment procedure of validation cohort? a. Inclusion criteria? b. Exclusion criteria? Describes patient sampling? (random, consecutive, all, etc.) Describes data collection? a. Who identified patients and ensured selection adhered to patient recruitment criteria? b. Who collected data? c. A priori data collection form? 1 U U 7d. How was disease classified? Was there a split sample (i.e. revalidation using a separate cohort) Describe number, training and expertise of persons reading reference standard? If >1 person reading reference standard, is kappa quoted? Were the readers of the reference (validation) test blinded to the results of the classification by administrative data for that patient/ (e.g. Was the reviewer of the charts blinded to how the chart was billed?) Describes methods of calculating/comparing diagnostic accuracy? Report when study done, start/end dates of enrollment Describe number of people who satisfied inclusion/exclusion criteria? Study flow diagram? Reports distribution of disease severity? Report cross-tabulation of index tests by results of reference standard Reports at least 4 estimates of diagnostic Rodriguez et al [62] 43

56 accuracy? (estimates reported in included studies) 18a. Sensitivity b. Specificity c. PPV d. NPV e. Likelihood ratios f. Kappa g. Area under the ROC curve/c-statistic h. Accuracy/ agreement i. Other Was the accuracy reported for any subgroup (e.g. age, geography, different sexes, and so on) If PPV/NPV reported, does ratio of cases/controls of validation cohort approximate prevalence of condition in the population? Reports 95 CIs for each of above? Discussion 22. Discusses the applicability of the findings? = yes, 0 = no, U = uncertain, NA = not applicable 4.2 Development of a Validated Case Definition for Pediatric Traumatic Brain Injury Characteristics of the Study Population Of the 7,101 children identified through REDIS, 1,422 (20%) were determined to have TBI, and 5,679 served as non-tbi cases. After administrative data linkage, 93% (6,639/7,101) of the original study population was available for analysis (successfully linked), leaving 1,343 with TBI and 5,296 without TBI (see Figure 3). Based on the available linked data, the study cohort was 56% male (3,372/6,639), with a mean age of 9.2 years (range 2.02 days 18.3 years) for males, and 8.5 years (range 2.61 days 18.6 years) for females. 44

57 Figure 3 Flow-Diagram of Study Population 7,101 Children with and without TBI identified from REDIS Original REDIS Data REDIS Chart Review 5,679 Children without TBI 1,422 Children with TBI Administrative Data Linkage 462 Children unable to link with administrative data 6,639 Children with and without TBI available after data linkage 93% of original REDIS data available for use after data linkage 5,296 Children without TBI 1,343 Children with TBI 45

58 4.2.2 Additional TBI Diagnosis Validation by Telephone Interviews and Clinic Visits Of the 715 children that were successfully contacted by telephone for follow-up by the ACH TBI Clinic, only 27 children (4%) were determined to not be TBI cases upon further questioning. All of the 45 children seen at the ACH TBI Clinic were true cases of TBI as originally determined by the REDIS chart review (100%) Validation of ICD-10 Coding The most frequently used ICD-10 code for TBI cases in our cohort was S09.9 unspecified injury of head, including injury of ear not otherwise specified (NOS), face NOS, and nose NOS (50.5%), followed by S concussion without loss of consciousness without open intracranial wound (22.7%) (see Table 16). Similarly, 88.9% of the time, the codes used for TBI were found in the primary diagnosis position of the administrative data. Table 16. Frequency of ICD-10 and ICD-10-CA Codes used in Administrative Data to Identify TBI Diagnostic Code Frequency Percent Cumulative Frequency Cumulative Percent S S S S S S S S S S S06.000* S S S S S

59 S S S S S S S S S S S S S S S S S S S S S S S S S S S S S S S S09.9* * Most frequently used codes to identify TBI in the Administrative Data 47

60 4.2.4 ICD-10 Codes used for False-Negative TBI Cases We investigated which ICD-10 codes were used to identify the false negative cases (n = 406) in our best case definition, to see if we were missing important codes used to capture children with TBI. The majority of false negative cases were classified with non-specific codes identifying superficial injuries, or general trauma, which with other information in the REDIS chart may have appeared to be a possible TBI, however was not coded as such in the administrative data Identification of Best Algorithm to Identify TBI Cases Of the nine algorithms assessed (Table 12), the coding algorithm with the best diagnostic accuracy to capture cases of pediatric TBI for future surveillance work was: 1 ER or 1 Hospital in 1 year, with a sensitivity of 69.8% (95% CI, ), specificity of 96.7% (95% CI, ), PPV of 84.2% (95% CI, %), and NPV of 97.2% (95% CI, ) (Table 17). 48

61 Table 17. TBI Validation Results for Algorithms including Hospital or ER visits Algorithm TP FN FP TN Sensitivity % (95% CI) Specificity % (95% CI) PPV % (95% CI) NPV% (95% CI) TBI Case Definition 1 Hospital or 1 ER in 1 year* ( ) 96.7 ( ) 84.2 ( ) 92.7 ( ) 1 Hospital or 1 ER in 2 years ( ) 95.0 ( ) 78.2 ( ) 92.8 ( ) 1 Hospital in 1 year ( ) 99.8 ( ) 84.7 ( ) 80.3 ( ) 1 Hospital in 2 years ( ) 99.4 ( ) 64.0 ( ) 80.4 ( ) 1 ER in 1 year ( ) 96.7 ( ) 84.3 ( ) 92.6 ( ) 1 ER in 2 years ( ) 95.0 ( ) 78.2 ( ) 92.7 ( ) 1 Hospital or 2 ER in 1 year ( ) 99.7 ( ) 90.1 ( ) 81.5 ( ) 1 Hospital or 2 ER in 2 years ( ) 99.3 ( ) 82.1 ( ) 81.7 ( ) Any TBI Diagnosis ( ) 88.0 ( ) 60.6 ( ) 92.8 ( ) *Algorithm best capturing children with TBI in administrative data TBI Traumatic Brain Injury ER Emergency Room TP True Positive FN False Negative FP False Positive TN True Negative PPV Positive Predictive Value NPV Negative Predictive Value CI Confidence Interval Note: For each algorithm, at least one DAD or ED record had an ICD-10-CA diagnosis code for TBI 49

62 CHAPTER 5: DISCUSSION 5.1 Key Findings This project sought to identify studies which have validated TBI coding in administrative data through a systematic review, and investigated whether two Alberta specific administrative databases (primarily hospital and ER databases) could be used to develop a validated case definition for pediatric TBI. Our two main findings were: 1. There are limited validation studies available for TBI, and none are pediatric specific or based on Canadian data. 2. The algorithm 1 ER or 1 Hospital in 1 year should be considered for use in future pediatric TBI surveillance research for those who visit the ER, are seen in a hospital based clinic setting, or who are hospitalized. 5.2 Discussion of Systematic Review Findings In our systematic review looking at validated case definitions for traumatic brain injury [78], only three studies met all eligibility criteria. All three studies were from the United States, and none were pediatric specific. This suggests there is a gap in existing knowledge, and that further validation studies are needed. The three studies differed in their use of administrative data for validation, and in their choice of diagnostic accuracy measures to report. Similarly, two studies validated ICD-9 coding in hospital data [60, 61], and one study validated ICD-10 coding in death certificate data [62]. Because of this heterogeneity, we were unable to perform a meta-analysis. Sensitivity was reported among different tests of diagnostic accuracy in two studies. Similar to our study, sensitivity was reported to be lower than optimal at 45.9% for the study comparing coding of an inpatient database to real-time assessment in the emergency department, and 78% 50

63 for the study utilizing death certificate data. The reference standard to which these algorithms were assessed may have had an impact on the quality of the results. For example, Rodriguez et al (2006) report that death certificate data can have issues with accuracy [62]. They report on existing literature that accuracy rates among physicians completing death certificate records have been reported to range from 23% to 68% [79-82]. If death certificates are not being completed correctly, or are missing codes, this may have an impact on the ability of an algorithm to pick up on a diagnosis, thereby lowering the sensitivity. These authors state that educational efforts to improve death certificate completion could help increase the accuracy of mortality statistics [62]. In 2011, Benchimol and colleagues developed a 40-point data collection tool for the evaluation of studies that validated algorithms using administrative data [53]. We applied this quality assessment tool to the three articles included from our systematic review. While these three articles met a large portion of the reporting criteria, there were still some gaps. For example, none of the studies described any exclusion criteria, and only one study reported at least four measures of diagnostic accuracy [60], a criteria recommended by the Benchimol et al group. Similarly, details of the administrative data collection were varied. For example, only two studies described patient sampling [60, 61], data collection [60, 62], who identified patients and ensured that selection adhered to patient recruitment criteria [60, 62], and who collected the data [60, 62]. Lastly, only one study described the expertise of those reading the reference standard [61]. Table 10 shows a complete explanation of the criteria met or not met by all three articles Strengths of Systematic Review The systematic review done for this project has many strengths. First, we used standard systematic review methodology which ensured a rigorous examination of the existing literature, 51

64 and inclusion of scientifically strong studies. Second, disagreements on article inclusion or exclusion were resolved by consensus. This facilitated thorough and critical discussion, and ensured that articles were included or excluded appropriately. In addition, reference lists of the full text articles were hand searched by experts in the field of administrative data and/or neurological disorders to ensure that no studies were missed. Study data were also abstracted in duplicate and reviewed to ensure that there were no errors in the data abstraction phase. Lastly, the application of the quality assessment tool [53] enhanced the quality of the systematic review by allowing us to evaluate the quality of the studies available Limitations of Systematic Review Our systematic review has some limitations. First, the literature only included studies published in English, and we did not search for validation studies within the grey literature. Therefore, we may be missing validation studies which meet these criteria. Second, it is possible that publication bias may have affected our results if authors reported only case definitions with the highest accuracy instead of all algorithms tested. Alternatively, it is possible that authors may have failed to report their validation findings all together if the results yielded poor sensitivity, specificity, positive and/or negative predictive value. However, in the three studies we identified the reported accuracy measures (specifically specificity and positive predictive value) were low. Thus, we feel that publication bias is unlikely to be a major concern. 52

65 5.3 Discussion of the Validation of a Case Definition for Pediatric Traumatic Brain Injury This study investigated whether two Alberta specific administrative databases (primarily ER and hospital databases) could be used to develop a validated case definition for pediatric TBI. Our results show that the administrative data were able to capture the majority of those who were diagnosed with TBI, with a sensitivity of 69.8%, and a PPV of 84.2% using the algorithm 1 Hospital or 1 ER visit in 1 year. Currently, there is not a single agreed upon case definition for TBI. Some experts recommend excluding ICD-10 code S09.9 (unspecified injury of head, including injury of ear NOS, face NOS, and nose NOS) from TBI case definitions, as it has been shown to inflate incidence and prevalence figures (personal communication with N. Jetté, member of PHAC CCDSS working group). We chose to include this code in our study, as it was the most frequently used (50.5%) to identify TBI in our pediatric cohort. Had we excluded this code, we would have missed 1,145 cases. There is concern that minor brain injury cases may be missed if codes such as S09.9 were excluded, given their ambiguous nature [83]. It has also been reported that up to 20% of S09.9 diagnoses may be cases of mild TBI [83]. Although S09.9 was used most frequently, it is important not to ignore the other ICD-10 codes included. Our recommended case definition, 1 Hospital or 1 ER visit over 1 year was selected as it is important to prioritize an algorithm s sensitivity over other measures, when the goal is to identify all persons with a certain characteristic (such as TBI) in a population [84]. This is relevant to our study as the ultimate goal is to use the best case definition for disease surveillance. A low sensitivity is of concern, especially for research targeted at health resource allocation, or surveillance, as we would be significantly underestimating the burden of disease. This ICD-10 53

66 coding validation is important since most countries are now using ICD-10 data to code for morbidity and mortality [44]. The other algorithms tested revealed varying results. The algorithm looking at the DAD (hospitalization data) alone missed the majority of TBI cases (sensitivity %), but was still associated with excellent specificity (up to 99.8%) and PPV (up to 84.7%). While this group of hospitalized patients may be good for accurately defining cases for hospital-based outcomes or follow-up studies, this is a poor choice for surveillance. Using the hospitalization data alone may also introduce selection bias, as only the more severe cases tend to be admitted. The algorithm identifying those from the ER alone may also be good for surveillance, but is likely missing the more severe cases. Thus, it is important to use an algorithm that captures children from both databases. Incorporating both of these databases for national surveillance of pediatric TBI is feasible in Canada as both hospitalization and ER visits are collected prospectively by provincial governments. It would be extremely time consuming and prohibitively expensive for a clinical network to develop their own active pediatric TBI surveillance program. Administrative data are more feasible, and readily available. In the future, electronic medical/health records (EMR or EHR) may become another source for TBI surveillance, but alternative case definitions incorporating both free text and ICD codes will then need to be developed. Numerous EMR and EHR vendors exist, rendering this option more challenging currently. While there is no cutoff or standard criteria which specify an optimal sensitivity, specificity, PPV, or NPV for case definitions, it is important to balance all four measures. The validity of our case definition for pediatric TBI is comparable to other conditions. A systematic review looking at validated case definitions for neurological conditions found low sensitivity for Alzheimer s disease and Dementia, Parkinson s disease, and spinal cord injury [78]. Similarly, two separate 54

67 studies validating ICD-9 codes for depression found low sensitivity ranging from 18.1% to 34.0% [85, 86]. Considering the results of validation studies for different neurological and mental health conditions, ours has high validity and should be considered for use in pediatric TBI surveillance. There are several possible explanations for the lower than optimal sensitivity found in our study. First, TBI may be more difficult to code than conditions which have a gold standard to confirm the diagnosis (e.g. Creatinine for renal failure). Similarly, unlike some conditions which have one specific code (e.g. multiple sclerosis: ICD-9: 340 and ICD-10: G35), TBI has many possible codes to choose from. Because of the lack of validated diagnostic criteria, and the wide range of codes for TBI, these diagnoses may have been missed in the administrative data. Similarly, TBI can sometimes accompany a complicated poly-trauma. Thus, it is possible that the diagnosis of TBI, especially if milder, was not coded due to the extent of multiple more serious injuries. Unlike chronic diseases (such as epilepsy or multiple sclerosis) which are likely to be coded repeatedly in administrative data over a patient s lifetime, codes for a TBI may or may not make repeat appearances due to the acuity of the condition. At this time, we do not have sufficient information from our data to identify the different types of TBI severity. The administrative data we used likely capture more moderate to severe cases of TBI, which are likely to be coded more than once in the administrative data, due to follow up visits. Lastly, the prevalence of TBI found in our selected study cohort (20%), is lower than the overall estimated 30% prevalence of TBI found over a 25 year period in a population-based birth cohort from Christchurch, New Zealand, for those aged 0-25 years [16]. However, the prevalence of TBI in our cohort is likely overestimated due to the fact that we only captured 55

68 children over a two year period. It is highly plausible that if we followed this study cohort over an additional 20 years, the prevalence would be even higher than the 30% prevalence reported from New Zealand Study Strengths Our study has a number of strengths. First, the administrative data sources we used (ACCS and DAD) are population based and capture the majority of children who access a hospital service (such as the emergency department, an inpatient unit, or a hospital based clinic). Thus, it is likely that we are getting a representative sample of children with TBI. Second, the electronic emergency department charts were reviewed by a pediatric neurologist with expertise in brain injury, which increased the likelihood that true cases of TBI were being captured as cases. In addition, we were able to further validate our cohort by looking at the number of patients with TBI who were contact by telephone or seen in the ACH TBI clinic to confirm diagnosis. This extra validation assured us that children were being accurately classified as TBI cases or not based on the REDIS chart review Study Limitations There are several limitations to this study. First, our reference standard was based only on a pediatric emergency department electronic chart review. It is a concern that misdiagnosis or incomplete documentation of clinical data in a more limited electronic record may affect results. However, when children are admitted to the emergency room, they often receive a comprehensive work-up. In addition, these emergency department electronic charts were reviewed by a pediatric neurologist with expertise in TBI, and therefore we feel that this 56

69 reference standard is most likely reflecting the true diagnosis. We also further confirmed the diagnosis by examining the percentage of children who were not deemed to have TBI upon telephone follow up (4%), and who were not deemed to have TBI upon a clinic visit (0%). Misclassification bias, which occurs when subjects are incorrectly classified, is a common concern with administrative data. However, one of the primary purposes of a validation study is to examine the validity of these codes, and determine how to best identify children with TBI, so that we are capturing the most accurate picture. Second, we were not able to obtain physician claims data which capture all visits, whether hospital based or community based. Therefore, it is highly likely that we are missing a portion of children with mild TBI who were seen in the community by a physician. As a result, our current validated TBI case definition is not comprehensive. Developing a population-based pediatric TBI surveillance program would require validation and incorporation of physician claims data in the case definition. However, the emergency database we used (ACCS) captures both emergency visits and hospital-based clinic visits; therefore, we are likely to have captured at least the more symptomatic mild TBI cases. In addition, we were not able to assess the severity of TBI cases in our cohort. Further research evaluating the severity of TBI using the Glasgow Coma Scale, the Abbreviated Injury Scale, or other measures would increase our understanding of which types of TBI are seen most often in the ED, as hospitalizations, and or by a community physician or clinic. Despite our inability to determine TBI severity, we hypothesized that the majority of cases seen in the ED or hospitalized would be the more moderate to severe TBI cases. Lastly, this cohort was selected from one hospital in a large metropolitan city (Calgary). Therefore, the results may not be generalizable to other regions. In fact, it has been stressed in 57

70 the literature that algorithms validated in one specific area or age group cannot be necessarily applied to other cohorts if you want to ensure accuracy [53]. However, in Canada, professional ICD coders receive standardized national training [49]. Therefore, results across different provincial administrative databases should be similar. Generalizing results internationally would likely be difficult as it has been shown there is coding variation from country to country [87] Potential Impact of Bias The study eliminates some sources of bias by design, however it is still important to consider if bias affected the validity of our study results. Bias is defined as, any trend in the collection, analysis, interpretation, publication, or review of data that can lead to conclusions that are systematically different from the truth [51]. There are many kinds of bias which have the potential to impact research. This section discusses selection bias, which we feel is the most likely source of bias to have potentially affected our results. Selection bias occurs when comparisons are made between groups of patients that differ in ways other than the main factors under study [51]. In this study, selection bias may have occurred due to the fact that we did not have access to physician claim s data, which would have increased the proportion of children captured with TBI who were seen in the community by a family physician or specialist. Since we were unable to obtain access to physician claims data, the children we identified for our cohort with and without TBI were already more likely to have more moderate to severe injuries as they were captured through the emergency department. Similarly, in order to optimize our sample, and increase the likelihood of capturing children who may have experienced a TBI, our cohort included children with primarily trauma, musculoskeletal, or central nervous system issues. This specific selection could have contributed 58

71 to the identification of children who were again more likely to have more moderate to severe injuries. As a result, our case definition may not be good for overall surveillance, but is useful for surveillance of cases of pediatric TBI which access the emergency room or hospital services. 5.4 Public Health Implications Traumatic brain injury is a significant health problem. With advances in research to enhance surveillance for TBI, we can work toward better prevention strategies, and management of outcomes for those who do experience a TBI. The results of our validation study are important to disseminate to key stakeholders in population health surveillance such as the Canadian Institute for Health Information (CIHI) and the Public Health Agency of Canada (PHAC). CIHI is heavily involved in developing and maintaining health information that enables policy, encourages effective health system management, and improves health and health care for Canadians [88]. Similarly, the PHAC plays an important role in promoting and protecting the health of Canadians [89]. Dissemination of our results will also be of interest to the Neurological Health Charities of Canada (NHCC), whose mission is to improve the quality of life for all persons with chronic brain conditions, and their caregivers [90]. It was due to the efforts of the NHCC that the Government of Canada identified neurological diseases as a priority, in the Speech from the Throne in November, 2008 [4]. In collaboration with the PHAC, the NHCC launched the National Population Health Study of Neurological Conditions, with TBI identified as one of the 14 priority conditions [4]. By developing a validated case definition for pediatric TBI using hospitalization and emergency room based administrative data, we have partially met one of the goals of this collaboration. 59

72 The results of our study will also be important to disseminate to other health services researchers, in order to emphasize the need for further validation studies. 5.5 Areas for Future Research Future research should include developing a validated case definition for pediatric TBI incorporating physician claim s data, to ensure that pediatric TBI case ascertainment is population-based, and not simply capturing children who are seen in a hospital based setting. Future studies should also consider examining TBI severity in relation to different algorithms. For example, does the algorithm used to best capture children with severe TBI differ from the algorithm best used to capture children with mild TBI? This study found that the algorithm to best capture children with pediatric TBI in administrative data was, 1 hospital or 1 ER visit in 1 year. Future research is needed to validate this case definition in external databases in other Canadian regions and elsewhere. 5.6 Conclusion Through a systematic review, this study found that there was limited data available on validated case definitions for TBI using administrative data. Secondly, this study found that the best algorithm to capture children with TBI for surveillance purposes utilizing emergency room and hospitalization data was, 1 hospital or 1 ER visit in 1 year. 60

73 Bibliography 1. Faul, M., Xu, L., Wald, M., and Coronado, V., Traumatic Brain Injury in the United States: Emergency Department Visits, Hospitalizations and Deaths , U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, Editor 2010: Atlanta (GA). 2. Canadian Institute for Health Information, The Burden of Neurological Diseases, Disorders and Injuries in Canada, CIHI, Editor 2007: Ottawa. 3. Toga, A.W., Thompson, P.M., and Sowell, E.R., Mapping brain maturation. Trends Neurosci, (3): p Neurological Health Charities of Canada. Implementation committee briefing note: National population study of neurological conditions in Canada. November 10, 2009; Available from: %20National%20Population%20Study%20of%20Neuro%20Conditions%20Nov%2010 %20FINAL%5B1%5D.pdf. 5. Middleton, J., Acquired brain injury. Psychiatry, (7): p Johnson, A.R., DeMatt, E., and Salorio, C.F., Predictors of outcome following acquired brain injury in children. Dev Disabil Res Rev, (2): p Thurman, D.J., Sniezek, J.E., and Johnson, D., Guidelines for Surveillance of Central Nervous System Injury 1995, Atlanta: Centers for Disease Control and Prevention. 8. Menon, D.K., Schwab, K., Wright, D.W., and Maas, A.I., Position statement: Definition of traumatic brain injury. Arch Phys Med Rehabil, (11): p Bruns, J. Jr., and Hauser, W.A., The epidemiology of traumatic brain injury: A review. Epilepsia, Suppl 10: p McCrory, P., Meeuwisse, W., Johnston, K., Dvorak, J., Aubry, M., Molloy, M. et al., Consensus statement on Concussion in Sport 3rd International Conference on Concussion in Sport held in Zurich, November Clin J Sport Med, (3): p American Academy of Neurology, Practice Parameter: The management of concussion in sports (summary statement). Neurology, : p Zuercher, M., Ummenhofer, W., Baltussen, A., and Walder, B., The use of Glasgow Coma Scale in injury assessment: A critical review. Brain Inj, (5): p

74 13. Thiessen, M.L., and Woolridge, D.P., Pediatric minor closed head injury. Pediatr Clin North Am, (1): p Hahn, Y.S., and McLone, D.G., Risk factors in the outcome of children with minor head injury. Pediatr Neurosurg, (3): p Hawley, C.A., Reported problems and their resolution following mild, moderate and severe traumatic brain injury amongst children and adolescents in the UK. Brain Inj, (2): p McKinlay, A., Grace, R.C., Horwood, L.J., Fergusson, D.M., Ridder, E.M., and MacFarlane, M.R., Prevalence of traumatic brain injury among children, adolescents and young adults: prospective evidence from a birth cohort. Brain Inj, (2): p Willer, B., and Moscato, B.S., National prevalence rates of traumatic brain injury among adults: The Canadian household and institutional health and activity limitation surveys, NeuroRehabilitation, : p Moscato, B.S., M. Trevisan, and B.S. Willer, The prevalence of traumatic brain injury and co-occurring disabilities in a national household survey of adults. J Neuropsychiatry Clin Neurosci, (2): p Gordon, K.E., Dooley, J.M., and Wood, E.P., Descriptive epidemiology of concussion. Pediatr Neurol, (5): p Colantonio, A., Croxford, R., Farooq, S., Laporte, A., and Coyte, P.C., Trends in hospitalization associated with traumatic brain injury in a publicly insured population, J Trauma, (1): p Colantonio, A., Saverino, C., Zagorski, B., Swaine, B., Lewko, J, Jaglal S., et al., Hospitalizations and emergency department visits for TBI in Ontario. Can J Neurol Sci, (6): p Willer, B., Dumas, J., Hutson, A., and Leddy, J., A population based investigation of head injuries and symptoms of concussion of children and adolescents in schools. Inj Prev, (3): p Zygun, D.A., Laupland, K.B., Hader, W.J., Koortbeek, J.B., Findlay C., Doig, C.J., et al., Severe traumatic brain injury in a large Canadian health region. Can J Neurol Sci, (1): p Langlois, J.A., Kegler, S.R., Butler, J.A., Gotsch, K.E., Johnson, R.L., Reichard, A.A., et al., Traumatic brain injury-related hospital discharges. Results from a 14-state surveillance system, MMWR Surveill Summ, (4): p

75 25. Canadian Institute for Health Information. Head Injuries in Canada: A Decade of Change ( to ). Analysis in Brief: Taking Health Information Further 2006 [cited 2012 August 3]; Available from: Bishop, N.B., Traumatic brain injury: a primer for primary care physicians. Curr Probl Pediatr Adolesc Health Care, (9): p Schneier, A.J., Shields, B.J., Hostetler, S.G., Xiang, H., and Smith, G.A., Incidence of pediatric traumatic brain injury and associated hospital resource utilization in the United States. Pediatrics, (2): p Gordis, L., Epidemiology. Fourth ed2009, Philadelphia, PA: Saunders Elsevier. 29. Thacker, S.B., and Berkelman, R.L., Public health surveillance in the United States. Epidemiol Rev, : p Centers for Disease Control and Prevention. Behavioral Risk Factor Surveillance System [cited 2012 September 27]; Available from: Canadian Primary Care Sentinel Surveillance Network, Frequently asked questions [cited 2012 September 27]; Available from: Canadian Institute for Health Information. Trauma and Injuries. 2010; Available from: ies/cihi Alberta Health Services. Alberta Trauma Registry. 2012; Available from: Canadian Institute for Health Information. National Trauma Registry. 2012; Available from: tr. 35. Canadian Institute for Health Information. Ontario Trauma Registry. 2012; Available from: vices_otr. 36. Canadian Institute for Health Information. National Trauma Registry 2004 Report: Injury Hospitalizations (Includes data) [cited 2012 September 24]; Available from: 63

76 37. Alberta Centre for Injury Control and Research. Alberta Trauma Registry Annual Report: January 1, 2009 to December 31, ; Available from: /Alberta%20Trauma%20Registry_2009%20Annual%20Report.pdf. 38. Manitoba Centre for Health Policy. Term: Administrative Data [cited 2012 September 4]; Available from: Quan, H., Khan, N., Hemmelgarn, B.R., Tu, K., Chen, G., Campbell, N., et al., Validation of a case definition to define hypertension using administrative data. Hypertension, (6): p Jette, N., Reid, A.Y., Quan, H., Hill, M.D., and Wiebe, S., How accurate is ICD coding for epilepsy? Epilepsia, (1): p Manitoba Centre for Health Policy. Education Resources [cited 2012 September 13]; Available from: World Health Organization. International Classification of Diseases. 2012; Available from: World Health Organization. International Classification of Dieases Information Sheet. 2012; Available from: Jette, N., Quan, H., Hemmelgarn, B., Drosler, S., Maass, C., Moskal, L., et al., The development, evolution, and modifications of ICD-10: Challenges to the international comparability of morbidity data. Med Care, (12): p World Health Organization. FAQ on ICD. 2012; Available from: Canadian Institute for Health Information. ICD-10-CA. 2012; Available from: d+coding/codingclass_icd Canadian Institute for Health Information. ICD-10-CA / CCI Implementation Schedule. 2012; Available from: d+coding/codingclass_icd10status. 48. Centers for Disease Control and Prevention. International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM). 2012; Available from: 64

77 49. Reid, A.Y., St. Germaine-Smith, C., Liu, M., Sadiq, S., Quan, H., Wiebe, S., et al., Development and validation of a case definition for epilepsy for use with administrative health data. Epilepsy Res, Article in Press. 50. Hux, J.E., Ivis, F., Flintoft, V., and Bica, A., Diabetes in Ontario: Determination of prevalence and incidence using a validated administrative data algorithm. Diabetes Care, (3): p Fletcher, R.W., Fletcher, S.W., Clinical Epidemiology: The Essentials. Fourth ed2005, Baltimore, MD: Lippincott Williams & Wilkins. 52. Akobeng, A.K., Understanding diagnostic tests 1: Sensitivity, specificity and predictive values. Acta Paediatr, (3): p Benchimol, E.I., Manuel, D.G., To, T., Griffiths, A.M., Rabeneck, L., and Guttman, A., Development and use of reporting guidelines for assessing the quality of validation studies of health administrative data. J Clin Epidemiol, (8): p Statistics Canada Community Profiles. 2007; Available from: Grafstein, E., Unger, B., Bullard, M., and Innes, G., Canadian Emergency Department Information System (CEDIS) Presenting Complaint List (Version 1.0). CJEM, (1): p Grafstein, E., Bullard, M.J., Warren, D., Unger, B., and Group, C.N.W., Revision of the Canadian Emergency Department Information System (CEDIS) Presenting Complaint List version 1.1. CJEM, (2): p Alberta Health and Wellness, Overview of administrative health databases, Information and Analysis Branch, Editor November 17, 2011, Government of Alberta. 58. Canadian Institute for Health Information. National Ambulatory Care Reporting System [cited 2012 September 18]; Available from: Canadian Institute for Health Information. Discharge Abstract Database (DAD) Metadata: Overview [cited 2012 September 18]; Available from: Bazarian, J.J., Veazie, P., Mookerjee, S., and Lerner, E.B., Accuracy of mild traumatic brain injury case ascertainment using ICD-9 codes. Acad Emerg Med, (1): p

78 61. Shore, A.D., McCarthy, M.L., Serpi, T., and Gertner, M., Validity of administrative data for characterizing traumatic brain injury-related hospitalizations. Brain Inj, (8): p Rodriguez, S.R., Mallonee, S., Archer, P, and Gofton, J., Evaluation of death certificatebased surveillance for traumatic brain injury--oklahoma Public Health Rep, (3): p Boake, C., McCauley, S.R., Levin, H.S., Contant, C.F., Song, J.X., Brown, S.A., et al., Limited agreement between criteria-based diagnoses of postconcussional syndrome. J Neuropsychiatry Clin Neurosci, (4): p Cuff, S., DiRusso, S., Sullivan, T., Risucci, D., Nealon, P., Haider, A., et al., Validation of a relative head injury severity scale for pediatric trauma. J Trauma, (1): p ; discussion Deb, S., ICD-10 codes detect only a proportion of all head injury admissions. Brain Inj, (5): p Guerrero, J.L., Thurman, D.J., and Sniezek, J.E., Emergency department visits associated with traumatic brain injury: United States, Brain Inj, (2): p Hill, J.J., 3rd, Mobo, B.H., Jr., and Cullen, M.R., Separating deployment-related traumatic brain injury and posttraumatic stress disorder in veterans: Preliminary findings from the Veterans Affairs traumatic brain injury screening program. Am J Phys Med Rehabil, (8): p Kashluba, S., Casey, J.E., and Paniak, C., Evaluating the utility of ICD-10 diagnostic criteria for postconcussion syndrome following mild traumatic brain injury. J Int Neuropsychol Soc, (1): p Kim, Y., et al., Validation of the International Classification of Diseases 10th Editionbased Injury Severity Score (ICISS). J Trauma, (2): p Kim, J., Jung, K.Y., Kim, C.Y., Kim, Y.I., and Shin, Y., Development and validation of the Excess Mortality Ratio-adjusted Injury Severity Score Using the International Classification of Diseases 10th Edition. Acad Emerg Med, (5): p Labelle, J., and Swaine, B.R., Reliability associated with the abstraction of data from medical records for inclusion in an information system for persons with a traumatic brain injury. Brain Inj, (8): p

79 72. McNaughton, H., and Wadsworth, K., Assessing the accuracy of hospital admission and discharge diagnosis of traumatic brain injury in a New Zealand hospital. N Z Med J, (1110): p Perel, P., Edwards, P., Shakur, H., and Roberts, I., Use of the Oxford Handicap Scale at hospital discharge to predict Glasgow Outcome Scale at 6 months in patients with traumatic brain injury. BMC Med Res Methodol, : p Powell, J.M., Ferraro, J.V., Dikmen, S.S., Temkin, N.R., and Bell, K.R., Accuracy of mild traumatic brain injury diagnosis. Arch Phys Med Rehabil, (8): p Ryu, W.H., Feinstein, A., Colantonio, A., Streiner, D.L., and Dawson, D.R., Early identification and incidence of mild TBI in Ontario. Can J Neurol Sci, (4): p Wirtz, S.J., and Trent, R.B., Passive surveillance of shaken baby syndrome using hospital inpatient data. Am J Prev Med, (4 Suppl): p. S Xiang, H., Sinclair, S.A., Yu, S., Smith, G.A., and Kelleher, K., Case ascertainment in pediatric traumatic brain injury: challenges in using the NEISS. Brain Inj, (3): p St Germaine-Smith, C., Metcalfe, A., Pringsheim, T., Robers, J.I., Beck, C.A., Hemmelgarn, B.R., et al., Recommendations for optimal ICD codes to study neurologic conditions: A systematic review. Neurology, (10): p Jordan, J.M., and Bass, M.J., Errors in death certificate completion in a teaching hospital. Clin Invest Med, (4): p Lakkireddy, D.R., Gowda, M.S., Murray, C.W., Basarakodu, K.R., and Vacek, J.L., Death certificate completion: How well are physicians trained and are cardiovascular causes overstated? Am J Med, (7): p Messite, J., and Stellman, S.D., Accuracy of death certificate completion: The need for formalized physician training. JAMA, (10): p Myers, K.A., and Farquhar, D.R., Improving the accuracy of death certification. CMAJ, (10): p Chen, A.Y., and Colantonio, A., Defining neurotrauma in administrative data using the International Classification of Diseases Tenth Revision. Emerg Themes Epidemiol, (1): p Chubak, J., Pocobelli, G., and Weiss, N.S., Tradeoffs between accuracy measures for electronic health care data algorithms. J Clin Epidemiol, (3): p e2. 67

80 85. Noyes, K., Liu, H., Lyness, J.M., and Friedman, B., Medicare beneficiaries with depression: comparing diagnoses in claims data with the results of screening. Psychiatr Serv, (10): p Singh, J.A., Accuracy of Veterans Affairs databases for diagnoses of chronic diseases. Prev Chronic Dis, (4): p. A Drosler, S.E., Romano, P.S., Tancredi, D.J., and Klazinga, N.S., International comparability of patient safety indicators in 15 OECD member countries: A methodological approach of adjustment by secondary diagnoses. Health Serv Res, (1 Pt 1): p Canadian Institute for Health Information. Vision and Mandate [cited 2012 September 24]; Available from: Public Health Agency of Canada. About the Agency [cited 2012 September 25]; Available from: Neurological Health Charities Canada. Mission and Vision [cited 2012 September 24]; Available from: 68

81 APPENDIX A: Data Abstraction Form for Systematic Review Condition: Endnote ID: Author, Year: Reviewer: Date: Location (State/Province, Country) Administrative Data Source: Hospital Discharge Data Yes or No (Circle One) Name: (e.g. DAD, ACCS) Emergency Visits Data Yes or No (Circle One) Name: (e.g. DAD, ACCS) Inpatient Claims Data Yes or No (Circle One) Name: (e.g. DAD, ACCS) Outpatient Claims Data Yes or No (Circle One) Name: (e.g. DAD, ACCS) Laboratory Data Yes or No (Circle One) Name: (e.g. DAD, ACCS) Other (specify: ) Yes or No (Circle One) Name: (e.g. DAD, ACCS) 69

82 Validation #1 Years Validated (e.g ) Name of database: Code Type Diagnostic Procedure Uncertain Code Format ICD-8 ICD-9 ICD-10 CPP Other Codes Used for Validation Diagnosis Code Field Population Source and Sample Size (describe how they got to sample size used for Validity calculation i.e. N=50000 MS patients -> N=2000 selected for chart review) Reference Standard All Most Responsible/Primary Secondary Complication Unspecified Other Data Source (e.g. Medical chart) Definition of positive case in reference standard Administrative data definition (i.e. ICD code or >1 claim with ICD code etc.) Sensitivity Specificity PPV NPV reported in paper reported in paper reported in paper Not Not Not Not 70

83 Validation #2 Kappa reported in paper reported in paper Not Years Validated (eg ) Name of database: Code Type Diagnostic Procedure Uncertain Code Format ICD-8 ICD-9 ICD-10 CPP Other Codes Used for Validation Diagnosis Code Field Population Source and Sample Size (describe how they got to sample size used for Validity calculation i.e N=50000 MS patients -> N=2000 selected for chart review) Reference Standard All Most Responsible/Primary Secondary Complication Unspecified Other Data Source (eg. Medical chart) Definition of positive case in reference standard Administrative data definition (i.e ICD code or >1 claim with ICD code etc) Sensitivity reported in paper Not 71

84 Validation #3 Specificity PPV NPV Kappa reported in paper reported in paper reported in paper reported in paper Not Not Not Not Years Validated (eg ) Name of database: Code Type Diagnostic Procedure Uncertain Code Format ICD-8 ICD-9 ICD-10 CPP Other Codes Used for Validation Diagnosis Code Field Population Source and Sample Size (describe how they got to sample size used for Validity calculation i.e N=50000 MS patients -> N=2000 selected for chart review) All Most Responsible/Primary Secondary Complication Unspecified Other Reference Standard Data Source (eg. Medical chart) Definition of positive case in reference standard 72

85 Administrative data definition (i.e ICD code or >1 claim with ICD code etc) Sensitivity Specificity PPV NPV Kappa reported in paper reported in paper reported in paper reported in paper reported in paper Not Not Not Not Not 73

86 Appendix B: Quality Assessment Tool from Benchimol et al. (2011) [53] 74

87 75

88 76