Abstract PERFORMANCE OF THE IMMEDIATE POST-CONCUSSION ASSESSMENT AND

Transcription

1 Abstract MANDERINO, LISA M., M.A., MARCH 2017 PSYCHOLOGICAL SCIENCES PERFORMANCE OF THE IMMEDIATE POST-CONCUSSION ASSESSMENT AND COGNITIVE TESTING PROTOCOL VALIDITY INDICES (40 pp.) Thesis Advisor: John Gunstad Immediate Post-Concussion Assessment and Cognitive Testing (ImPACT) is a widely used, computerized neuropsychological test battery for the diagnosis and management of sport-related concussions (SRC). As SRC is known to affect neurocognitive performance, athletes provide pre-injury, baseline ImPACT scores, to which post-injury scores can be compared in the event of SRC. However, if an athlete s baseline scores are not fully representative of his abilities, the utility of post-injury score comparison is diminished. Return-to-play release may be granted prematurely, putting the athlete at risk for second injury or long-term consequences. For this reason, the ImPACT includes low score thresholds on five validity indices to identify insufficient effort, though evidence of these indices performance is limited. The present study compares existing ImPACT validity indices, as well as three proposed indices not currently being used to inform protocol validity, to external validity measures. The ImPACT, Word Memory Test (WMT) and Minnesota Multiphasic Personality Inventory 2- Restructured Form (MMPI-2-RF) were administered to 242 undergraduate students. Participants were instructed to either give full effort on testing or to simulate SRC. The ImPACT demonstrated significantly higher specificity (0.94) and lower sensitivity (0.42) as compared to the WMT and MMPI-2-RF. Alternative score thresholds for the existing ImPACT validity indices may maintain high specificity while

2 improving sensitivity. The three proposed indices showed higher sensitivities than the existing ImPACT indices, though lower specificities. The existing ImPACT indices high specificity at the expense of lower sensitivity compared to external validity measures raises concern, as unidentified poor-effort may result in premature return-to-play decisions for athletes with SRC. Improvements or additions to the existing indices may raise sensitivity while maintaining acceptable specificity, aiding in the protection of athletes and safe athletic participation.

3 PERFORMANCE OF THE IMMEDIATE POST-CONCUSSION ASSESSMENT AND COGNITIVE TESTING PROTOCOL VALIDITY INDICES A thesis submitted To Kent State University in partial Fulfillment of the requirements for the Degree of Master of Arts by Lisa Marie Manderino March 2017 Copyright All rights reserved Except for previously published materials

4 Thesis written by Lisa Marie Manderino B.A., Wake Forest University, 2014 M.A., Kent State University, 2017 Approved by John Gunstad, Advisor Maria Zaragoza, Chair, Department of Psychological Sciences James L. Blank, Dean, College of Arts and Sciences

5 ABSTRACT... i TABLE OF CONTENTS..... v LIST OF TABLES vi ACKNOWLEDGMENTS... vii INTRODUCTION. 1 METHODS.. 12 RESULTS 18 DISCUSSION.. 21 REFERENCES APPENDICES A. Simulated malingering script.. 41 v

6 LIST OF TABLES Table 1. Examination of group differences on key demographic and testing variables.. 28 Table 2. ImPACT subtests used to calculate composite scores and protocol validity Table 3. Number of participants properly and improperly identified.. 30 Table 4. Results of t-tests examining group differences on validity indices Table 5. Resulting p values from McNemar s tests, comparing sensitivities.. 32 Table 6. Resulting p values from McNemar s Tests, comparing specificities 32 Table 7. Classification accuracy statistics for exploratory score cutoffs. 33 vi

7 ACKNOWLEDGMENTS I wish to thank my thesis committee for their time and careful guidance. Thank you to Dr. Yossef Ben-Porath for his generous contribution to study measures, to Dr. Anthony Tarescavage for his analytical expertise, and to the MMPI-2-RF Research Group at Kent State University for their advice and technical support. Thank you to my colleagues and friends in the Kent State Neuropsychology Research Laboratory for their vital support, both professionally and personally. I would especially like to thank Dr. John Gunstad for his continued contributions towards my professional development as a clinical psychologist. Finally, I thank my mother, Betsy, and my sister, Corrine, for their endless laughter and encouragement, and I thank my grandfather, Eugene, for always championing the importance of hard work and perseverance. vii

8 Introduction Epidemiology of Traumatic Brain Injury Traumatic brain injury (TBI) has garnered significant public attention in the past two decades, as the already high prevalence of TBI appears to be on the rise. In 2010, a reported 2.5 million emergency department visits, hospitalizations, and deaths in the United States were associated with TBI, either alone or in combination with other injuries (Center for Disease Control and Prevention [CDC], 2014). Over the last decade, TBI-related hospitalizations have increased by 11%, and TBI-related emergency visits have increased by an astonishing 70% (CDC, 2014). Despite evidence that TBI is becoming increasingly common, TBI-related deaths have decreased by 7% over the last decade (CDC, 2014). This likely speaks to science and medicine s rapidly increasing knowledge about TBI diagnosis, treatment, and rehabilitation. Even with the decline in TBI-related deaths, however, the societal cost of TBI remains high. Most individuals having sustained a TBI suffer at least a short period of impaired functioning, resulting in lost productivity at work or school and lost financial earnings. In fact, the societal cost of TBI in lost productivity alone is estimated to be greater than $326,000,000 annually (Finkelstein, Corso, & Miller, 2006). For some, the period of impaired functioning is longer lasting. Between medical care, lost productivity, and other associated costs, it is estimated that TBI in the United States costs $60 billion annually (Langlois, Rutland-Brown, & Wald, 2006). TBI also causes significant distress, both for patients and their support systems. Injured persons can experience a range of symptoms, including cognitive difficulties, physical 1

9 complaints, and psychological symptoms (Hyatt, 2014). Studies have found that these symptoms are often distressing to the patient s family, caregivers, and support networks. Notable outcomes from such studies include depression, anxiety, decreased family functioning, and decreased marital satisfaction (Hyatt, 2014). Diagnostic Definitions and Severity of Traumatic Brain Injury Despite extensive research over the past 20 years, little consensus exists in the literature regarding the pathology, long-term consequences, and even diagnostic criteria for TBI. The CDC defines TBI as a disruption of normal brain function due to a bump, blow or jolt to the head, or to a penetrating head injury (2014). However, not all bumps or blows to the head result in TBI, and TBI can range in severity from mild to severe regardless of cause (e.g., vehicular accident, military trauma, athletic participation, etc.). The CDC reports that severe TBI can be characterized by an extended period of unconsciousness or amnesia post-injury, whereas a mild TBI (mtbi) can be defined as a brief change in mental status or consciousness (CDC, 2014). Notably, these definitions lack specific, objective diagnostic information, such as what constitutes a brief period of time as opposed to an extended period. A separate definition of mtbi frequently used in the field states that mtbi can be diagnosed by any of the following: 1) any period of lost consciousness, 2) any loss of memory for events immediately before or after the blow to the head, 3) any alteration in mental state at the time of the impact, or 4) focal neurological deficits. However, to be considered an mtbi, the severity cannot exceed any of the following: 1) loss of consciousness of less than approximately 30 minutes, 2) an initial Glasgow Coma Scale of 13-15, and 3) loss of memory for events following the impact cannot last for greater than 24 hours (American Congress of Rehabilitation 2

10 Medicine, 1993). While this definition provides more objective criteria, such as citing timeperiods for symptoms like loss of consciousness, it still only approximates time and score cutoffs. Sports-Related Concussion Sports-related mild traumatic brain injury, or sports-related concussion (SRC), refers to mtbi in an athletic setting. Here, too, consensus definitions have not yet been reached; though working definitions proposed by researchers in the field largely overlap. The Concussion in Sport Group produced a vague but inclusive consensus definition in 2012, calling sport-related concussion A complex pathophysiological process affecting the brain, induced by biomechanical forces (McCrory, et al., 2012). This definition also highlighted several common features of sport-related concussion, incorporating clinical, pathological, and biomechanical injury constructs. Regardless of the criteria used, acute neurological dysfunction in the absence of significant structural damage is ubiquitously considered a hallmark feature of SRC (King, 2014; McCrory, et al., 2012). SRC is common, with yearly estimates ranging from 1.6 to 3.8 million injuries occurring annually (Langlois, Rutland-Brown, & Wald, 2006). Such variable estimates result from the difficulty of obtaining exact estimates due to underreporting by athletes and inconsistent detection by clinicians. The clinical management of SRC is distinguished from that of TBI by several unique clinical considerations. While TBI is generally more common in men than women (CDC, 2014), SRC is actually more likely in female athletes (Daneshvar, Nowinski, McKee, & Cantu, 2011). SRC is also significantly more common in children and adolescents than other age groups (CDC, 2014). Additionally, second injury rates are particularly high in SRC due to 3

11 inappropriate return-to-play decisions (Guskiewicz, Weaver, Padua, & Garrett, 2000). This is a concerning finding, as second injury comes with risks of long lasting impairment and even death (Cantu, 1998; Kelly & Rosenberg, 1997). Cognitive Testing in Sports-Related Concussion Due to the high incidence of and dangers associated with misdiagnosed and mismanaged SRC, efforts are being made to refine diagnosis. As the subtlety of concussion symptoms and pathology make diagnosis challenging, it is recommended that multiple sources of information be used in diagnostic practices (King, 2014). Common sources of diagnostic information include self-report, neurological evaluation, observable signs, and cognitive testing. It is known that concussion results in acute cognitive changes, and that some injuries can result in longer-lasting cognitive effects (Guskiewicz & Broglio, 2015). Repeated, subconcussive injuries over time may also produce chronic cognitive changes (Guskiewicz & Broglio, 2015). Thus, documenting an individual s pre-injury cognitive functioning allows for comparison after suspected SRC and provides a benchmark of typical functioning which should be reached again before return-to-play. A widely used tool for neurocognitive evaluation in SRC is the Immediate Post- Concussion Assessment and Cognitive Testing (ImPACT). The ImPACT assesses cognitive changes of SRC by administering a computerized battery of neuropsychological tests and producing user-friendly score reports for cognitive domains including attention, working memory, problem solving, and reaction time (Lovell, 2015). Post-injury ImPACT scores can be compared to preseason baselines to aid in return-to-play decision-making. For this reason, obtaining an accurate assessment of pre-injury functioning is critical for the utility of the test. 4

12 Low Effort and Invalid Protocol Warnings The ImPACT Administration Manual states that the majority of athletes find the test to be challenging enough to evoke a full effort performance, though this is not always the case (Lovell, 2015). For example, some athletes may be distracted or fatigued at the time of testing and thus unable to give their maximum performances (Lovell, 2015). In other instances, athletes may intentionally suppress their scores, or sandbag their baseline, in order to obscure post-injury deficits in the event of a later SRC (Lovell, 2015). That is, an athlete may attempt to suppress his baseline assessment scores so that his genuinely impaired post-injury test scores look similar to his artificially poor baseline test scores, decreasing the likelihood that he will be held from competition and miss playing time for impaired neurocognitive scores. However, obscuring postinjury impairments in this manner may lead to premature return-to-play for an injured athlete, which can lead to chronic cognitive changes and repeat injury, as discussed above (Cantu, 1998; Kelly, 1997). While sandbagging is inherently difficult to study empirically, as it is a phenomenon entirely built upon the desire to evade detection, there is anecdotal evidence to suggest that its incidence is high enough to be of concern. Speculation regarding possible motives for sandbagging has included athletes reluctance to miss playing time, fear of losing a position on the team, and desire to contribute to the team (Echemendia & Cantu, 2003). Regardless of motive, the appeal of minimizing SRC symptoms appears to span all levels of athletic participation. A survey in high school football players showed that 53% intentionally did not report SRC symptoms (McCrea, Hammeke, Olsen, Leo, & Guskiewicz, 2004), and a poll of current National Football League players indicated that 56% of said that they would not report 5

13 SRC if they felt that the symptoms were mild enough that they could be successfully hidden (Sporting News, 2012). While there are other methods of hiding SRC symptoms, and these findings do not necessarily indicate that greater than 50% of all athletes are sandbagging baseline neurocognitive tests, even high profile athletes have spoken to this method specifically. In a 2011 interview, two time Super Bowl champion quarterback Peyton Manning stated,... after a concussion, you take the same test and if you do worse... you can t play. So I just try to do badly on the first test, (Reilly, 2011). With high percentages of athletes concealing SRC symptoms and role model athletes publicly lauding sandbagging, developing efficient safeguards against this strategy is important. In an effort to identify sandbagging athletes and avoid problematic outcomes associated with premature return-to-play, ImPACT uses embedded protocol validity indices that alert practitioners of low scores unlikely to be representative of the average athlete s true abilities. Specifically, five ImPACT scores are used as validity indices and contain validity score thresholds. A score below the predetermined validity threshold on any one of these five indices automatically triggers the printing of an invalid protocol warning on the ImPACT report. This warning states, Test results were very low. Test might be invalid. The score thresholds for the five ImPACT validity indices are based on confidence intervals, such that 95% of athletes taking the ImPACT score higher than the validity threshold (Lovell, 2015). While scores in the fifth percentile and lower are uncommon compared to scores in the sixth percentile and greater, it does not necessarily indicate that such scores are not representative of a full effort performance for an athlete s true abilities. 6

14 Causes of Invalid Protocols The purpose of ImPACT s invalid protocol warning is to identify protocols by which the test-taker s true abilities are not captured. Common causes of baseline test invalidity include excessive fatigue or distraction by a noisy testing environment. Athletes may also fail to properly read testing instructions or confuse the left and right mouse buttons (Lovell, 2015). In these cases, while the invalid protocol warning is not necessarily indicative of poor effort or sandbagging, athletes would still benefit from retaking the baseline test to ensure availability of scores representative of their pre-injury cognitive abilities. On the other hand, some athletes may have preexisting conditions that result in their functioning below the fifth percentile, even with full effort. For example, an athlete with attention deficit hyperactivity disorder (ADHD) may have impaired sustained attention and be unable to sustain attention throughout baseline test administration. This inability to concentrate may produce globally suppressed test scores and trigger an invalid protocol warning on any one of the five validity indices (Elbin, et al., 2013; Manderino & Gunstad, in press; Schatz, Moser, Solomon, Ott, & Karpf, 2012). Athletes with other academically related histories, such as learning disorders, may also be at risk for producing an invalid protocol warning despite giving maximum effort on baseline testing (Elbin, et al., 2013; Johnson, Pardini, Sandel, & Lovell, 2014; Manderino & Gunstad, in press; Schatz, Moser, Solomon, Ott, & Karpf, 2012). These athletes may not benefit from retaking the baseline test, as their low scores are indicative of their uninjured abilities and are unlikely to improve. As the existing validity indices are unable to distinguish low scores due to poor effort as opposed to genuine poor performances, the ImPACT administration manual states that the onus is on the practitioner administering the test to 7

15 determine the reason for an invalid protocol warning and to administer a retest if appropriate (Lovell, 2015). Performance of ImPACT Protocol Validity Indices Reported incidences of invalid protocols on the online version of ImPACT range from 2.7% (Nelson, Pfaller, Rein, & McCrea, 2015) to 6.3% (Schatz, Moser, Solomon, Ott, & Karpf, 2012). Invalidity rates have been shown to be highest among youth athletes (Gaudet & Weyandt, 2017), football players (Gaudet & Weyandt, 2017), and athletes who have ADHD or learning disorders (Manderino & Gunstad, in press). The incidence rate of invalid protocol warnings is also higher for the desktop version of the ImPACT than for the online version. In high school athletes, the desktop version flagged 11.9% of baseline protocols as invalid, whereas the online version flagged only 6.3% (Schatz, Moser, Solomon, Ott, & Karpf, 2012). This difference was shown to be similar in collegiate athletes, with 10.2% being flagged in the desktop version and 4.1% in the online version (Schatz, Moser, Solomon, Ott, & Karpf, 2012). These findings suggest that the ImPACT online version may be less sensitive to low effort than previous versions and support the ImPACT Administration Manual s caveat that an invalid protocol warning may be caused by factors beyond poor effort alone. Few studies have empirically investigated the ImPACT validity indices identification of poor effort. Erdal (2012) asked ex-collegiate athletes who had already completed an ImPACT baseline assessment to feign poor performance. They found that 11% of participants were able to successfully lower scores from baseline without reaching threshold on any of the five embedded validity indices (i.e., 89% of participants were correctly identified as giving poor effort). Another study utilizing a simulated malingering design examined whether additional scores on the 8

16 ImPACT could be used as validity indices to improve identification of sandbagging in naïve malingering and coached malingering groups. Only 70% of naïve malingerers and 65% of coached malingerers were correctly identified as feigning by the embedded validity indices (Schatz & Glatts, 2013). However, Schatz and Glatts (2013) identified two forced choice recognition tasks, a common task design used in external measures of effort, lying dormant within the ImPACT. With the addition of the score thresholds Word Memory Correct Distractors (Immediate + Delayed) (WMCD) < 22 and Design Memory Correct Distractors (Immediate + Delayed) (DMCD) < 16, a full 95% of naïve malingerers and 100% of coached malingerers were correctly identified. Their results suggest that the addition of validity score thresholds on these two indices have the potential to significantly improve the identification of poor effort on the ImPACT, though the small sample size used in the study led the authors to interpret their findings with caution. External Validity Measures While evidence of the ImPACT s detection of poor effort is limited, many external measures of performance and symptom validity exist and have been substantiated for use in TBI evaluations in a variety of settings. The Minnesota Multiphasic Personality Inventory-2- Restructured Form (MMPI-2-RF) includes over-reporting scales intended to identify individuals engaging in feigning somatic, psychological, or cognitive complaints. The MMPI-2-RF is frequently administered alongside neuropsychological test batteries and has been well validated for use in TBI evaluations in disability and criminal forensic settings and (Larrabee, 2008; Sellbom, Toomey, Wygant, Kucharski, & Duncan, 2010; Tarescavage, Wygant, Gervais, & Ben- Porath, 2013; Wygant, et al., 2009). The Word Memory Test (WMT) is a standalone measure for 9

17 the detection of feigned neurocognitive impairment that is also frequently used in disability and forensic settings. The WMT has face validity as a challenging memory test, though scores on the WMT have shown high sensitivity to effort and minimal sensitivity to genuine cognitive impairments in brain-injured patients (Green, Flaro, & Courtney, 2009; Green, Iverson, & Allen, 1999; Iverson, Green, & Gervais, 1999). The success of these external validity measures in forensic settings, where feigned symptoms and impairments are highly incentivized (e.g., disability claims), suggest that they and other such measures are likely to be generally effective in samples implicitly motivated to suppress performance on neuropsychological testing. In fact, both the MMPI-2-RF and the WMT are recommended for inclusion in neuropsychological evaluations of SRC specifically, to inform clinical judgments with information regarding possible malingering, somatization, or symptom magnification due to comorbid psychological conditions (Barr, 2014). Present Study Given the anecdotal evidence for sandbagging baseline ImPACT administrations, contrasted with the relatively low prevalence rates of invalid protocols reported in existing research, the present study seeks to provide clinically useful evidence of the ImPACT s protocol validity classification accuracy. By comparing the performance of the ImPACT to external validated measures of performance/symptom validity, evidence for the ImPACT validity indices concurrent validity is provided, filling a gap in the existing literature. Finally, the investigation of exploratory validity indices (the two indices proposed by Schatz and Glatts [2013] and the total symptom score) may yield additional improvements for the current ImPACT protocol validity indices and future directions for research in this area. 10

18 Aims/Hypotheses The current study examines the performance of the five ImPACT validity indices, particularly in comparison to external validity measures, and explores whether identification of poor effort on baseline ImPACT administrations can be improved. Specific aims and hypotheses are as follows: Aim 1: Compare ImPACT s protocol validity indices to validated, external measures of effort, the MPPI-2-RF and the WMT. Hypothesis 1: The identification of feigning by the existing ImPACT validity indices will be lower than by the MMPI-2-RF and the WMT. Aim 2: Determine if the ImPACT s sensitivity to feigning may be improved by utilizing existing scores embedded in the ImPACT. Hypothesis 2: Sensitivity to feigning by the ImPACT will be improved by using more liberal score thresholds on the existing validity indices. Hypothesis 3: Sensitivity to feigning as identified by WMCD and DMCD, proposed by Schatz and Glatts (2013), will be higher than sensitivity of the existing ImPACT validity indices. Hypothesis 4: The Total Symptom Score will be significantly different between individuals feigning poor performances and individuals giving full effort performances. Aim 3: Provide information on the performance of the ImPACT validity indices that may help to inform clinical practice at various testing sites. Hypothesis 5: The positive and negative predictive power of the ImPACT validity indices will be inherently affected by the base rate of poor effort. 11

19 Methods Participants: A total of 277 participants were recruited from the psychology department subject pool. Analyses were limited to include only participants who had complete data, spoke English as a first language, and appeared to follow testing instructions per their group assignment (n = 242). Participants were randomly assigned to either the simulating (n= 118) or control group (n = 124). Participant age ranged from (mean age = 19.6 ± 1.99). A total of 23.1% of participants reported at least one previous concussion. Independent t-tests revealed only one significant between-groups difference on key demographic or concussion history variables. The average number of years of education differed significantly between groups, such that the simulating group (12.92 ± 1.93 years) had approximately one half year of education more than the full effort group (12.33 ± 2.18 years) on average. Although this is a statistically significant difference, one half year of education is not expected to be a clinically meaningful confound between the two groups. See Table 1 for details on participant characteristics. Due to technical errors, data regarding previous exposure to the ImPACT were available for only a subset of participants (Subset n = 102). Of this subset, 32.4% of participants reported that they had taken the ImPACT at least once in the past, and this proportion did not significantly differ between groups (χ 2 (1) = 0.08, p = 0.47). 12

20 Measures: Immediate Post-Concussion Assessment and Cognitive Testing (ImPACT) (Lovell, 2015): ImPACT is a widely used computerized neuropsychological test battery for the assessment of neurocognitive functioning specifically in the context of SRC. ImPACT includes six subtest modules, each of which generates several scores, and from the subtest scores produces five composite scores. The composite scores include verbal memory, visual memory, visual motor speed, reaction time, and impulse control. Notably, the impulse control composite score is calculated using errors on subtest modules and was developed for use solely as an effort measure rather than as a measure of cognitive functioning. The proposed validity indices, WMCD and DMCD, are calculated by summing the correct distractor (i.e., true negatives) scores from the immediate and delayed recall portions of the Word Memory and Design Memory subtests, respectively. Functionally, these subtests present forced choice recognition tasks, a task design frequently used in measuring effort. The ImPACT also includes a total concussion symptom score. Examinees are presented with a list of 22 common SRC symptoms (e.g., headache, nausea, dizziness) and are asked to rate the severity of each symptom on a 6-point Likert scale ranging from 1 (minor discomfort) to 6 (severe) or to check a box to indicate that they are not experiencing the symptom at all. The Total Symptom Score is the sum of all symptom ratings (see Table 2 for information on subtests, composite scores, and validity indices). The construct and concurrent validity of the ImPACT as a test of neurocognitive function has been previously documented. The ImPACT shows significant overlap with traditional neuropsychological measures used for the assessment of SRC (Allen & Gfeller, 2011; Maerlender, et al., 2010) and is sensitive to cognitive changes following SRC even in athletes denying SRC symptoms (Schatz & Sandel, 2012). Recent studies using the online version of 13

21 ImPACT have demonstrated acceptable test-retest reliability, concluding adequate stability for the ImPACT neurocognitive scales to be used in conjunction with other clinical tools for the assessment of SRC (Elbin, Schatz, & Covassin, 2011; Nakayama, Covassin, Schatz, Nogle, & Kovan, 2014). Schatz and Ferris demonstrated that the ImPACT s alternate forms successfully guard against practice effects for all but speeded components of the ImPACT (2013). Word Memory Test (WMT) (Green, 2003): The WMT is a computerized, forced choice recognition symptom validity test. Participants are presented with a 10 pairs of semantically related words, and, in a series of immediate and delayed tasks, must demonstrate memory of the learned list. The WMT can be completed in approximately 20 minutes and, in a variety of patient samples, has demonstrated good sensitivity and specificity as a measure of effort (Green, 2003; Green, Flaro, & Courtney, 2009). While the task is meant to appear challenging enough that a malingering individual would be likely to suppress performance, patients with genuine psychological and neurological conditions consistently score greater than 90% correct (Green, 2003). Minnesota Multiphasic Personality Inventory 2 Restructured Form (MMPI-2-RF) (Ben-Porath & Tellegen, 2008): The MMPI-2-RF contains 338 True/False items regarding aspects of personality and psychopathology, from which clinical and validity scales are derived. The final protocol includes five over-reporting validity scales, which have been previously validated for use in civil forensic settings (Tarescavage, Wygant, Gervais, & Ben-Porath, 2013; Wygant, et al., 2009). An elevation on the Infrequent Responses scale (F-r) indicates that the individual has answered items from the scale in a direction that less than 10% of the normative sample had answered. The Infrequent Psychopathology Responses scale (Fp-r) is elevated by answers on its items observed at less than 10% in a sample of individuals with severe 14

22 psychopathology, while the Infrequent Somatic Responses scale (Fs) includes items endorsed by less than 10% of medical and chronic pain patients. The Fp-r and Fs scales provide comparisons for the F-r scale, which can be confounded by genuine, albeit infrequent, psychological and somatic complaints. The Symptom Validity scale (FBS-r) includes items related to exaggerated somatic and neurocognitive complaints, as elevations on FBS-r are associated with poor effort and malingering rather than any genuine neurological or psychological conditions (Wygant, et al., 2009). Finally, the Response Bias Scale (RBS) was developed explicitly as a measure of feigned test performance and has shown high associations with external symptom validity tests (Tarescavage, Wygant, Gervais, & Ben-Porath, 2013). Procedures: All study procedures were approved by the local ethical review board, and all participants provided written informed consent before participating in the study. Participants completed computerized measures in groups of approximately 10 to 20 per testing session in a university computer laboratory. Groups of participants were randomly assigned to either the control or simulating conditions (all participants in a given session in the same condition) and instructions were read aloud by a study team member at the beginning of the testing session. Instructions to either put forth full effort or simulate a concussion were also reiterated by computerized prompts. Participants in both groups were informed that the tests they would be taking were designed to identify individuals not putting forth full effort. Instructions for the control condition emphasized that participants should try their hardest on testing, and that all participants identified by the tests as putting forth adequate effort would be entered into a gift card raffle. Instructions for the simulating condition described common deficits experienced by individuals with a 15

23 concussion and instructed participants to feign such deficits without performing so poorly that the test identifies them as putting forth inadequate effort. Simulating participants were told that only those individuals who suppressed their scores without triggering a poor effort warning would be entered into a gift card raffle, in order to provide external incentive for successful sandbagging (See Appendix 1 for simulating group instructions). All participants received the computerized tests in a standardized order to optimize timing of the delayed recall portion of the WMT: 1) Demographic survey, 2) MMPI-2-RF, 3) WMT Learning, 4) ImPACT Test, 5) WMT Delayed Recall, 6) Exit survey. This ordering allowed for an average delay of approximately 30 minutes between WMT Learning and Delayed Recall. After completing the demographic questionnaire honestly, participants were prompted to either begin trying their hardest or feigning a head injury on the following measures. The exit survey began with instructions to discontinue feigning while answering the remaining questions. This exit survey asked participants to describe their group instructions as a validity check for feigning and to answer questions regarding the amount of effort put into following group instructions. Participants were then debriefed and informed that all participants (regardless of whether or not they triggered a poor effort warning) would be entered into the raffles. Protocols for each measure (i.e., WMT, ImPACT, MMPI-2-RF) were initially classified as invalid based on published score thresholds (Table 3). In the cases of scales with multiple published validity thresholds, the most lenient cutoffs were used, as there is no presumed impairment in the present sample of college students. Power Analysis: A previous study on the ImPACT validity indices was able to detect differences in ImPACT scores with a smaller sample size (n = 60) than that used in the present study (Schatz & 16

24 Glatts, 2013). Moreover, a power analysis (α = 0.05, power = 0.80) indicates that a total sample of 80 participants would yield sufficient power to detect the previously observed group differences on ImPACT scores. Given this analysis, the present sample of 242 participants should be adequate to detect even small group differences on the Total Symptom Score (Hypothesis 4) and to provide adequate cell sizes for the comparisons of sensitivities and specificities among the ImPACT, WMT, and MMPI-2-RF (Hypothesis 1). Statistical Analyses: A series of t-tests was performed on all standard and proposed validity indices to demonstrate group differences for full-effort and sandbagging performances. Next, true positive (sensitivity) and true negative (specificity) rates were calculated for all validity indices, to characterize their abilities to detect sandbagging and differentiate sandbagging from full effort. Positive predictive power (PPP) and negative predictive power (NPP) were also calculated for all validity indices, first for the data available (i.e., approximately 50% of the sample feigning) and then for hypothetical base rates of sandbagging as informed by the literature. These classification accuracy statistics were also calculated for each instrument as a whole (e.g., the sensitivity and specificity for the ImPACT overall, as a product of identification by any one of the five validity indices). Classification accuracy statistics were then calculated for exploratory score thresholds on the five existing and three proposed ImPACT validity indices, to determine whether more liberal cutoffs may improve sensitivity without detrimentally affecting specificity. Classification accuracy for the standard ImPACT as a whole and the three exploratory ImPACT indices were then statistically compared to the MMPI-2-RF and the WMT using McNemar s Tests to determine whether any current or exploratory score thresholds can reach the same level of performance as these gold-standard instruments. 17

25 Results Group Differences on Validity Indices T-tests examining group differences on the five ImPACT validity indices, three WMT validity indices, and five MMPI-2-RF validity indices were all significant (p < 0.001). Significant group differences were also observed on all three ImPACT exploratory validity indices (WMCD, DMCD, and Total Symptom Score) (Table 4). As the Total Symptom Score is suggested here for the first time as a possible validity index, a score threshold has not been previously proposed. Group means and standard deviations were examined to determine a proposed Total Symptom Score invalidity threshold of > 20 for initial classification accuracy investigations, though other score thresholds are examined below. Sensitivity and Specificity Comparisons for Existing Score Cutoffs Classification accuracy statistics for the published cutoffs of each instrument (ImPACT, MMPI-2-RF, and WMT) as a whole, as well as each index individually, are presented in Table 3. Of the 58 protocols that were flagged as invalid by the standard ImPACT indices, 50.0% surpassed the validity threshold for only 1 index, 29.3% invalidated 2 indices, 6.9% invalidated 3 indices, and 12.1% invalidated 4 indices. Only one protocol (1.7%) surpassed thresholds on all five standard ImPACT validity indices. Overall, the ImPACT demonstrated very high specificity (0.94) at the expense of considerably lower sensitivity (0.42). Consequently, PPP (0.86) was higher than NPP (0.63). The three proposed ImPACT validity indices demonstrated comparatively higher sensitivity rates, though with lowered specificity. Specifically, WMCD 18

26 demonstrated a sensitivity of 0.74 with specificity of 0.66, while DMCD demonstrated a sensitivity of 0.69 and a specificity of 0.65, and Total Symptom Score demonstrated a sensitivity of 0.75 and a specificity of The sensitivity of the standard ImPACT was significantly lower than the sensitivity rates of the MMPI-2-RF and the WMT, as well as each of the three exploratory validity indices alone (all p < 0.001, Table 5). The sensitivity of WMCD was significantly higher than the sensitivity of the standard ImPACT (p < 0.001) and the WMT (p < 0.005), though it was not significantly different from the sensitivity of the MMPI-2-RF (p = 0.088). The sensitivity of DMCD was also significantly higher than that of the standard ImPACT (p < 0.001) and higher than the MMPI-2- RF (p < 0.050), though it was not significantly different from the WMT (p = 0.080). The Total Symptom Score was significantly more sensitive than both the standard ImPACT (p < 0.001) and the WMT (p < 0.05), though it was not significantly different from the MMPI-2-RF (p = 0.143). The specificity of the standard ImPACT was significantly higher than the MMPI-2-RF (p < 0.001) and the WMT (p < 0.05), as well as all three exploratory ImPACT indices (all p < 0.001, Table 6). WMCD was significantly less specific than the standard ImPACT (p < 0.001) and the WMT (p < 0.001), though was not significantly different from the MMPI-2-RF (p = 0.220). The specificity of DMCD was also significantly lower than that of the standard ImPACT (p < 0.001) and the WMT (p < 0.001), and was not significantly different from the MMPI-2-RF (p = 0.161). The Total Symptom Score was significantly less specific than the standard ImPACT (p < 0.001), the WMT (p < 0.001), and the MMPI-2-RF (p < 0.05). 19

27 Exploratory Score Thresholds and Base Rate Analyses Finally, the effects on sensitivity and specificity of more liberal validity score thresholds were explored. In selecting alternate score thresholds, sensitivity was maximized while a minimum specificity of 0.90 was preserved. These results are presented in Table 7. Generally, the sensitivities of individual validity indices were able to be marginally improved while maintaining acceptable specificity with adjusted score thresholds, though rarely to above 50%. As positive and negative predictive powers are affected by base rates, these were calculated for each presented alternate score threshold at various hypothetical base rates of sandbagging. The selected hypothetical base rates (5%, 10%, 15%, 20%, and 25%) were informed in part by the current literature on ImPACT protocol invalidity (ranging from 2.7% [Nelson, et al., 2015] to 6.3% [Schatz, Moser, Solomon, Ott, & Karpf, 2012]), as well as by literature on the prevalence of malingering in forensic settings (Mittenberg, Patton, Canyock, & Condit, 2002). 20

28 Discussion The present results indicate that the currently employed protocol validity indices of the ImPACT have high specificity at the expense of poor sensitivity as compared to external performance validity measures. An exploration of alternate validity thresholds on the ImPACT revealed that the sensitivity of the current indices can be only marginally improved while maintaining high specificity. An examination of two additional ImPACT validity indices previously proposed in the literature (WMCD and DMCD) suggested that their documented high sensitivity relative to traditional ImPACT indices is accompanied by considerably lower specificity. The sensitivity and specificity of a suggested novel validity index, the Total Symptom Score, indicated at least comparable sensitivity and specificity to the currently used ImPACT validity indices. Several aspects of these findings warrant further discussion. The high specificity of the ImPACT s protocol validity indices is particularly striking when compared to the specificities of external performance validity measures. The MMPI-2-RF and the WMT are frequently used alongside neuropsychological testing in forensic settings with TBI populations (Green, Flaro, & Courtney, 2009; Green, Iverson, & Allen, 1999; Hartman, 2002; Iverson, Green, & Gervais, 1999; Larrabee, 2008; Sellbom, Toomey, Wygant, Kucharski, & Duncan, 2010; Tarescavage, Wygant, Gervais, & Ben-Porath, 2013; Wygant, et al., 2009). In the present study, both of these measures were found to have significantly higher sensitivity to poor effort and lower specificity than the ImPACT validity indices (Tables 5 and 6). Due to the inherent risks and potential for serious repercussions of false positives in forensic settings, tests used in these settings are held to a high standard of specificity. Comparatively, the costs 21

29 associated with a false positive on the ImPACT appear minor, likely including only the costs involved with a second test administration. As the test administrator is obligated to investigate possible causes of an invalid protocol (e.g., fatigue, distraction, poor effort, diagnosed attentional or learning disorders) and re-administer only if necessary, there appears to be less justification for the observed conservative specificity, particularly at the cost of low sensitivity. The risks associated with premature return-to-play and repetitive SRC encourage weighing the higher cost of false negatives than false positives. Athletes whose true abilities are not fully captured on baseline testing may not appear to be impaired on post-injury testing, putting them at risk for early return-to-play and, thus, second injury and lasting impairment (Cantu, 1998; Guskiewicz, 2000; Kelly, 1997). For these reasons, maximizing the identification of poor effort, despite some degree of sacrificed specificity, may be best practice for the safety of athletes. Conversely, the argument could be made that there are indirect costs associated with false invalid warnings. Greater numbers of invalid baseline assessments would demand additional resources from clinicians and training staff in order to follow up with athletes regarding their testing performances. The ImPACT s user-friendly interface and low-burden administration is appealing to athletic departments already facing a high ratio of athletes to training staff. A study on the implementation of the ImPACT across a variety of sports medicine settings indicated that, while almost all treatment settings using the ImPACT administer a baseline test (94.7%), only half of these sites examine baseline test validity (54.8%) (Covassin, Elbin, Stiller-Ostrowski, & Kontos, 2009). As many test users are already noncompliant with published guidelines, increasing the level of responsibility on these over-burdened examiners may diminish the utility of the ImPACT. In the cases that test users do examine test validity but fail to follow up with athletes, unnecessary retests may be administered, again creating a drain on limited site 22

30 resources. Further, unnecessary retests may contribute to athletes negative feelings toward the test, or prompt them to change their strategies during testing. In both situations, retest baseline scores may in fact be less representative of an athlete s typical approach to testing, thus confounding observed post-injury changes. Further discussion of the two previously proposed validity indices, WMCD and DMCD, is also warranted. Consistent with past results indicating high sensitivity to sandbagging (identifying 90% to 100% of malingerers in a simulation design; Schatz & Glatts, 2013), these indices demonstrated much higher sensitivity than the traditional ImPACT validity indices (0.74 sensitivity for WMCD, 0.69 for DMCD). While the existing literature on these measures states that DMCD had a 20% false positive rate and thus may be less useful than WMCD (Schatz & Glatts, 2013), a more thorough evaluation of specificity is lacking. In the current study, the increased sensitivity was found to be at the expense of considerably lower specificity than the existing ImPACT validity indices (0.66 and 0.65, respectively). Lower specificity may not preclude the use of WMCD and DMCD as validity indices, due to the relatively low risk of false positives on baseline testing as discussed above. Test administrators should note their limitations, as protocol invalidity as identified by these indices warrants more cautious interpretation and investigation of cause. Despite this, even while maintaining a minimum specificity of 0.90, the sensitivity of WMCD (0.54) is greater than any one of the existing ImPACT indices alone, suggesting its potential to increase the overall sensitivity of the ImPACT. The investigation of Total Symptom Score as a validity index also indicated potential for its use as a validity index. While maintaining high levels of specificity, the Total Symptom Score demonstrated higher sensitivity than any one of the existing validity indices alone. Specifically, at a score threshold of 44, sensitivity was 0.54 and specificity was Conceptually, the Total 23

31 Symptom Score has face validity as a validity index. Due to the nonspecific nature of SRC symptoms (e.g., fatigue, headache), uninjured athletes may experience some symptoms from the Total Symptom Scale at baseline testing. However, in the absence of SRC these symptoms should be few and mild in severity, if experienced at all. This conceptualization of the Total Symptom Score, in addition to the present findings, suggest that a validity score threshold for the Total Symptom Score may help to differentiate feigning athletes from athletes giving full effort, regardless of neurocognitive testing performance. This is a strength compared to the existing validity scores, which are derived from neurocognitive scores that may be genuinely suppressed due to preexisting conditions, such as ADHD (Manderino & Gunstad, in press). It should be noted, however, that simulated malingering studies have been shown to yield effect sizes much larger than those of actual malingerers on effort measures (Vickery, Berry, Inman, Harris, & Orey, 2001). For this reason, the range of scores yielding acceptable sensitivity and specificity in the present study may not provide the same degree of clinical utility in genuine clinical settings. While it is likely that a naïve sandbagger would elevate the Total Symptom Score at baseline, it would take very little coaching or experience with the test for an athlete to adopt a more sophisticated approach and limit symptom reporting. Further research on the Total Symptom Score, particularly non-simulating study designs, is needed to further determine its potential as a validity index. In addition to highlighting directions for future research on improving the validity indices of the ImPACT, the current results have direct implications for current clinical practice. The costs and benefits associated with either maintaining high specificity or increasing sensitivity to poor effort prevent a clear, ubiquitous best answer for test users to be gleaned from the current findings. Rather, the optimal validity thresholds and resulting sensitivity and specificity may 24

32 vary by treatment setting. Clinicians may be informed by unique site characteristics, such as burden of treatment staff and likely prevalence of sandbagging, to guide selection of validity thresholds. The hypothetical base rate analyses presented here may provide clinicians with information to aid interpretations of protocol validity based on such site-specific characteristics. Factors limited to individual testing sessions, such as temporal proximity to a rigorous practice or time of day, may further help to inform clinical judgment when examining an athlete s baseline scores. If an athlete s score is close to but not surpassing a validity threshold, though the examiner is aware that the athlete had a rigorous, early morning practice, follow up regarding fatigue during testing is warranted, and a retest may be appropriate despite the absence of an invalid protocol warning. Additionally, knowledge of the ImPACT validity indices low sensitivity as compared to external performance validity tests may prompt clinicians to administer collateral performance or symptom validity measures in addition to the ImPACT. The inclusion of collateral effort measures may improve the detection of poor effort at baseline testing, as well as help to inform clinicians of the characteristics of their testing site and athletes to improve their interpretations of the embedded ImPACT validity indices. The current findings are limited in several ways. Although unlikely to be clinically meaningful, the significant difference in years of education between groups poses a possible confound to the score differences on validity indices. This and other key demographic variables were collected through self-report, and thus their accuracy is inherently limited, and the extent to which they affected ImPACT scores is unknown. Additionally, testing was performed in group settings, which has been demonstrated in past work to affect test scores and protocol validity (Moser, Schatz, Neidzwski, & Ott, 2011). While this may introduce extraneous sources of variance, it does make the present results generalizable to the many athletic settings that 25