Archives of Clinical Neuropsychology Advance Access published April 27, John H. Denning*

Transcription

1 Archives of Clinical Neuropsychology Advance Access published April 27, 2012 The Efficiency and Accuracy of The Test of Memory Malingering Trial 1, Errors on the First 10 Items of The Test of Memory Malingering, and Five Embedded Measures in Predicting Invalid Test Performance John H. Denning* Mental Health Care Line, Department of Psychology, Tennessee Valley Healthcare System, Alvin C. York Veterans Affairs Hospital, Murfreesboro, TN, USA *Corresponding author at: Mental Health Care Line, Department of Psychology, Tennessee Valley Healthcare System, Alvin C. York Veterans Affairs Hospital, Murfreesboro, TN 37129, USA. Tel.: ; fax: address: (J.H. Denning). Accepted 29 March 2012 Abstract The current study attempted to improve upon the efficiency and accuracy of one of the most frequently administered measures of test validity, the Test of Memory Malingering (TOMM) by utilizing two short forms (TOMM trial 1 or TOMM1; and errors on the first 10 items of TOMM1 or TOMMe10). In addition, we cross-validated the accuracy of five embedded measures frequently used in malingering research. TOMM1 and TOMMe10 were highly accurate in predicting test validity (area under the curve [AUC] ¼ 92% and 87%, respectively; TOMM1 40 and TOMMe10 1; sensitivities.70% and specificities.90%). A logistic regression of five embedded measures showed better accuracy compared with any individual embedded measure alone or in combination (AUC ¼ 87%). TOMM1 and TOMMe10 provide evidence of greater sensitivity to invalid test performance compared with the standard TOMM administration and the use of regression improved the accuracy of the five embedded cognitive measures. Keywords: Test of Memory Malingering; Medical Symptom Validity Test; symptom validity testing; Malingering; Veterans; Neuropsychology Introduction Over the past two decades, there has been an increased interest in the assessment of test validity/malingering in neuropsychological assessment. There are now comprehensive reviews and assessment compilations available to the neuropsychologist regarding the assessment of test validity and malingering (Boone, 2007; Larrabee, 2007) as well as consensus statements from neuropsychological organizations clearly articulating the importance of validity assessment in clinical, research, and forensic settings (ABCN, 2007; Bush et al., 2005; Heilbronner et al., 2009). Despite the variety of tests developed to assess for invalid test performance, there is a constant need to update and refine these measures as patients become more aware of these methods and, therefore, putting at risk the validity of these valuable assessment techniques (Bauer & McCaffery, 2006; Kaufmann, 2009; Morel, 2009). Both freestanding and embedded measures of effort have been utilized across a variety of settings (Sharland & Gfeller, 2007). A robust relationship reflecting lower cognitive performance across neuropsychological tests has been repeatedly found in those failing validity measures (Constantinou, Bauer, Ashendorf, Fisher, & McCaffrey, 2005; Gervais, Rohling, Green, & Ford, 2004; Green, 2006; Green, Rohling, Lees-Haley, & Allen, 2001; Gunner, Miele, Lynch, & McCaffrey, 2012; Locke, Smigielski, Powell, & Stevens, 2008; Marshall et al., 2010; Meyers, Volbrecht, Axelrod, & Reinsch-Boothby, 2011; Schiehser et al., 2011; Suhr, Hammers, Dobbins-Buckland, Zimak, & Hughes, 2008). The expected relationship between the severity of brain injury pathology and neurocognitive measures is often confounded/reduced when validity measures are failed (Fox, 2011). One drawback of many freestanding measures is that administration time is often long, and the end result is often a yes or no finding regarding the validity of performance on that particular task. Increasing the Published by Oxford University Press on behalf of US Government doi: /arclin/acs044

2 2 J.H. Denning / Archives of Clinical Neuropsychology administration efficiency of freestanding measures (while at the same time maintaining high sensitivity to non-credible performance) would be highly valued given the often time-limited nature of many neuropsychological evaluations. In order to assess test validity more efficiently, we will attempt to improve upon the Test of Memory Malingering (TOMM; Tombaugh, 1996) which is already one of the most commonly administered freestanding measures of cognitive test validity (Sharland & Gfeller, 2007). The TOMM has an extensive research base identifying those exaggerating cognitive deficits (Boone, 2007; Larrabee, 2007; Sollman & Berry, 2011; Tombaugh, 1996) with very low false-positive rates in many clinical populations (Greve, Bianchini, Black, et al., 2006; Greve, Ord, Curtis, Bianchini, & Brennan, 2008; Haber & Fichtenberg, 2006; Iverson, Le Page, Koehler, Shojania, & Badii, 2007; Tombaugh, 1996). A recent meta-analytic review of the TOMM by Sollman and Berry (2011) found good accuracy statistics across a range of settings and populations (sensitivity ¼ 65%, specificity ¼ 94%, overall hit rate ¼ 80%). Because the TOMM is often perceived as very easy or not likely measuring cognitive ability (Tan, Slick, Strauss, & Hultsch, 2002), it may not be as sensitive to poor effort compared with other freestanding measures (Armistead-Jehle & Gervais, 2011; Bauer, O Bryant, Lynch, McCaffrey, & Fisher, 2007; Gervais, Rohling, Green, & Ford, 2004; Green, 2007, 2011). Greiffenstein and colleagues (2008) proposed that incorporating all three trials of the TOMM in decision-making (utilizing a cutoff of,45 on Trial 1, Trial 2 or retention trials) provides equivalent concordance rates with the Word Memory Test (WMT, Green, 2003). However, Greiffenstein and colleagues (2008) noted a 15% false-positive rate using this particular method. Gunner and colleagues (2012) recently described the Albany Consistency Index (ACI) for the TOMM showing that this index increased the sensitivity of the traditional TOMM cutoffs from 31% to 71% while maintaining adequate specificities ( 90%). This index shows promise, but it requires additional validation given the small sample size (n ¼ 48), all were (primarily) mild traumatic brain injury (mtbi) litigants, and calculating the index requires that all three trials of the TOMM be given. In an effort to increase the efficiency of administration and possibly increasing the sensitivity of the TOMM to exaggerated cognitive deficits (while still maintaining low false-positive rates in good effort clinical populations), we will build upon previous research that has utilized the score from TOMM Trial 1 (TOMM1) as a measure of test validity. A summary of TOMM1 cutoff scores extracted from several studies (providing sufficient TOMM1 information) as well as from the TOMM test manual (Tombaugh, 1996) is found in Tables 1 and 2. Several studies only provided specificity scores on Table 1. TOMM trial 1 cut scores reflecting 100% specificity for passing the TOMM Study (first author) N Sample TOMM trial 1 cut score Armistead-Jehle (2011) 75 Active duty military (clinical) 32 a 45 Ashendorf (2004) 197 Non-clinical elderly (Anx/Depx) 40 a Bauer (2007) 105 mtbi litigants 44 a Brooks (2012) 53 Pediatric neurology patients 36 a 45 Etherton (2005) 20 Student controls 45 a 20 Acute pain controls 45 Gavett (2005) 77 mtbi litigants 45 Gierok (2005) 20 Psychiatric inpatients 36 a Hilsabeck (2011) 229 Mixed clinical 41 Iverson (2007) 54 Fibromyalgia with depression/pain 40 a Kirk (2011) 101 Pediatric clinical patients 33 a Morgan (2009) 14 Litigants b 39 a Musso (2011) 54 Student controls 39 a 45 O Bryant (2008) 306 Non-clinical elderly 40 a Mixed clinical 41 a 45 Rees (2001) 26 Depressed inpatients 45 Ryan (2010) 72 Student controls 42 a Teichner (2004) 21 Clinical-elderly normal 40 a Tombaugh (1996) 142 Mixed clinical c 41 a 45 Vanderslice-Barr (2011) 96 Student controls 40 a Yanez (2006) 20 Controls 45 Notes: TOMM ¼ Test of Memory Malingering; mtbi ¼ mild traumatic brain injury; Anx ¼ anxiety, Depx ¼ depression. a Denotes inclusion of the entire sample. b Selected case series. c TOMM administration manual (1996).

3 Table 2. Accuracy statistics of TOMM1 in predicting invalid test performance or malingering J.H. Denning / Archives of Clinical Neuropsychology 3 Study (first author) N Sample Comparison Cut score SN SP Armistead-Jehle (2011) 75 Active military (clinical) TOMM Bauer (2007) 105 mtbi litigants TOMM Brooks (2012) 53 Pediatric neurology patients TOMM Duncan (2005) 50 Forensic/psychotic Consensus a Greve (2006) 50 Toxic exposure Slick criteria Greve (2006) 76 mtbi litigants Slick criteria b Mild/severe TBI Slick criteria b Memory disordered Greve, Etherton, et al. (2009) 604 Pain litigants MPRD Pain litigants MPRD Pain simulators Hilsabeck (2011) 229 Mixed clinical TOMM Hill (2003) 105 PNES, TLE TOMM Horner (2006) 114 Mixed clinical TOMM Kirkwood (2010) 6 Pediatric c TOMM and MSVT Morgan (2009) 14 Litigants c TOMM and others Musso (2011) 108 Students/simulators O Bryant (2007) % disability TOMM retention WMT/Slick criteria Rees (2001) 26 Depressed inpatients TOMM Schroeder (2011) 40 mtbi litigants Slick criteria d Tombaugh (1996) 142 Mixed clinical e TOMM Non-demented clinical e TOMM Wisdom (2012) 213 Inpatient epilepsy TOMM Weighted average across samples Notes: TOMM ¼ Test of Memory Malingering; SN ¼ sensitivity (failed comparison measure); SP ¼ specificity (passed comparison measure); MMPI-2, Minnesota Multiphasic Personality Inventory-Second Edition; MPRD ¼ Malingered Pain-Related Disability; MSVT ¼ Medical Symptom Validity Test; mtbi ¼ mild traumatic brain injury; WMT ¼ Word Memory Test; PNES ¼ Psychogenic Non-epileptic Seizure; TLE ¼ Temporal Lobe Epilepsy. Bold values are average of the entire table. a Invalid test results were removed based on the review of medical records. b Failure on Portland Digit Recognition Test or 2 embedded measures (Reliable Digit Span; Mittenberg formula for Wechsler Adult Intelligence Scale-Revised/Wechsler Adult Intelligence Scale-III; Millis formula for the California Verbal Learning Test; Wisconsin Cart Sorting Test: unique responses, Suhr formula; Minnesota Multiphasic Personality Inventory-Second Edition: F, Fb, Fp, and FBS). c Selected case series. d Failure on 2 effort measures (Reliable Digit Span, Word Memory Test, Validity Indicator Profile, or MMPI-2: FBS). e TOMM administration manual (1996). TOMM1 based on the standard cutoffs used for the latter trials ( 45), but others provided data on all individuals that eventually passed the full administration of the TOMM (often showing slightly lower cutoff scores to maintain 100% specificity). The cutoffs in Table 1 provide evidence that those performing above a particular score for TOMM1 will go on to pass the full administration of the TOMM 100% of the time (providing some justification for discontinuing the TOMM after TOMM1). When several cutoffs for TOMM1 were available, cut scores in Table 2 were used that provided specificities 90%. Four studies and the TOMM test manual also provided data showing that 100% of patients scoring 33 (Tombaugh, 1996), 31 (Bauer et al., 2007), 29 (O Bryant, Engel, Kleiner, Vasterling, & Black, 2007), 27 (Horner, Bedwell, & Duog, 2006), or 25 (Hilsabeck, Gordon, Hietpas-Wilson, & Zartman, 2011) on TOMM1 did not pass Trial 2 and/or the retention trials suggesting that further trials may be unnecessary (e.g., reflecting 100% sensitivity). Overall, prior studies across a variety of populations/settings (e.g., analog malingerers, litigants, TBI, pain, normal undergraduates, and psychiatric) indicate that a cut score of approximately 40 (weighted mean sensitivity ¼ 0.77, specificity ¼ 0.92) provides high levels of accuracy in predicting performance on the full administration of the TOMM and/or other freestanding/embedded measures of test validity. In general, samples that included patients with dementia (Hilsabeck et al., 2011; Horner et al., 2006; Tombaugh, 1996) or severe amnestic disorders (Greve, Bianchini, & Doane, 2006) required slightly lower TOMM1 cut scores to maintain specificity rates at 90% or higher. Compared with the latter two trials of the TOMM, which is the standard procedure in determining failure on the test, there is evidence to suggest that TOMM1 shows similar or even slightly greater effect sizes, hit rates, and/or sensitivity to poor effort (Etherton, Bianchini, Greve, & Ciota, 2005; Greve, Bianchini, Black, et al., 2006; Greve, Bianchini, & Doane, 2006; Greve et al., 2008; Greve, Etherton, Ord, Bianchini, & Curtis, 2009; Jasinski et al., 2011; Lindstrom, Lindstrom, Coleman,

4 4 J.H. Denning / Archives of Clinical Neuropsychology Nelson, & Gregg, 2009; Marshall et al., 2010; Musso, Barker, Jones, Roid, & Gouvier, 2011; Powell, Gfeller, Hendricks, & Sharland, 2004; Sollman, Ranseen, & Berry, 2010; Tan et al., 2002). TOMM1 has also shown equivalent or slightly higher correlations with other measures of psychiatric/cognitive malingering compared with Trial 2 and/or the retention trials (McCaffrey, O Bryant, Ashendorf, & Fisher, 2003; Ruocco et al., 2008; Whiteside, Dunbar-Mayer, & Waters, 2009; Whitney, Davis, Shepard, & Herman, 2008). Extracting additional information from TOMM1 might also be helpful in quickly and efficiently identifying exaggerated memory dysfunction without having to administer Trial 2 or the retention trial. The TOMM initially appears quite difficult, but as the test progresses (as one continues through Trial 1 and eventually shown the items again during Trial 2), in our experience, patients have commented on the simplicity of the task. Therefore, those who perform poorly (initially) on TOMM1 may then choose to perform much better on Trial 2 and the retention trial (resulting in an overall passing score). This highly variable pattern is consistent with a simulation study showing the initial trial of the WMT (Immediate Recognition trial ¼ 63% failure rate) being much more sensitive to suspect effort compared with the Delayed Recognition trial (18% failure rate, Marshall et al., 2010). In addition, greater failure rates on the Abbreviated Hiscock Forced Choice Procedure were found when the test was administered at the beginning of a test battery compared with at the end (Guilmette, Whelihan, Hart, Sporadeo, & Buongiorno, 1996), suggesting that with experience/exposure to more demanding test measures, individuals begin to realize that these types of freestanding measures are very easy. The National Academy of Neuropsychology statement on effort assessment (Bush et al., 2005) noted that one measure of test validity should be administered early in the test battery which further emphasizes the importance of assessing test validity (presumably) before the patient becomes more aware of the nature/purpose of these types of tests. It was our intention to capitalize on the patient s initial misperception of the TOMM being a difficult test. We predicted that if patients were choosing to exaggerate memory deficits on the TOMM, they would do this very early in the test administration of TOMM1. In order to capture this particular approach to testing (e.g., poor effort very early in the testing), we tabulated the number of errors a patient committed on the first 10 items of TOMM1 as a measure of test validity (referred to as TOMMe10). This is the first study that we are aware of that has assessed this type of error pattern on the TOMM and how well it might predict non-credible performance on other well-validated validity measures. It has also been recommended that validity testing be carried out throughout the test battery to ensure adequate effort across multiple areas of cognitive and behavioral functioning (Boone, 2009; Heilbronner et al., 2009). This may be very timeconsuming if one relies on administering several freestanding measures scattered throughout testing. However, by utilizing embedded measures of test validity (tests already part of one s standard battery), one can minimize not only the time needed to administer extra tests but also monitor the patient s level of effort across several hours of testing. Embedded measures often rely on simple cut points below which most credible, clinically impaired patients do not typically perform, or more complex statistical calculations are used to combine a variety of test scores (Larrabee, 2003; Meyers et al., 2011; Schutte, Millis, Axelrod, & VanDyke, 2011; Victor, Boone, Serpa, Beuhler, & Ziegler, 2009; Wolfe et al., 2010). Therefore, another goal of the current project was to take advantage of embedded measures to predict failure on other effort tests using logistic regression analysis (using all embedded measures as a continuous variable) or cut scores for each test. It was predicted that a combination of embedded measures (already well validated in previous malingering research) would also provide an accurate prediction of test validity. The current project will assess the relative accuracy of two short forms of the TOMM (TOMM1 and TOMMe10) and a combination of five embedded measures (encompassing eight individual test scores) in predicting exaggerated cognitive deficits as measured by a well-validated freestanding measure of test validity, the Medical Symptom Validity Test (MSVT, Green, 2004). Method Participants A total of 497 patients referred to an outpatient Veterans Affairs Neuropsychology Clinic in the southern USA were included in this retrospective, anonymous, database study that was approved by the local Veteran s Affairs IRB. Referrals were primarily from Primary Care Clinics, Psychiatry, Neurology, and 15% were from Compensation and Pension disability evaluations mostly for TBI (99% were mtbi). Those excluded from the sample were patients diagnosed with dementia, but no other medical, psychiatric, or substance abuse disorders were used to exclude subjects in an attempt to generalize to a wide range of outpatient populations. The most common psychiatric diagnoses (based on Diagnostic and Statistical Manual of Mental Disorders-fourth edition-text revision diagnostic criteria) were some type of Depressive Disorder (56%), post-traumatic stress disorder (38%), Anxiety Disorder, not otherwise specified (NOS) (28%), Cognitive Disorder NOS/Mild Cognitive

5 Impairment (28%), Alcohol/Substance Abuse/Dependence (20%), No Diagnosis (5%), Bipolar Disorder (4%), and Psychotic Disorder (3%). The final sample was primarily male (93%), Caucasian (90%), and African American (9%). The most common medical conditions were hypertension (58%), chronic pain (49%), elevated lipids (38%), sleep apnea (25%), diabetes (22%), coronary artery disease (22%), hearing problems (21%), chronic obstructive pulmonary disease (10%), cerebrovascular accident (6%), and seizure disorder (4%). Procedures J.H. Denning / Archives of Clinical Neuropsychology 5 All patients were tested by the author, psychometrician, or a trained/supervised psychology intern over a 5-year period ( ) in an outpatient setting. Tests administered, order of administration, and the specific measures of interest are found in Table 3. All patients had completed the tests of interest, but other tests were also administered as part of a comprehensive neuropsychological evaluation. The validity measures were typically given during the early and middle portions of the testing session and are relatively consistent with recent suggestions that the assessment of test validity include measures during the early part of the evaluation and continuously throughout the test battery (Boone, 2009; Bush et al., 2005). The MSVT was used to categorize patients as either good effort (passing) or poor effort (failing) according to the standard scoring rules found in the test manual across the Immediate Recognition, Delayed Recognition, and Consistency trials. Scores for the two short forms of the TOMM were obtained for each patient: Total number of items correct for TOMM trial 1 (TOMM1) and total number of errors on the first 10 items of TOMM1 (TOMMe10). These two scores were assessed separately to predict performance on the MSVT using binomial logistic regression. In addition, it is also clinically useful to have a variety of possible cutoff scores depending on the population/clinical setting of interest. This would allow the individual clinician to decide what is most critical in making diagnostic decisions using TOMM1 and TOMMe10 (e.g., maximizing sensitivity or specificity). Therefore, analyses will provide a range of sensitivity/specificity values for all possible cut scores as well as utilizing receiver operating characteristic (ROC) analyses to calculate the overall accuracy of the TOMM measures. Four of the five embedded measures chosen for this study have been evaluated previously. The Wechsler Adult Intelligence Scale, Third Edition (WAIS-III; Wechsler, 1997) Processing Speed Index and Working Memory Index (WMI) have been Table 3. General order of administration and measures used for data analyses # Test administered Scores used for analysis 1 Test of Memory Malingering (Trial 1) Raw Score; errors on the first 10 items 2 Brief Visuospatial Memory Test-Revised (Immediate Recall) 3 Medical Symptom Validity Test (Immediate Recognition) Percent correct 4 Finger Tapping Test Average of first three trials, dominant hand 6 WAIS-III: Digit-Symbol Coding Standard score (PSI) 7 WAIS-III: Symbol Search Standard score (PSI) 5 Medical Symptom Validity Test (Delayed Recognition) Percent correct 8 Brief Visuospatial Memory Test-R (Recall/Recognition) Correct hits (recognition trial) 9 Trail-making Test Parts A, B 10 CVLT-II (Immediate Recall) 11 WAIS-III: Block Design 12 WAIS-III: Digit span Standard score (WMI) 13 WAIS-III: Arithmetic Standard score (WMI) 14 CVLT-II (Delayed Recall/Yes No Recognition) 15 WAIS-III: Letter-Number Sequencing Standard score (WMI) 16 CVLT-II (Forced Choice) Raw score correct 17 Stroop Color-Word Test (Golden Version) 18 Auditory Consonant Trigrams (Stuss Version) 19 WAIS-III: Similarities 20 Rey Complex Figure Test: Copy (Meyers Version) 21 Controlled Oral Word Association Test (FAS, CFL) 22 Animal Naming 23 Brief Test of Attention 24 Boston Naming Test 25 Wide Range Achievement Test, Third Edition (Reading) 26 Wechsler Test of Adult Reading 27 Wisconsin Card Sorting Test (WCST) or modified WCST (Nelson, 48-card version) Notes: WAIS-III ¼ Wechsler Adult Intelligence Scale, Third Edition; CVLT-II ¼ California Verbal Learning Test, Second Edition; WMI ¼ Working Memory Index; PSI ¼ Processing Speed Index. Boldface in this table is to highlight only the specific tests that were used in the statistical analyses.

6 6 J.H. Denning / Archives of Clinical Neuropsychology shown to reliably identify those providing invalid test performance (Curtis, Greve, & Bianchini, 2009; Etherton, Bianchini, Ciota, Heinly, & Greve, 2006; Etherton, Bianchini, Heinly, & Greve, 2006; Greve et al., 2008). The Finger Tapping Test (FTT) has also been shown to have acceptable accuracy statistics in predicting poor effort on cognitive testing (Arnold et al., 2005; Boone, 2007; Larrabee, 2007). The California Verbal Learning Test, Second Edition (CVLT-II) Forced Choice (FC) trial was initially designed to assess poor effort (Delis, Kramer, Kaplan, & Ober, 2000), and studies have found this measure to be an acceptable indicator of poor effort (Moore & Donders, 2004). The final embedded measure, the Brief Visuospatial Memory Test-Revised (BVMT-R) Recognition Hits Trial (Benedict, 1997) has not been studied previously, but given the relatively easy nature of this yes/no delayed recognition of six designs, it was hypothesized that this may be useful in assessing test validity. The number of correct hits out of a possible six was utilized in the current study. This measure of visual recognition memory also complemented the broad areas of cognitive functioning already covered by the four other embedded measures (e.g., auditory attention, psychomotor processing speed and visual attention, fine motor speed, and verbal recognition memory). What has not been done, to our knowledge, is to combine these specific embedded measures in an effort to increase their accuracy over and above any single embedded measure alone. These five measures will first be assessed for any problems with collinearity that may over-inflate the accuracy of the regression equations. The embedded measures will then be entered into a binary logistic regression (backwise step procedure) with MSVT pass/fail as the outcome variable. The accuracy statistics of each individual embedded measure, based on empirically derived cutoffs from the literature, will also be presented as a comparison. The accuracy of the TOMM trial 1 measures using logistic regression will also be assessed with pass/fail of the MSVT as the binary outcome predictor. Results General demographic information and differences between groups passing/failing the MSVT are presented in Table 4. There was an overall failure rate on the MSVT of 33% based on the three easy subtests (immediate recognition, delayed recognition, or consistency 85%). There were no significant differences in age, educational attainment, or estimated WAIS-III Full-Scale IQ between those passing or failing the MSVT. There was a significant difference in WRAT-3 Reading with those failing the MSVT scoring slightly lower, but both groups were still within the average range. Test performance of those passing or failing the MSVT are presented in Table 5 along with respective effect sizes (Cohen, 1988). All t-test comparisons between those passing or failing the MSVT were significant with effect sizes ranging from medium to large (d ¼ 0.7 [FTT] to d ¼ 2.0 [TOMM1]). The MSVT showed the greatest effect sizes but this would be expected given that it was the grouping variable of interest. The two TOMM1 measures showed the largest effect sizes compared with the individual embedded measures. It should be noted that those passing the MSVT performed within the average range across all measures of interest. TOMM Trial 1 The results of the binary logistic regression analyses for TOMM1 and TOMMe10 are found in Table 6. Sensitivity rates for TOMM1 (cutoff 40) and TOMMe10 (cutoff 1 errors) were quite similar (TOMM1 ¼ 72%, TOMMe10 ¼ 71%). Specificities were above 90% for both measures, an overall hit rate is 87% for TOMM1 and 86% for TOMMe10, and likelihood Table 4. Demographics and estimated premorbid functioning of the sample (N ¼ 497) Mean (SD) Range MSVT t Pass (n ¼ 331) Fail a (n ¼ 166) Mean (SD) Mean (SD) Age (years) 48.6 (15.6) (16.0) 46.9 (14.5) 1.74 Education (years) 12.4 (2.5) (2.5) 12.2 (2.3) 1.47 WRAT-3: Reading (Standard Score) b 95.7 (12.0) (11.8) 92.9 (12.1) 3.52* WTAR: Estimated Full-Scale IQ c 99.2 (10.0) (10.5) 97.9 (8.9) 1.45 Notes: WRAT-3 ¼ Wide Range Achievement Test, Third Edition; WTAR ¼ Wechsler Test of Adult Reading (corrected for age, education, gender, ethnicity); MSVT ¼ Medical Symptom Validity Test. a 33% failed the MSVT. b n ¼ 470. c n ¼ 231. *p,.001.

7 J.H. Denning / Archives of Clinical Neuropsychology 7 Table 5. Performance on freestanding and embedded measures of those passing or failing the MSVT Test measures MSVT t* d Pass Mean (SD) Fail Mean (SD) TOMM1 47 (3.6) 35 (8.7) TOMMe10.3 (.8) 2.7 (1.9) MSVT Immediate Recognition (% correct) 99 (2.4) 78 (16.2) Delayed Recognition (% correct) 98 (3.0) 72 (17.7) Consistency (% correct) 98 (3.4) 73 (12.3) FTT (dominant hand, mean of first three trials) 44 (10.9) 36 (11.5) WAIS-III Processing Speed Index (Standard Score) 92 (12.7) 80 (10.4) Working Memory Index (Standard Score) 99 (12.6) 89 (13.1) CVLT-II: FC(Raw Score) 15.8 (.6) 14.1 (2.5) BVMT-R (Recognition Hits) 5.5 (.8) 4.4 (1.4) Notes: TOMM1 ¼ Test of Memory Malingering Trial 1; TOMMe10 ¼ Errors on the first 10 items of TOMM1; MSVT ¼ Medical Symptom Validity Test; FTT ¼ Finger Tapping Test; WAIS-III ¼ Wechsler Adult Intelligence Scale, Third Edition; CVLT-II ¼ California Verbal Learning Test, Second Edition; FC ¼ Forced Choice; BVMT-R ¼ Brief Visuospatial Memory Test, Revised. d ¼ effect size (Cohen s d). *p,.001 for all comparisons. Table 6. Results of binary logistic regression analysis of TOMM1 and TOMMe10 in predicting passing or failing the MSVT Variable B SE Wald p-value Exp(B) 95% CI Exp(B) Sen Spec Overall LR TOMM1 a Constant TOMMe10 b Constant Notes: TOMM1 ¼ Test of Memory Malingering Trial 1; TOMMe10 ¼ Errors on the first 10 items of TOMM1; Sen ¼ sensitivity; Spec ¼ specificity; Overall ¼ overall hit rate; LR ¼ likelihood ratio. a TOMM1 cut-off 40. b TOMMe10 cut-off 1. ratios (LRs) are 12 and 11, respectively. The overall models were statistically reliable for both variables in predicting MSVT performance (TOMM1: x 2 ¼ , p,.001; TOMMe10: x 2 ¼ , p,.001) and the amount of variance explained by the model was excellent (TOMM1: Nagelkerke R 2 ¼ 63.8%; TOMMe10: Nagelkerke R 2 ¼ 55.3%). ROC analyses showed excellent to outstanding model discrimination (Hosmer & Lemeshow, 2000) across both measures (TOMM1: area under the curve [AUC] ¼ 92%, 95% CI ¼ ; TOMMe10: AUC ¼ 87%, 95% CI ¼ ). Tables 7 and 8 show the full range of scores for TOMM1 and TOMMe10 with corresponding sensitivities, specificities, as well as positive and negative predictive values at various base rates of malingering. Embedded Measures The accuracy and LRs of each embedded measure in predicting MSVT performance was based on empirically derived cutoffs from the literature and are presented in Table 9. The CVLT-II: FC was the best overall predictor (sensitivity ¼ 40%, specificity ¼ 95%, hit rate ¼ 77%, LR ¼ 8) with sensitivities across all embedded measures ranging from 15% to 45%, with three of five measures showing specificities 90%. The relative accuracy of simply adding the number of failed embedded measures to predict performance on the MSVT is also shown in Table 9. Others have viewed this method as relatively straightforward and highly accurate using a variety of embedded measures and methods of grouping good/poor effort groups (Jasinski et al., 2011; Larrabee, 2003, 2008; Meyers & Volbrecht, 2003; Pella, Hill, Shelton, Elliott, & Gouvier, 2012; Schroeder & Marshall, 2011; Victor et al., 2009). Consistent with these studies, we found that failing 2 embedded measures provided the greatest accuracy (hit rate ¼ 75%, LR ¼ 6), but sensitivity was low (40%) with acceptable specificity (93%). However, there did not appear to be any significant advantage to adding failed embedded measures as the CVLT-II: FC trial alone showed slightly better accuracy statistics. Table 10 provides the results of the binary logistic regression for the five embedded measures as continuous predictors of the MSVT. There were no significant problems with collinearity given a variance inflation factor of,5.0 (Kutner, Nachtsheim, &

8 8 J.H. Denning / Archives of Clinical Neuropsychology Table 7. Sensitivity, specificity, PPV, and NPV of TOMM1 in predicting passing or failing the MSVT Cut score SN SP Base rate (0.40) Base rate (0.25) Base rate (0.10) PPV NPV PPV NPV PPV NPV Notes: Embolden values indicate cut scores with the greatest accuracy based on binary logistic regression analyses; SN ¼ sensitivity; SP ¼ specificity; PPV ¼ positive predictive value; NPV ¼ negative predictive value. Table 8. Sensitivity, specificity, PPV, and NPV of TOMMe10 in predicting passing or failing the MSVT Cut score SN SP Base rate (0.40) Base rate (0.25) Base rate (0.10) PPV NPV PPV NPV PPV NPV Notes: Embolden values indicate cut scores with the greatest accuracy based on binary logistic regression analyses; SN ¼ sensitivity; SP ¼ specificity; PPV ¼ positive predictive value; NPV ¼ negative predictive value.

9 J.H. Denning / Archives of Clinical Neuropsychology 9 Table 9. Results of the five embedded measures predicting passing or failing the MSVT based on empirically derived cut scores from the literature Embedded measure Sensitivity Specificity Hit rate LR Cutoff Reference FTT* Arnold et al. (2005) WAIS-III: PSI Etherton et al. (2006) WAIS-III: WMI Etherton et al. (2006) CVLT-II (FC) Moore and Donders (2004) BVMT-R (Hits) NA Any two tests failed Notes: FTT ¼ Finger Tapping Test; WAIS-III ¼ Wechsler Intelligence Scale, Third edition; PSI ¼ Processing Speed Index; WMI ¼ Working Memory Index; CVLT-II (FC) ¼ California Verbal Learning Test, Second edition, Forced Choice trial; BVMT-R (Hits) ¼ Brief Visuospatial Memory Test-Revised, correct number of hits on Yes/No recognition trial. LR ¼ likelihood ratio. *FTT based on mean of first three trials of dominant hand. Table 10. Results of backward step-wise binary logistic regression analysis of the five embedded measures as continuous variables predicting passing or failing the MSVT (N ¼ 307) Variables B SE Wald p-value Exp(B) 95% CI Exp(B) Sen Spec Overall LR Model FTT* WAIS-III: PSI WAIS-III: WMI CVLT-II (FC) BVMT-R (Hits) Constant Notes: FTT ¼ Finger Tapping Test; WAIS-III ¼ Wechsler Intelligence Scale, Third edition; PSI ¼ Processing Speed Index; WMI ¼ Working Memory Index; CVLT-II (FC) ¼ California Verbal Learning Test, Second edition, Forced Choice trial; BVMT-R (Hits) ¼ Brief Visuospatial Memory Test-Revised, correct number of hits on Yes/No recognition trial; Sen ¼ sensitivity; Spec ¼ specificity; LR ¼ likelihood ratio; Overall ¼ overall hit rate; based on regression equation cut-off *FTT based on mean of first three trials of dominant hand. Neter, 2004) for each variable based on a linear regression of the five measures with both the MSVT Immediate and Delayed Recognition trials. The overall model was statistically reliable (x 2 ¼ , p,.001) and the amount of variance explained by the model was excellent (Nagelkerke R 2 ¼ 47.0%). Results indicate lower sensitivity but comparable specificity to the TOMM1 variables (sensitivity ¼ 58%, specificity ¼ 94%, overall hit rate ¼ 83%, LR ¼ 9), with model discrimination seen as excellent (AUC ¼ 86.9%, 95% CI ¼ ). The accuracy statistics noted above utilized a range of patients and performance levels including those with mild cognitive deficits (denoted as Cognitive Disorder NOS). Therefore, these findings would hopefully generalize to a wide range of populations and settings. However, if one could be confident that the population or individual patient is not suspected of having any significant cognitive decline (e.g., remote mtbi, chronic pain, mild depression/anxiety, etc.) than more stringent cutoffs could provide a more accurate assessment of non-credible performance. With this in mind, the above calculations were re-run excluding those with dementia as well as those diagnosed with clearly evident cognitive deficits (Cognitive Disorder NOS diagnoses) and can be found in Table 11. The overall accuracy statistics for TOMM1 and TOMMe10 showed modest increases in overall accuracy based on AUC and LRs, but the embedded measures regression dramatically improved (LR increased from 9 to 17). All three predictors showed AUC 90% with the greatest improvements in sensitivity shown by the embedded measures regression formula (sensitivity increased 20% from 58% to 78%) and TOMM1 (sensitivity increased 11% from 72% to 83%) with the cutoff being raised to 41. Therefore, this table may be more useful in settings/populations where there is no suspected neurological dysfunction or significant medical/psychiatric history that would tend to significantly impact cognitive functioning. Discussion The purpose of this study was to further assess the accuracy of TOMM1 in predicting performance on a well-validated and commonly used measure of cognitive test validity (MSVT). In addition, we explored the accuracy of a novel measure from TOMM1 by tabulating TOMMe10. Finally, given the recommendations to assess the validity of a patient s performance

10 10 J.H. Denning / Archives of Clinical Neuropsychology Table 11. Accuracy of TOMM1, TOMMe10, and the five embedded measures regression formula after removing those with a diagnosis of Cognitive Disorder NOS from the sample (N ¼ 344) Measures Sensitivity Specificity Hit rate AUC LR TOMM1 a TOMMe10 b Regression c FTT* WAIS-III: PSI WAIS-III: WMI CVLT-II (FC) BVMT-R (Hits) Notes: WAIS-III ¼ Wechsler Intelligence Scale, Third edition; PSI ¼ Processing Speed Index; WMI ¼ Working Memory Index; CVLT-II (FC) ¼ California Verbal Learning Test, Second edition, Forced Choice trial; BVMT-R (Hits) ¼ Brief Visuospatial Memory Test-Revised, correct number of hits on yes/no recognition trial; AUC ¼ area under the curve; LR ¼ likelihood ratio. a TOMM1 cut-off 41. b TOMMe10 cut-off 1 errors. c Regression equation cut-off *FTT based on the mean of first three trials of dominant hand. continuously throughout testing (Boone, 2009; Heilbronner et al., 2009), we also assessed the accuracy of a group of embedded measures frequently cited in the malingering research literature. TOMM Trial 1 Consistent with an informal review of available studies providing TOMM1 accuracy statistics (Table 2), the current study found a similar pattern of results with the greatest accuracy in our sample found at a cutoff of 40 (sensitivity ¼ 72%, specificity ¼ 94%, hit rate ¼ 87%, LR ¼ 12) with the performance of TOMM1 overall showing a classification accuracy of 92%. There appears to be converging evidence across multiple studies, as well as in our mixed clinical sample of non-demented veterans, that TOMM1 is a robust measure predicting performance on other freestanding validity measures. Current results also suggest that TOMM1 shows greater sensitivity to invalid test performance compared with the standard TOMM administration as reviewed by Green (2007) and in a recent meta-analysis of the TOMM (Sollman & Berry, 2011). Our results are relatively consistent with the recently proposed Albany Consistency Index (ACI) which increased the sensitivity of the TOMM to poor effort as measured by the WMT (standard TOMM: sensitivity ¼ 33%, ACI ¼ 71%). Given that the standard TOMM requires administration of two or three trials, and analysis of response consistency as described by the ACI requires all three trials to be administered (Gunner et al., 2012), TOMM1 appears to provide a more efficient and more accurate measure of test validity compared with the latter two trials of the TOMM. Armistead-Jehle and Gervais (2011) and Green (2011) have recently shown that the standard cutoffs for the TOMM show acceptable specificity rates (.90%), but other freestanding measures such as the Non-verbal MSVT (NV-MSVT, Green, 2008) are approximately twice as sensitive to invalid test performance. However, consistent with our hypotheses, utilizing TOMM1 40 as a cutoff and using pass/fail on the MSVT as the comparison measure (similar to the methods of Armistead-Jehle et al.), we found sensitivity rates for TOMM1 to be over twice as high compared with the standard TOMM administration in that study (sensitivity ¼ 35% vs. 72%). In fact, our sample, along with previous studies with TOMM1 data (see Table 2: TOMM1 40, sensitivity ¼ 0.77, specificity ¼ 0.92), tends to show greater sensitivity than the NV-MSVT described in the Armistead-Jehle and colleagues study. Furthermore, when removing those diagnosed with any level of suspected cognitive impairment from our sample (similar to the sample characteristics of Armistead-Jehle & Gervais, 2011: Disability seeking, non-head-injured sample), and increasing the cut score to 41, the sensitivity of TOMM1 increased 11% (from 72% to 83%) while still maintaining high specificity (93%, AUC ¼ 94%). Therefore, accuracy statistics in Table 11, and TOMM1 41, may be more appropriate in settings/populations where there is no suspected neurological dysfunction or severe medical/psychiatric disorder that would tend to significantly impact cognitive functioning. With regard to the TOMMe10, at a cutoff of 1 errors on the first 10 items, this measure showed comparable accuracy statistics with TOMM1 noted above (sensitivity ¼ 71%, specificity ¼ 93%, hit rate ¼ 86%, LR ¼ 11) with the performance of TOMMe10 overall showing a classification accuracy of 87%. This is not surprising due to the very high correlation between TOMM1 and TOMMe10 (r ¼ 2.91, p,.001) and suggests that administering the entire TOMM1 may actually

11 J.H. Denning / Archives of Clinical Neuropsychology 11 be redundant and lack significant incremental validity over the TOMMe10. However, TOMMe10 showed only a modest improvement in sensitivity and overall accuracy when those with cognitive impairment were removed from the sample. Nevertheless, the accuracy of TOMMe10 in our sample provides preliminary evidence of a feasible short short form of the TOMM that essentially reduces the number of items administered by up to 93% (150 items for all three trials to just 10 items). This savings in administration time may have the advantage of not giving away too much about the nature of the test and preserve the TOMM s intended purpose if a repeat administration at a later time is warranted. The TOMMe10 could also be used as a very brief screening measure of effort particularly when time constraints might be an issue. Overall, when the assessment of test validity is carried out very early in the testing session with TOMM1 or TOMMe10, the sensitivity to poor effort is enhanced (over the standard TOMM administration) while at the same time maintaining the low rates of false-positive errors. The original TOMM administration instructions include decision rules based on poor performance on Trial 2 and the retention trial, but only include the interpretation of below chance performance for TOMM1 (which is quite rare in known malingerers, see Greve, Binder, & Bianchini, 2009; Kim et al., 2010). It is unclear why the TOMM was originally designed to essentially ignore almost 30% 50% of the test data collected (if administering all three trials or just Trials 1 and 2, respectively). We are unaware of any neuropsychological test in frequent use today, let alone the most frequently used in the assessment of malingering/effort (Sharland & Gfeller, 2007) that essentially ignores almost half of the test data collected. We believe that the previously neglected TOMM1 scores will be more closely scrutinized by clinicians/researchers in the future as others become more aware of the greater efficiency and improved sensitivity of these measures over the standard TOMM administration. The current results of TOMM1 could be used immediately by the average clinician with little need for supplemental statistical calculations or alteration in clinical practice. If one uses the TOMM, one already has the scores for TOMM1 as well as the patient s performance across the first 10 items of Trial 1. The TOMM1 and TOMMe10 tables may further assist the practicing clinician by providing a range of cutoffs and sensitivity/specificity rates allowing one to decide what cutoffs are most appropriate for their particular setting (based on tolerance for false-positive or false-negative errors). Previous reports have raised concerns about the use of TOMM1 as a freestanding measure of test validity in forensic contexts and note that it may only be appropriate as a screening measure in clinical settings (Bauer et al., 2007; Hilsabeck et al., 2011; O Bryant et al., 2007, 2008). However, based on the current study and review of TOMM1 accuracy statistics across a range of populations/contexts covering over 2,600 individuals (Table 2), it appears that TOMM1 shows acceptable accuracy statistics in predicting non-credible performance patterns and could be considered an independent measure of test validity in its own right. The cutoffs proposed in the current study for TOMM1 and TOMMe10 should be used with caution in various clinical patient samples where moderate to severe cognitive limitations are commonplace. The current cut scores appear to be most appropriate in patient samples that include those with relatively mild cognitive deficits, but these cutoffs may be too high if the sample includes individuals with dementia (Horner et al., 2006; Greve et al., 2009) or memory disorders (Greve, Bianchini, Black, et al., 2006). Frequently, lower cutoff scores for TOMM1 in the literature (Table 2) included patients diagnosed with dementia, so future research should develop more accurate cut scores for that population. Other patient groups where the proposed TOMM1 and TOMMe10 cutoffs should be viewed with significant caution include those with mental retardation or those receiving inpatient treatment for psychotic disorders. Those with mental retardation have shown high rates of poor performance on TOMM1 with 30% performing 39 in one study (Shandera et al., 2010). Full administration of the TOMM would be recommended in mentally retarded populations as the retention trial has shown specificity rates at acceptable levels (.90%; Shandera et al., 2010). Similar cautions for TOMM1 cutoffs are highlighted by a study by Duncan (2005) who screened psychotic forensic inpatients for adequate effort. He found that 24% of those with concentration problems (as measured by an index on the Conners Continuous Performance Test-II) scored 39 on TOMM1, but only 5% of those without concentration problems performed this poorly. Weinborn and colleagues (2003) also found substantial rates of low TOMM1 scores in psychiatric inpatients with or without greater implied incentives to provide poor effort (TOMM1 40: civil commitment ¼ 39%, competency to stand trial ¼ 48%). Future research should address adjusting possible cutoffs in these specific populations in order to take into account moderate to severe cognitive limitations. Embedded Measures Another purpose of the current study was to cross-validate and provide additional information regarding the accuracy of a variety of individual and combined embedded measures of test validity. Most of the embedded measures in the current study have been used by others in malingering research across a range of patient populations. In our sample, individual measures showed sensitivities ranging from 15% to 45% with specificities above 90% (except FTT ¼ 78% and BVMT-R hits ¼ 89%). The overall hit rates for each individual measure ranged from 70% to 77%. Based on previous work showing that a

12 12 J.H. Denning / Archives of Clinical Neuropsychology regression-based approach may improve sensitivity without negatively impacting specificity (Schutte et al., 2011; Victor et al., 2009), we used the same approach by utilizing embedded measures as continuous variables. We were able to improve sensitivity (58%) while maintaining specificity 90%. Hit rate (83%) and overall accuracy (AUC ¼ 87%) of the regression equation of embedded measures was also more accurate compared with any individual embedded indicator based on empirically derived cutoffs. Others have also proposed adding the number of failed embedded measures to simplify decision-making, increase sensitivity, and reduce false-positive errors (Larrabee, 2003, 2008; Myers, Volbrecht, Axelrod, & Reinsch-Boothby, 2011; Pella et al., 2012; Victor et al., 2009). By using this method, failure on any two or more measures showed the greatest balance between sensitivity (40%), specificity (93%) and had a hit rate of 75%. However, this method was less accurate than utilizing the embedded measures as continuous variables noted above which contrasted with Larrabee s (2003) findings where any pair-wise comparison of failed measures was more accurate. This highlights the importance of how a group of five different embedded measures used by Larrabee, as well as the criterion for group membership (e.g., below chance performance on the Portland Digit Recognition Test [PDRT] and failure on PDRT and one additional measure), can impact the accuracy statistics of various measures. Therefore, caution should be exercised when combining any two failed embedded measures across any test battery as this may be less sensitive/accurate than the other methods of combining validity indicators. Several studies have highlighted the usefulness of combining embedded measures across different tests to predict poor effort. Schutte and colleagues (2011) created a composite measure of embedded memory measures (very similar to the current study using logistic regression techniques) based on the CVLT-II (Trial 5, FC), Rey Complex Figure Test (RCFT) (immediate recall), and Wechsler Memory Scale-III Verbal Paired Associates-2 to predict MSVT and TOMM performance. Results were strikingly similar to our findings despite using (mostly) different measures (sensitivity ¼ 59% specificity ¼ 95%, AUC ¼ 84%). Victor and colleagues (2009) also used regression techniques and combined four embedded validity indicators (based on the RCFT, Reliable Digit Span, Rey Auditory Verbal Learning Test, FTT) to predict pass/fail on 2 freestanding validity measures. Overall, the accuracy of the regression analysis was quite impressive (sensitivity ¼ 86%, specificity ¼ 96%, hit rate ¼ 92%), and contrary to our findings, Victor and colleagues found that failure on any two embedded indicators was just as accurate as the regression equation. Several studies have also found that combining a variety of embedded measures across multiple tests (using a variety of techniques and criteria for test validity) to be highly accurate and deserves ongoing study (Larrabee, 2003, 2008; Myers, Volbrecht, Axelrod, & Reinsch-Boothby, 2011; Schroeder & Marshall, 2011; Whiteside, Wald, & Busse, 2011). The current study contributes to this growing research literature by providing three methods of determining the validity of various embedded measures: Utilizing them as continuous measures (regression equation), individually (based on empirically derived cut scores), or combining two or more failures in predicting test validity. The regression equation using all five variables (WMI, PSI, CVLT-II FC, FTT, and BVMT-R Hits) was more accurate in predicting performance on the MSVT than any individual embedded measure alone or in combination. There are numerous validity measures that have been derived from embedded indices (Boone, 2007; Larrabee, 2007). Unfortunately, because there are so many different calculations possible across multiple tests, the average clinician may be quickly overwhelmed by which ones and/or how many one should actually interpret when making a determination of valid/ invalid test results. Given that neuropsychologists often administer multiple tests to their patients, there are dozens of possible calculations that could be derived with very little research to guide them in which combination of measures are the most efficient to administer/score as well as being the most accurate (and not overly redundant). For example, Miele and colleagues (2012) assessed the utility of 15 embedded measures derived from the WAIS-R and the Halstead-Reitan Neuropsychological Battery. They found reliable digit span alone (cutoff 7) correctly classified 74% of individuals and no other embedded measure significantly impacted the classification rate. In other words, the other 14 embedded measures were essentially redundant and unnecessary. Additional research analyzing the combination of embedded measures will be helpful in further determining the most efficient group of test scores to interpret based on patient population, context of evaluation, and specific cutoffs. Another unique finding from the current study was the BVMT-R recognition hits trial contributed significantly to the regression equation in predicting MSVT performance and was one of the more sensitive individual freestanding measures (45%) at a cut score of 4 correct hits. It is not surprising that this measure is useful in predicting test validity as it tends to be a somewhat easy task and follows the typical forced-choice format (e.g., yes/no) that many freestanding validity measures have utilized. Finally, using a measure of visual memory complements the other cognitive domains assessed by the four other embedded measures of test validity (e.g., verbal memory, auditory attention, fine motor speed, and visual attention/motor speed). This also is in line with suggestions to cover a variety of cognitive domains when assessing test validity (Heilbronner et al., 2009). Cross-validation of the newly introduced BVMT-R recognition hits trial as an embedded measure would also be helpful as no other study has utilized a cutoff of 4 correct hits to indicate invalid test performance. Likewise, cross-validation